Introduction

When searching for information on the website, it is common to come across a flood of both relevant and irrelevant results. In that case, how can people discern the required information?

This article explores and utilises the integration of knowledge-enhanced pre-trained language models (KE-PLMs) and Knowledge Graphs to advance open-domain conversational question answering. KE-PLMs use external knowledge sources to enhance language understanding and generation abilities, while Knowledge Graphs use a graph-structured model to represent knowledge extracted from text. By combining these approaches, we aim to improve the accuracy, relevance and informativeness of the question answering system. Furthermore, the article will delve into potential future directions and improvement in KE-PLMs and Knowledge Graphs across diverse domains.

Purposes

Open-Domain Conversational Question Answering (QA) systems designed to provide accurate and relevant responses to user queries. While traditional QA approaches often rely solely on machine learning models trained on large text corpus which has limitations of lack of external knowledge context to answer questions effectively. To solve this problem, KE-PLMs and Knowledge Graphs are proposed to enhance the performance of QA systems.

Knowledge-Enhanced Pre-trained Language Models (KE-PLMs)

Pre-trained Language Models (PLMs) like BERT or GPT which are trained with unsupervised objectives on large text corpus, offer promise in Natural Language Processing (NLP). However, they have limitations such as rely on susceptibility to superficial cues, vulnerability to misprimed probes, often failing in reasoning tasks within Natural Language Understanding (NLU) due to the lack of external context and generating unlogical and insensible text within the Natural Language Generation (NLG) field.

NLU is a subpart of NLP which enables machines to understand and comprehend textual data, facilitating tasks such as text classification, relation extraction, named entity recognition (NER), and dialogue system. NLG enables machines to generate human-readable language texts, thereby improving text generation performance.

However, by pre-training on large text corpora and incorporating external knowledge resources, KE-PLMs can solve these problems, enabling machines to understand and comprehend textual data within NLU field as well as enable machines to generate human-readable language texts and improve text generation performance within NLG field.

Knowledge Graph Enabled Question Answering for Language Models

There are challenges associated with deploying Large Language Models (LLMs), such as hallucinations, erroneous statements, loss of interpretability, and factual accuracy. To address these challenges, the integration of LLMs with knowledge base is proposed, with a focus on utilising Knowledge Graphs to enhance LLMs’ abilities. Traditionally, dense neural retrievers have been used to link LLMs to knowledge base, but they struggle with capturing complex relational dynamics. To overcome this limitation, two approaches are suggested: Chain-of-thought prompting and incorporation of KGs. While the former offers benefits, the latter is the primary focus of the article, as Knowledge Graphs are known for their symbolic reasoning capabilities and ability to continuously evolve with new knowledge and can be specialised for domain-specific requirements.

A lightweight and effective approach called AutoKG for automated Knowledge Graph generation is proposed to facilitate integration of LLMs and Knowledge Graphs. AutoKG eliminates the need for training or fine-tuning neural networks and instead utilises pre-trained LLMs as nodes to extract keywords. Graph Laplace learning is employed to evaluate edge weights between these extracted keywords, facilitating the creation of a Knowledge Graph. Additionally, a hybrid search strategy and prompt engineering are utilised to empower LLMs to effectively utilise information from generated Knowledge Graphs.

Knowledge graphs provide structured representations of knowledge extracted from text, enabling efficient querying and retrieval of relevant information. In the context of question answering, Knowledge Graphs serve as essential sources of context and domain-specific knowledge. Converse, an approach which utilises Knowledge Graphs for capturing information to solve the open-domain question answering can be used. Converse transforms natural language text into Knowledge Graphs and queries them to retrieve relevant information for answering user queries. This approach draws Knowledge Graphs directly from the body of text instead of using fixed ontology thereby allowing the system to function on the natural language domain with no training.

Integration of Knowledge-Enhanced Models (KE-PLMs) and Knowledge Graphs

The integration of KE-PLMs and Knowledge Graphs offers promise in enhancing open-domain conversational question answering systems. Combining strengths of both approaches, we can improve the accuracy, relevance, informative and domain coverage of QA systems. Potential synergies between KE-PLMs and knowledge graphs, such as utilising KE-PLMs to enhance the understanding of text and utilising Knowledge Graphs to provide structured context for QA.

Future Advancements

There are several areas for future research and improvement:

• Application/Domain-specific Knowledge: Explore the integration of domain-specific knowledge, such as sports, business and entertainment to enhance their performance in diverse fields.
• Temporal Knowledge Graphs: Investigate effective methods to integrate temporal knowledge into LLMs, considering the dynamic nature of information over time.
• Integrating Complex Knowledge: Develop more effective methods to integrate complex knowledge into LLMs, striking a balance between performance and complexity.
• Optimising Computational Burden:
• Time Overhead: Minimise the workload while maintaining strong performance. Reduce computational cost during construction.
• Space Overhead: Address concerns about the additional space consumption. Explore techniques such as model compression to improve time and space efficiency.
• Noise Resilient: Investigate methods to mitigate noise, ensuring robustness to noise sources and maintaining performance on downstream tasks.
• Upper Bound Study: Explore the potential of LLMs with massive parameters and knowledge incorporation from various sources.

Future Directions

There are several potential future research directions:

• Integrating Knowledge from different type of sources: This can effectively increase the knowledge coverage, so that models can generate more reliable answers.
• Exploring Multi-modal Knowledge: Multi-model including text, images, videos, audio etc. This can further improve their understanding of input queries.
• Providing Interpretability Evidence: Giving an explicit reasoning path will help improve model interpretability and make more rational predictions.
• Learning Knowledge in a Continuous Way: Incorporating knowledge continuously is a promising direction, which will effectively improve PLMs’ universality and solve the forgetting problem while incorporating more knowledge.
• Increasing the Variety of Results Generated: Generate alternative outputs or predict all possible results for the real situation.

Conclusion

In conclusion, while Knowledge-Enhanced Models (KE-PLMs) focus on natural language processing (NLP) tasks, Knowledge Graphs serve as knowledge bases for querying and reasoning about structured knowledge. The combination of these two approaches holds great potential for advancing open-domain conversational question answering systems by improving the accuracy, relevance, informativeness, and domain coverage of generated responses.

References

• Chen, B., & Bertozzi, A. L. (2023). AutoKG: Efficient Automated Knowledge Graph Generation for Language Models (Version 1). https://doi.org/10.48550/arXiv.2311.14740
• Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., & Li, J. (2022). A Survey of Knowledge Enhanced Pre-trained Language Models (Version 4). arXiv. https://doi.org/10.48550/arXiv.2211.05994
• Oduro-Afriyie, J., & Jamil, H. (2023). Knowledge Graph Enabled Open-Domain Conversational Question Answering. In Flexible Query Answering Systems (pp. 63–76). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-42935-4_6
• Sun, K., Xu, Y. E., Zha, H., Liu, Y., & Dong, X. L. (2023). Head-to-Tail: How Knowledgeable are Large Language Models (LLM)? A.K.A. Will LLMs Replace Knowledge Graphs? (Version 1). arXiv. https://doi.org/10.48550/arXiv.2308.10168
• Wei, X., Wang, S., Zhang, D., Bhatia, P., & Arnold, A. (2021). Knowledge Enhanced Pre Trained Language Models: A Comprehensive Survey (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2110.08455