TY - GEN
T1 - Incorporating Domain Knowledge Repository into Large Language Models for Question Answering of Construction Laws and Regulations in China
AU - Zhou, Shenghua
AU - Wang, Hongyu
AU - Cheng, Xiaohan
AU - Li, Dezhi
AU - Chen, Zhengyi
AU - Wei, Ran
AU - Thomas Ng, S.
N1 - Publisher Copyright:
© ASCE.
PY - 2024
Y1 - 2024
N2 - The smart question-answering of construction laws and regulations (QAoCLR) could significantly reduce the CLR query cost. Most existing QAoCLR studies rely heavily on conventional small-scale deep learning models (e.g., BERT), resulting in inadequate language understanding and subpar QA performance. Despite large language models (LLMs) showing impressive QA potential in general fields, they remain ill-performed in professional areas (e.g., CLR) due to the lack of domain-specific knowledge incorporation. Hence, our study integrates a domain knowledge repository with LLMs for QAoCLR. It involves (1) creating a repository of 275 filtered laws and regulations, (2) developing a QAoCLR-oriented test data set of 1960 multiple-choice questions from the Chinese Registered Constructor Examinations, (3) retrieving relevant knowledge vectors and incorporating them into LLMs (i.e., GPT-3.5, GPT-4.0, textdavinci-003, and ChatGLM2-6B), and (4) comparing the performance of LLMs with and without domain knowledge incorporation. The results show that LLMs with domain knowledge exceed original LLMs in QAoCLR accuracy by an average of 27.95%. Specifically, there is a 21.25% improvement in single-answer questions and a 44.62% enhancement in multi-answer questions. This work reveals the effectiveness and necessity of domain knowledge incorporation into LLMs for QAoCLR.
AB - The smart question-answering of construction laws and regulations (QAoCLR) could significantly reduce the CLR query cost. Most existing QAoCLR studies rely heavily on conventional small-scale deep learning models (e.g., BERT), resulting in inadequate language understanding and subpar QA performance. Despite large language models (LLMs) showing impressive QA potential in general fields, they remain ill-performed in professional areas (e.g., CLR) due to the lack of domain-specific knowledge incorporation. Hence, our study integrates a domain knowledge repository with LLMs for QAoCLR. It involves (1) creating a repository of 275 filtered laws and regulations, (2) developing a QAoCLR-oriented test data set of 1960 multiple-choice questions from the Chinese Registered Constructor Examinations, (3) retrieving relevant knowledge vectors and incorporating them into LLMs (i.e., GPT-3.5, GPT-4.0, textdavinci-003, and ChatGLM2-6B), and (4) comparing the performance of LLMs with and without domain knowledge incorporation. The results show that LLMs with domain knowledge exceed original LLMs in QAoCLR accuracy by an average of 27.95%. Specifically, there is a 21.25% improvement in single-answer questions and a 44.62% enhancement in multi-answer questions. This work reveals the effectiveness and necessity of domain knowledge incorporation into LLMs for QAoCLR.
UR - https://www.scopus.com/pages/publications/105024993972
U2 - 10.1061/9780784486115.054
DO - 10.1061/9780784486115.054
M3 - Conference contribution
AN - SCOPUS:105024993972
T3 - Computing in Civil Engineering 2024: Artificial Intelligence, Automation and Robotics, and Human-Centered Innovations - Selected papers from the ASCE International Conference on Computing in Civil Engineering 2024
SP - 514
EP - 521
BT - Computing in Civil Engineering 2024
A2 - Akinci, Burcu
A2 - Berges, Mario
A2 - Jazizadeh, Farrokh
A2 - Menassa, Carol C.
A2 - Yeoh, Justin
PB - American Society of Civil Engineers (ASCE)
T2 - 2024 ASCE International Conference on Computing in Civil Engineering, i3CE 2024
Y2 - 28 July 2024 through 31 July 2024
ER -