Skip to main navigation Skip to search Skip to main content

Identifying the cause of performance issues of Pretrained Language Model for Educational Technology

Research output: Contribution to conference › Paper › peer-review

Abstract

Online self-learning platforms, such as question-answering (Q&A) websites, are popular technology for learning, and utilising Pretrained Language Models (PLMs) to maintain their content qualities is a common practice. However, it is challenging to identify the cause that affects the performance of PLMs. In this study, we propose using machine common sense to identify the cause affecting the performance of PLMs. We conducted an empirical experiment with three PLMs using a publicly available dataset. We select 45000 data points as the training data and 1000 as the testing data. We first train the PLMs with training data, then run machine common sense tests to examine their reasoning abilities. We define the causal relationship between content quality and reasoning ability, and use the cause to derive Metamorphic Relations (MR) of Metamorphic Testing (MT) for creating follow-up testing datasets. We analyse the changes between source and follow-up outputs to see whether the identified cause affects the performance. Results show that the reasonableness of the content is the cause that affects the performance of PLM, which has reasoning abilities. In addition, the proposed approach in this study is effective for identifying and validating the cause that affects the performance of PLMs, even on devices with limited computer resources. Future research can apply our approach and seek different machine common sense tests and counterfactual analysing techniques to identify different causes of performance issues of different PLMs.
Original languageEnglish
Publication statusAccepted/In press - 14 May 2025
Event11th International Symposium on Educational Technology - Bangkok, Thailand
Duration: 22 Jul 2025 → 25 Jul 2025
https://hksmic.org.hk/iset/2025/

Conference

Conference11th International Symposium on Educational Technology
Abbreviated titleISET
Country/TerritoryThailand
CityBangkok
Period22/07/25 → 25/07/25
Internet address

Fingerprint

Dive into the research topics of 'Identifying the cause of performance issues of Pretrained Language Model for Educational Technology'. Together they form a unique fingerprint.

Cite this