Using generative AI for reading question creation based on PIRLS 2011 framework

This study examined the use of ChatGPT 3.5 for reading questions generation based on four processes of comprehension of PIRLS 2011 assessment framework. Using an instrumental case study approach and a usability testing method, we employed an input story text to assess ChatGPT 3.5’s effectiveness. A total of twenty questions were generated and evaluated using specific criteria. We analyzed the content of the obtained questions and the quality of revised questions after further adjustment instructions. Findings reveal that ChatGPT 3.5 excels at generating factual questions, leveraging explicit details from text, and demonstrates improvement in interpret-and-integrate questions when detailed instructions are provided. However, it shows significant limitations with higher-order questions requiring inference and evaluation, where contextual accuracy and deeper comprehension are critical. Statistical analysis revealed significant differences between question types (p = 0.004). Thematic analysis highlighted recurring challenges, such as content misalignment, oversimplification, and difficulty processing complex or nuanced user requirements. Tips for users on potential issues and ethical concerns were also discussed. Despite limitations, ChatGPT 3.5 remains a valuable tool for educators and students to enhance question creation productivity. These findings contribute to understanding generative AI’s role in education and provide actionable insights for improving question generation efficiency.