Optimizing Confidence Scoring in RAG-Based LLM Chatbots for Technical Support Services: A Prompt Engineering Approach

The global shortage of skilled personnel in technical support has spurred significant interest in AI-driven solutions, particularly Large Language Model (LLM)-based customer service chatbots. However, a critical challenge in deploying these systems lies in addressing AI hallucination, wherein models generate responses that are plausible yet factually incorrect. This study investigates a prompt engineering approach to enhance confidence estimation and mitigate AI hallucinations in LLM chatbots. Three distinct prompt strategies - Basic, Advanced, and Combo prompts-are systematically evaluated to improve response reliability. Given that LLMs inherently lack the ability to explicitly express uncertainty (e.g., by stating 'I don't know'), a structured confidence scoring mechanism is employed to refine accuracy and reduce the Expected Calibration Error (ECE). Experimental results reveal that Basic prompts achieve an accuracy of 69.33% (ECE: 23.33), Advanced prompts improve accuracy to 75.33% (ECE: 14.87), and Combo prompts further enhance accuracy to 81.33% while reducing ECE to 8.4. These findings underscore the efficacy of prompt engineering in mitigating AI hallucinations and advancing the performance of LLM chatbots in real-world customer support applications.