Design and Evaluation of a Fully on-Premises RAG-Based Enterprise Chatbot for Technical Support

Enterprise customer and technical support is rapidly shifting toward large language model (LLM)-based assistants and agentic workflows to improve response quality, consistency, and operating efficiency. However, cloud-hosted LLM services raise concerns about data privacy, regulatory compliance, and long-term running costs. This paper presents the design and evaluation of a modular, fully on-premises RetrievalAugmented Generation (RAG) chatbot system for enterprise technical support. The system integrates an n8n workflow platform, locally hosted LLMs via Ollama, and a Qdrant vector database to enable privacy-preserving deployment without external API dependency. A multi-pipeline ingestion framework processes manuals and structured enterprise data into semantically indexed knowledge units. To improve reliability, the chatbot uses strict prompt constraints with standardized fallback behavior and an automated LLM-as-a-Judge benchmark pipeline to evaluate correctness, rejection behavior, and latency. Experiments on 260 benchmark queries achieve 89.2% overall accuracy, 100% correct rejection on out-of-domain queries, and an average end-to-end latency of 2.46 s. The results suggest that medium-scale open-source LLMs, when combined with local retrieval and workflow control, can provide a practical and privacy-preserving support solution. However, performance remains constrained by model capacity, possible evaluation bias, and deployment-scale limitations.