Improved Key Information Extraction Capabilities for Small Vision-Language Model

This paper presents a document image processing pipeline that enhances key information extraction (KIE) using Small Vision-Language Models (SVLMs). KIE is pivotal for enterprise document automation, and the move toward zero-/few-shot MLLMs motivates a privacy-preserving, on-premise SVLM-based approach to improve KIE efficiency and accuracy. The pipeline couples document layout analysis with OCR and QLoRA fine-tuning to segment the document images and construct keyword-guided refined composites for SmolVLM-256M-Instruct, reducing tokens and improving efficiency. On FUNSD and SROIE, refined inputs outperform original images in field-level accuracy, while an upper-bound target segmentation achieves substantially higher accuracy, highlighting the promise of dedicated KVP region detection; token counts also drop markedly. An ablation clarifies why choosing a high fuzzy-match threshold and a small number of segments is effective, providing empirical justification for design choices. Our current evaluation compares KIE performance of the proposed pipeline using SmolVLM-256M-Instruct against other SVLMs. Future work will broaden model coverage, support multi-page inputs, and output structured JSON to enable end-to-end, reproducible deployment.