The generalization and error detection in LLM-based Text-to-SQL systems

Abstract

Text-to-SQL systems streamline human-database interactions, improvingdata retrieval and decision-making. Although large languagemodels (LLMs) can now generate SQL code, challenges withgeneralization and uncontrolled generation hinder their use in production.Text-to-SQL tasks are particularly sensitive to distributionshifts, where performance declines with unfamiliar database elementsor novel queries. Effective systems must maintain quality,measured in terms of generalization (correct processing of noveluser requests) and error detection (identification of incorrect generations).This study empirically assesses LLM-based Text-to-SQLsystems limitations, defining reliable production scenarios. Currentcontributions include a cross-lingual generalization research,study on generative model generalization abilities and the qualityof selective classification for error detection risk under differentdistribution shifts in task of Text-to-SQL.