Confidence Estimation for Error Detection in Text-to-SQL Systems

Аннотация

Text-to-SQL enables users to interact with databases throughnatural language, simplifying the retrieval and synthesis ofinformation. Despite the success of large language models(LLMs) in converting natural language questions into SQLqueries, their broader adoption is limited by two main challenges:achieving robust generalization across diverse queriesand ensuring interpretative confidence in their predictions. Totackle these issues, our research investigates the integrationof selective classifiers into text-to-SQL systems. We analysethe trade-off between coverage and risk using entropy basedconfidence estimation with selective classifiers and assess itsimpact on the overall performance of text-to-SQL models.Additionally, we explore the models’ initial calibration andimprove it with calibration techniques for better model alignmentbetween confidence and accuracy. Our experimental resultsshow that encoder-decoder T5 is better calibrated thanin-context-learning GPT 4 and decoder-only Llama 3, thusthe designated external entropy-based selective classifier hasbetter performance. The study also reveal that, in terms oferror detection, selective classifier with a higher probabilitydetects errors associated with irrelevant questions rather thanincorrect query generations.