ru
ru
Source
ACL
DATE OF PUBLICATION
05/21/2022
Authors
Artem Shelmanov Mikhail Burtsev Kirill Fedyanin Maxim Panov Manvel Avetisian Leonid Zhukov Alexander Panchenko Artem Vazhentsev Gleb Kuzmin Akim Tsvigun Evgenii Tsymbalov Gleb Gusev
Share

Uncertainty Estimation of Transformer Predictions for Misclassification Detection

Uncertainty estimation, Transformers, Mahalanobis distance, Dropout, Determinantal point process

Abstract

Uncertainty estimation (UE) of model predictions is a crucial step for a variety of tasks such as active learning, misclassification / adversarial attack / out-of-distribution detection, etc. Most of the works on modeling the uncertainty of deep neural networks evaluate these methods on image classification tasks. Little attention has been paid to UE in natural language processing. To fill this gap, we perform a vast empirical investigation of state-of-the-art UE methods for Transformer models on misclassification detection in named entity recognition and text classification tasks and propose two computationally efficient modifications, one of which improves the state of the art and outperforms computationally intensive methods.

Full text

Similar publications

ACL
A System for Answering Simple Questions in Multiple Languages
Anton Razzhigaev, Mikhail Salnikov, Valentin Malykh, Pavel Braslavski, Alexander Panchenko
SOURCE
Neural Computation
Associative Learning and Active Inference
Petr Anokhin, Artyom Sorokin, Mikhail Burtsev, Karl Friston
SOURCE
EMNLP
xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics
Daniil Larionov, Mikhail Seleznyov, Vasiliy Viskov, Alexander Panchenko, Steffen Eger
SOURCE
EMNLP
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
SOURCE
EMNLP
LLMs to Replace Crowdsourcing For Parallel Data Creation: The Case of Text Detoxification
Daniil Moskovskiy, Sergey Pletenev, Alexander Panchenko
SOURCE
ACL
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
SOURCE
ACL
Industry vs Academia: Running a Course on Transformers in Two Setups
Irina Nikishina, Maria Tikhonova, Viktoriia Chekalina, Alexey Zaytsev , Artem Vazhentsev, Alexander Panchenko
SOURCE
Contact us
Join AIRI
You can ask us a question or suggest a joint project in the field of AI

Message sent.

Thank you!

Something went wrong. Try again