ru
ru
Source
AISTATS
DATE OF PUBLICATION
05/27/2022
Authors
Tatyana Shavrina Alena Fenogenova Alexander Panchenko Daryna Dementieva Varvara Logacheva Irina Nikishina Irina Krotova
Share

A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification

Abstract

It is often difficult to reliably evaluate models which generate text. Among them, text style transfer is a particularly difficult to evaluate, because its success depends on a number of parameters.We conduct an evaluation of a large number of models on a detoxification task. We explore the relations between the manual and automatic metrics and find that there is only weak correlation between them, which is dependent on the type of model which generated text. Automatic metrics tend to be less reliable for better-performing models. However, our findings suggest that, ChrF and BertScore metrics can be used as a proxy for human evaluation of text detoxification to some extent.

Full text

Similar publications

ACL
A System for Answering Simple Questions in Multiple Languages
Anton Razzhigaev, Mikhail Salnikov, Valentin Malykh, Pavel Braslavski, Alexander Panchenko
SOURCE
EMNLP
xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metrics
Daniil Larionov, Mikhail Seleznyov, Vasiliy Viskov, Alexander Panchenko, Steffen Eger
SOURCE
EMNLP
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
SOURCE
EMNLP
LLMs to Replace Crowdsourcing For Parallel Data Creation: The Case of Text Detoxification
Daniil Moskovskiy, Sergey Pletenev, Alexander Panchenko
SOURCE
ACL
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
SOURCE
ACL
Industry vs Academia: Running a Course on Transformers in Two Setups
Irina Nikishina, Maria Tikhonova, Viktoriia Chekalina, Alexey Zaytsev , Artem Vazhentsev, Alexander Panchenko
SOURCE
ACL
MERA: A Comprehensive LLM Evaluation in Russian
Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Anastasia Minaeva, Denis Dimitrov, Alexander Panchenko, Sergei Markov
SOURCE
Contact us
Join AIRI
You can ask us a question or suggest a joint project in the field of AI

Message sent.

Thank you!

Something went wrong. Try again