ru
ru
Source
AISTATS
DATE OF PUBLICATION
05/27/2022
Authors
Tatyana Shavrina Alena Fenogenova Alexander Panchenko Daryna Dementieva Varvara Logacheva Irina Nikishina Irina Krotova
Share

A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification

Abstract

It is often difficult to reliably evaluate models which generate text. Among them, text style transfer is a particularly difficult to evaluate, because its success depends on a number of parameters.We conduct an evaluation of a large number of models on a detoxification task. We explore the relations between the manual and automatic metrics and find that there is only weak correlation between them, which is dependent on the type of model which generated text. Automatic metrics tend to be less reliable for better-performing models. However, our findings suggest that, ChrF and BertScore metrics can be used as a proxy for human evaluation of text detoxification to some extent.

Full text

Similar publications

ACL
A System for Answering Simple Questions in Multiple Languages
Anton Razzhigaev, Mikhail Salnikov, Valentin Malykh, Pavel Braslavski, Alexander Panchenko
SOURCE
ACL
Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov
SOURCE
ACL
Industry vs Academia: Running a Course on Transformers in Two Setups
Irina Nikishina, Maria Tikhonova, Viktoriia Chekalina, Alexey Zaytsev , Artem Vazhentsev, Alexander Panchenko
SOURCE
ACL
MERA: A Comprehensive LLM Evaluation in Russian
Alena Fenogenova, Artem Chervyakov, Nikita Martynov, Anastasia Kozlova, Maria Tikhonova, Albina Akhmetgareeva, Anton Emelyanov, Denis Shevelev, Pavel Lebedev, Leonid Sinev, Katerina Kolomeytseva, Daniil Moskovskiy, Elizaveta Goncharova, Nikita Savushkin, Polina Mikhailova, Anastasia Minaeva, Denis Dimitrov, Alexander Panchenko, Sergei Markov
SOURCE
ACL
Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing
Dmitry Ustalov, Alexander Panchenko, Yanjun Gao, Irina Nikishina, Arti Ramesh, Andrey Sakhovskiy, Elena Tutubalina, Gerald Penn, Gerald Valentino, Ricardo Usbeck
SOURCE
ACL
Super donors and super recipients: Studying cross-lingual transfer between high-resource and low-resource languages
Vitaly Protasov, Elisei Stakovskii, Ekaterina Voloshina, Tatyana Shavrina, Alexander Panchenko
SOURCE
ACL
TaxoLLaMA: WordNet-based Model for Solving Multiple Lexical Semantic Tasks
Viktor Moskvoretskii, Ekaterina Neminova, Alina Lobanova, Alexander Panchenko, Irina Nikishina
SOURCE
Contact us
Join AIRI
You can ask us a question or suggest a joint project in the field of AI

Message sent.

Thank you!

Something went wrong. Try again