ru
ru
Source
AISTATS
DATE OF PUBLICATION
05/27/2022
Authors
Tatyana Shavrina Alena Fenogenova Alexander Panchenko Daryna Dementieva Varvara Logacheva Irina Nikishina Irina Krotova
Share

A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification

Abstract

It is often difficult to reliably evaluate models which generate text. Among them, text style transfer is a particularly difficult to evaluate, because its success depends on a number of parameters.We conduct an evaluation of a large number of models on a detoxification task. We explore the relations between the manual and automatic metrics and find that there is only weak correlation between them, which is dependent on the type of model which generated text. Automatic metrics tend to be less reliable for better-performing models. However, our findings suggest that, ChrF and BertScore metrics can be used as a proxy for human evaluation of text detoxification to some extent.

Full text

Similar publications

TACL
mGPT: Few-Shot Learners Go Multilingual
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, Tatyana Shavrina
SOURCE
EMNLP
LM-Polygraph: Uncertainty Estimation for Language Models
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, Artem Shelmanov
SOURCE
EMNLP
Kandinsky: an Improved Text-to-Image Synthesis with Image Prior and Latent Diffusion.
Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, V. S. Akhripkin, Igor Pavlov, Ilya Ryabov, Angelina Kutz, Alexander Panchenko, , Denis Dimitrov
SOURCE
AACL-IJCNLP
Uncertainty Estimation for Debiased Models: Does Fairness Hurt Reliability?
Gleb Kuzmin, Artem Vazhentsev, Artem Shelmanov, Xudong Han, Simon Šuster, Maxim Panov, Alexander Panchenko, Timothy Baldwin
SOURCE
AACL-IJCNLP
Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxification
Daryna Dementieva, Daniil Moskovskiy, David Dale, Alexander Panchenko
SOURCE
AACL-IJCNLP
Predicting Terms in IS-A Relations with Pre-trained Transformers
Irina Nikishina, Polina Chernomorchenko, Anastasiia Demidova, Alexander Panchenko, Chris Biemann
SOURCE
WMT
Semantically-Informed Regressive Encoder Score
Vasiliy Viskov, George Kokush, Daniil Larionov, Steffen Egger, Alexander Panchenko
SOURCE
Contact us
Join AIRI
You can ask us a question or suggest a joint project in the field of AI

Message sent.

Thank you!

Something went wrong. Try again