Text Detoxification using Large Pre-trained Neural Models

Published in The 2021 Conference on Empirical Methods in Natural Language Processing, 2021

We present two novel unsupervised methodsfor eliminating toxicity in text. Our firstmethod combines two recent ideas: (1) guidance of the generation process with small style-conditional language models and (2) use of paraphrasing models to perform style transfer. We use a well-performing paraphraser guidedby style-trained language models to keep thetext content and remove toxicity. Our second method uses BERT to replace toxic words with their non-offensive synonyms. We make the method more flexible by enabling BERT toreplace mask tokens with a variable number of words. Finally, we present the first large-scale comparative study of style transfer models on the task of toxicity removal. We compare our models with a number of methods for style transfer. The models are evaluated in a reference-free way using a combination of unsupervised style transfer metrics. Both methods we suggest yield new SOTA results.

Download here

Methods for Detoxification of Texts for the Russian Language

Published in Multimodal Technologies and Interaction, 5(9), 2021, 2021

We introduce the first study of the automatic detoxification of Russian texts to combat offensive language. This kind of textual style transfer can be used for processing toxic content on social media or for eliminating toxicity in automatically generated texts. While much work has been done for the English language in this field, there are no works on detoxification for the Russian language. We suggest two types of models—an approach based on BERT architecture that performs local corrections and a supervised approach based on a pretrained GPT-2 language model. We compare these methods with several baselines. In addition, we provide the training datasets and describe the evaluation setup and metrics for automatic and manual evaluation. The results show that the tested approaches can be successfully used for detoxification, although there is room for improvement.

Download here

Crowdsourcing of Parallel Corpora: the Case of Style Transfer for Detoxification

Published in Proceedings of the 2nd Crowd Science Workshop: Trust, Ethics, and Excellence in Crowdsourced Data Management at Scale co-located with 47th International Conference on Very Large Data Bases (VLDB), 2021, 2021

One of the ways to fighting toxicity online is to automatically rewrite toxic messages. This is a sequence-to-sequence task, and the easiest way of solving it is to train an encoder-decoder model on a set ofparallel sentences (pairs of sentences with the same meaning, where one is offensive and the other isnot). However, such data does not exist, making researchers resort to non-parallel corpora. We close thisgap by suggesting a crowdsourcing scenario for creating a parallel dataset of detoxifying paraphrases.In our first experiments, we collect paraphrases for 1,200 toxic sentences. We describe and analyse thecrowdsourcing setup and the resulting corpus.

Download here

Cross-lingual Evidence Improves Monolingual Fake News Detection

Published in Proceedings of the ACL-IJCNLP 2021 Student Research Workshop, ACL 2021, 2021

Misleading information spreads on the Internet at an incredible speed, which can lead to irreparable consequences in some cases. Therefore, it is becoming essential to develop fake news detection technologies. While substantial work has been done in this direction, one of the limitations of the current approaches is that these models are focused only on one language and do not use multilingual information. In this work, we propose a new technique based on cross-lingual evidence (CE) that can be used for fake news detection and improve existing approaches. The hypothesis of the usage of cross-lingual evidence as a feature for fake news detection is confirmed, firstly, by manual experiment based on a set of known true and fake news. Besides, we compared our fake news classification system based on the proposed feature with several strong baselines on two multi-domain datasets of general-topic news and one newly fake COVID-19 news dataset showing that combining cross-lingual evidence with strong baselines such as RoBERTa yields significant improvements in fake news detection.

Download here

SkoltechNLP at SemEval-2020 Task 11: Exploring Unsupervised Text Augmentation for Propaganda Detection

Published in International Workshop on Semantic Evaluation, 2020

This paper presents a solution for the Span Identification (SI) task in the “Detection of Propaganda Techniques in News Articles” competition at SemEval-2020. The goal of the SI task is to identify specific fragments of each article which contain the use of at least one propaganda technique. This is a binary sequence tagging task. We tested several approaches finally selecting a fine-tuned BERT model as our baseline model. Our main contribution is an investigation of several unsupervised data augmentation techniques based on distributional semantics expanding the original small training dataset as applied to this BERT-based sequence tagger. We explore various expansion strategies and show that they can substantially shift the balance between precision and recall, while maintaining comparable levels of the F1 score.

Fake News Detection using Multilingual Evidence

Published in IEEE 7th International Conference on Data Science and Advanced Analytics, 2020

Nowadays, misleading information spreads over the internet at an incredible speed, which can lead to irreparable consequences. As a result, it is becoming more and more essential to combat fake news, especially in the early stages of its origins. Over the past years, a lot of work has been done in this direction. However, all existed solutions have their limitations. One of the main limitations of the current approaches is that the majority of the models are focused only on one language and do not use any multilingual information. In this work, we investigate the new approach of fake news detection based on multilingual evidence. We show effectiveness of the proposed approach in a manual and an automated evaluation experiments. Paper presentation