Bias Mitigation for Capturing Potentially Illegal Hate Speech

In research, broad offensive phenomena are often analyzed and subcategorized, which include more general forms of hate speech. For example, Ruppenhofer et al. [18] consider subcategories of offensive language as Profanity, Insult and Abuse. The focus in their work lies on the linguistic content of messages, thus, their definition is substantiated by an annotated dataset of example messages. This is a common research practice where, in general, the task of hate speech detection is considered as a short text classification task for a concrete dataset. Methods are then to be developed on such data that predict whether an unseen, new message contains hate speech or not, and in particular which subcategory of hate speech a given short text can be ascribed to. A survey on general approaches for hate speech detection is given by Schmidt and Wiegand [20]. Recent approaches are mostly based on models using the Transformer architecture [22]. For example, the pre-trained language model BERT [8] has proven to show promising results when being applied to hate speech detection [13, 16, 17, 23].

The detection of hate speech or of similar phenomena has been extensively explored in several shared tasks, e.g. in the following: EVALITA 2018 Hate Speech Detection Task [3], GermEval 2018 Shared Task on the Identification of Offensive Language [24], SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) [27], SemEval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter [1], GermEval Task 2, 2019 Shared Task on the Identification of Offensive Language [21], HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages [14], HASOC Subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages [15]. The usual approach is to focus on the binary case, i.e. to distinguish hate speech from acceptable content. Therefore, the definition of hate speech has to cover a wide range of variants of the phenomenon and remains rather vague. In work by Mandl et al. [15] the annotation almost entirely relies on the subjective assessments of the annotators who are only provided with a very vague conceptual definition of hate speech. This, however, results in categories which are often very difficult to interpret, which in turn makes the differences of the performance of detection systems on such data difficult to interpret. Thus, the explainability of a systems’ performance is limited and a qualitative evaluation usually does not reveal clear patterns.

In order to reduce the vagueness of hate speech definitions, we proposed in previous work [19] an approach to define more well-founded subcategories of illegal hate speech and to develop a dataset on the basis of applicable German laws. At the same time in the project DeTox [7], Demus et al. also consider the criminal relevance of comments amongst other categories. In the present paper, I focus on the process of creating our German dataset for illegal hate speech as well as on different bias avoidance steps we took into account.

The description of hate speech in the EU code of conduct mentioned above already points at certain groups of people who can be the target of hate speech. These groups are often explicitly mentioned in the messages and can be detected on a lexical level as shown by ElSherief et al. [9]. Based on how data have been sampled, different biases can be observed in the resulting dataset, such as topic bias or identity term bias [25]. Researchers have investigated components of this issue, for example racial bias [6]. We followed different techniques to avoid bias or at least to measure it in our dataset for illegal hate speech as presented in the following.

The remainder of this paper is structured as follows. In Sect. 2, a dataset for illegal hate speech is presented for which we gathered examples and annotated them on the basis of guidelines developed in earlier work [19]. In this section, I also discuss the applied bias avoidance techniques and give results from the annotation of illegal hate speech categories and identity term mentions. Subsequently, Sect. 3 reports on experiments to learn the annotated categorization using a Transformer-based neural network model. Sect. 4 discusses the results of the experiments and serves to asses the value of the developed dataset. I conclude in Sect. 5 with prospects for future endeavors to capture illegal hate speech online.

Empirically collecting hate speech data is not a straightforward process, due to certain peculiarities of the phenomenon. If one simply collected a random sample of social media data, only a very small proportion of the messages would actually contain hate speech. The annotation would be unfeasible as the effort would be too high to produce a considerable number of positive examples. Instead, there are different approaches to automatically filter data first. Here, it is crucial to choose the method carefully in order not to introduce too much bias in the data.

Wiegand et al. [24] gather data by using seed accounts of users who have been observed to post hate speech. They widen their search given these accounts and manage to find more examples of hate speech with sufficient precision. A desired advantage of this approach is that it does not impose limitations on the lexical content which is collected. However, for our data we decided not to follow this method as it is difficult to avoid an author bias here and to avoid getting stuck in echo chambers of specific user groups when collecting data.

Instead, we find examples using a set of keywords³. We use Twitter as a source due to the easy accessibility of their API and the popularity of their platform. Social media platforms such as Twitter do filter for illegal content themselves. Nevertheless, we assume that we find a sufficient number of relevant cases, for the following reasons: When querying the API, we explicitly include recently published posts which presumably have not been checked by the platform. While some of the instances considered for our data might since have been reported by other users and have been deleted due to being hate speech, they are still in our corpus. Additionally, we also consider less severe cases for our categories which show patterns similar to illegal examples. However, these cases are possibly not excluded by the stricter filter which has been used by the platform.

A keyword-based collection method naturally has the effect that collected messages always contain at least one of the keywords and thus are biased towards these keywords. In order to mitigate this issue, we include words from several different categories (in total 82 keywords) to get some variety in the results. We choose words which can also be used in neutral contexts like certain swear words, e.g. pig (“Schwein”); words referring to frequently mentioned groups, e.g. police (“Polizei”) or certain action verbs, e.g. hit (“schlagen”). We investigated the results and found that there is an intended bias towards hate speech in the data – we have a higher precision than random sampling. However, the results still contain a substantial amount of neutral messages. Table 1 shows the top keywords which are mentioned most frequently in instances in the iHS dataset with the percentage of the respective instances which are annotated to contain illegal hate speech. For all of these keywords the resulting instances are found to contain illegal hate speech in substantially fewer than 50% of the cases. Thus, a detection system cannot necessarily infer from the presence of a keyword that a message contains hate speech.

Another form of data bias can be a specific topic of the messages. This can occur, for example, when data is collected in a short time period after popular public events. In such cases many social media posts will discuss the popular topic, thus resulting in a dataset being biased towards it. To avoid this issue, we run queries in random intervals over a larger time period from 05/05/2021 to 11/19/2021. We assume this method covers a wide set of topics in posts by different user types. We did not conduct a comprehensive analysis of the topics discussed in our data, however, after screening a sample of the data, we are convinced that there is no strong topic bias. In total, we collected 274,347 Twitter messages using our keyword queries. We randomly sampled instances from the collected messages and annotated 1,385 cases. The resulting iHS dataset comprises all these annotated instances.

The text messages contained in the iHS dataset are annotated with six different labels: illegal hate speech categories (four subclasses) vs. offensive language vs. a neutral class.

The annotation guidelines for illegal hate speech [19] were developed on the basis of the abovementioned six laws, but the annotation categories distinguish only four types of illegal hate speech (A, V, N, B). Here, § 241 StGB Threatening commission of serious criminal offense (“Bedrohung”) was disregarded as in public Twitter messages it is an extremely rare phenomenon and we did not manage to gather a substantial amount of examples where this law might be applicable. Additionally, we combined § 186 StGB Malicious gossip (“üble Nachrede”) and § 187 StGB Defamation (“Verleumdung”) into a single category (N) as an assignment to one of these cannot be decided based on the linguistic content only. The label A marks cases related to § 111 StGB Public incitement to commit offenses (“Öffentliche Aufforderung zu Straftaten”), Label V marks cases related to § 130 StGB Incitement of masses (“Volksverhetzung”) and the label B marks cases related to § 185 StGB Insult (“Beleidigung”).

We differentiate illegal hate speech from (untargeted) offensive language (OL) and neutral content (Other). While hate speech itself often contains offensive words, we exclude cases where the offense is not targeted towards a certain individual or group of persons and annotate such cases as offensive language. We believe that it is important to distinguish such cases since this difference is crucial for assessing whether a message is illegal or not. Furthermore, we annotate offenses merely directed at non-human things or swearing as OL, to differentiate this category from neutral messages. Social media platform operators might consider such cases of offensive language as unacceptable content.

We performed two early annotation experiments to refine the guidelines [19]. A first annotation experiment employed 39 students as annotators which had no prior experience with research on hate speech. However, the quality of the resulting annotation is insufficient as the annotators did not strictly follow the guidelines (average Fleiss’ kappa [10] of approximately 0.38, considered as fair agreement according to Landis and Koch [12]). In a second experiment with five annotators (experts of research on hate speech) on a dataset of 100 Tweets, we achieved a moderate inter-annotator agreement (Fleiss’ kappa [10] of 0.50, interpretation according to Landis and Koch [12]). The highest Cohen’s kappa [5] of 0.59 was achieved between the two annotators with the most experience with the guidelines (the authors of the guidelines). We present further details of these experiments in our previous report [19].

After discussions of deviating cases and of the annotators’ feedback from using the guidelines, we revised the annotation guidelines to be more precise. As a result, we also developed a decision support schema for annotation (see Fig. 1). This schema is modeled as a decision tree and meant to serve as a guiding tool during annotation. An annotator should first decide whether a specific message contains any offensive content, and second, if the offense is targeted at an individual or a group. These two steps cover the necessary decisions to distinguish the labels Other and OL from illegal hate speech. Subsequently, if the instance does contain a targeted offense, the annotator should evaluate other features which are significant for the four illegal hate speech categories to conclude the annotation of the given message.

Given the refined annotation guidelines with detailed label descriptions and example cases, we assume that even a layperson in law can be trained to annotate cases of the mentioned concept of illegal hate speech. However, the actual illegality of individual cases can only be interpreted by lawyers taking all factors from contexts into account. Therefore, we limit our analysis to an annotation of cases which can potentially be illegal based on similarities of patterns of the linguistic content.

We annotated the iHS dataset using the described schema (one trained annotator). The resulting class distribution for the different labels is shown in Table 2. Out of the total 1,385 instances, almost two thirds (850 instances) belong to the neutral Other class. These are counter-examples and neither contain hate speech nor offensive language. Approximately 16% (224 instances) of the messages are to be considered as offensive language and 22% (311 instances) are annotated to fall into one of the four illegal hate speech categories. However, the distribution of the four illegal hate speech categories A, V, N and B, is imbalanced. The labels A and V are annotated only 16 and 11 times, respectively, whereas the labels N and B are considerably more frequent as they are annotated 106 and 178 times. The labels in Table 2 are not sorted by their frequency in the data but rather in decreasing order by our estimation of the severity of the respective offense. We can thus conclude that the more severe cases of illegal hate speech are more rarely to be found in social media.

In addition to the annotation of the hate speech labels, identity term mentions are also annotated for each message. We consider identity term mentions often to be separable from expressions which constitute hate speech in a message. The decision whether a message is to be considered as hate speech or not should be in general made independently of the specific target (group) identity. An incitement of masses should not change its status as being hate speech if the mentioned, e.g. religious, group is replaced by another one.

For annotating identity terms in our data, no specific guideline document has been created. Instead, we discussed a possible definition of what counts as an identity term. We deliberately decided against annotating only a predefined set of entities as we intend to include all terms that mention any identities. We empirically gather identity terms on data and only then intend to group them into categories. In recent publications, the focus often lies on selected groups of identity terms. For example, Davidson et al. [6] investigate racial bias by considering the categories black and white. A larger set of different identity terms consisting of 24 discrete classes is annotated in the Civil Comments dataset [2]. However, a general analysis of identity term mentions in such data is limited by definition as they disregard all other identity term mentions which do not fall into the predefined categories. In contrast, we annotate strings from the messages directly and do not rely on a specific set of identity term categories, which allows for a general analysis.

In this annotation, we only consider expressions which are used to specify the identity of an individual or a group of people. We do not consider named entities referring to non-human objects or abstract things, e.g. names of geographical places or branded products. Here, however, the distinction is in some cases not trivial. Names of geographical places can sometimes be mentioned to refer to the people living or working in that area, in which case we would consider the item as an identity term. The following question is essential for the annotation: Is an expression used to address a message to a specific individual or a group by mentioning a specific identity of that individual or group?

In order to demonstrate the annotation procedure, two example messages are shown in Table 3. Consider the first example for which a total of five identity term mentions are annotated. @‑marked user name mentions are always annotated automatically as they unambiguously refer to the identities of specific Twitter users. The term “Frauen” (women) denotes a specific gender identity, the term “Islam” denotes a specific religion and the term “Lehrerin” (teacher) denotes a specific professional group. It may perhaps be surprising that we consider specific occupation groups as identity terms since they do not refer to a specifically protected group; however, during annotation we also ask the question whether a mentioned group could possibly be a target of hate speech. If this is the case, a bias towards such group could appear in the hate speech data. Equally, we also consider, for example, the group of COVID-vaccinated people as an identity term as they can be targeted by hate speech and we want to avoid bias towards such a group when training systems.

The second example given in Table 3 contains a sub-word identity term “rechts” (right-wing) which is annotated as part of the word “rechts-radikal” (radical right-wing). Here, “radikal” (radical) is part of the hate speech expression, however, the direction of the political conviction “rechts” (right-wing) is a personal identity. Thus, we only annotate the sub-word character sequence which refers to the specific identity.

Using these criteria, identity term mentions were identified in each message in our dataset by one annotator. So far, we did not conduct an agreement study for this annotation.

Strings which refer to identity terms are identified as character sequences from each message and then listed for each message in the XML corpus. In the iHS dataset, there are 3,217 identity term mentions annotated, i.e. approximately 2.3 identity terms per message on average. 1,910 of the annotated identity term mentions are automatically identified @‑marked Twitter username mentions. The remaining 1,307 annotations are manually identified strings from the messages referring to identities of individuals or groups.

The average number of identity terms for messages with different labels in the corpus is displayed in Table 4. Here, the number substantially varies for the different labels from 1.3 identity terms per message for label A to 4.0 for label N. As expected, messages with labels that do not necessarily require a mentioned target person or group contain a lower number of identity terms – i.e. label A which is based on § 111 StGB Public incitement to commit offenses (“Öffentliche Aufforderung zu Straftaten”) and label OL which marks (untargeted) offensive language.

Table 5 shows all identity terms in the iHS dataset which are mentioned in more than ten messages. For each term, we additionally display the proportion of cases which are annotated belonging to any one of the four categories of illegal hate speech.

Here, a keyword bias as a result of our data acquisition method (see Sect. 2.1) can be observed. We find that seven different identity terms were also used as keywords for data sampling: “behindert” (disabled), “Flüchtling” (refugee), “Frauen” (women), “homosexuell” (homosexual), “Nazi”, “Männer” (men) and “schwul” (gay). In total, these constitute for 157 identity term mentions out of the 1,307 manually identified strings. For future work, we suggest to not include identity terms in keyword-based sampling strategies.

Besides, we observe that several frequent identity terms refer to politicians (“@ABaerbock”) or political parties (“AfD”, “CDU”, “@Die_Gruenen”). Furthermore, terms referring to sexual, gender or national identities are also mentioned frequently. Many identified strings are specific Twitter usernames or abbreviations where it is not obvious which identity category they could belong to. Thus, for some identity terms we tried to deduce the full name and included it as an attribute for each identified identity term mention in the XML corpus. We have recently started to assign a category to some identity terms (e.g. politician), however, this categorization is still at an early stage.