'Gay' filled 3% of toxic comments, so the classifier marked the word toxic

What "a less toxic model" is counting

Put two language models side by side, and the claim that one is less toxic than the other usually rests on a single measurement. Feed each model a batch of prompts, collect what it writes, and send every continuation through a classifier that returns a number between 0 and 1 for how toxic the text is. Average those numbers, and the model with the lower average wins. The classifier behind almost all of these numbers is Jigsaw's Perspective API, and the benchmark that turned it into a standard is RealToxicityPrompts, which scores model output against Perspective's TOXICITY attribute.¹

The word toxic is doing narrow, specific work here, well short of its everyday sense.

Two things about the number are easy to miss. First, it is a probability. Perspective is calibrated with isotonic regression, so its score reads as the probability that a piece of text is toxic, and RealToxicityPrompts draws the toxic line at 0.5.¹ Second, that probability is one classifier's estimate, learned from examples people labeled by hand. Everything the score can and cannot support follows from how those two facts play out down the chain.

From one comment to one score

The chain from a human judgment to a model ranking has five links. Each one replaces a person's call with a stand-in for it.

1. People judge the comments

   Perspective's English classifier is trained on comments from online forums, mainly English Wikipedia talk pages and New York Times comments, each labeled toxic or not by crowd raters.² The Wikipedia labels came from a project that had crowd workers rate talk-page comments. The classifier it produced was built to match the aggregate of three such raters, on a corpus where personal attacks run about 1% of comments.³

2. A model copies the judgments

   The shipped English model is a convolutional neural network over word embeddings. RealToxicityPrompts scored against a version of it, trained on those Wikipedia-and-news comments, with an area under the ROC curve of 0.97. That figure says how cleanly the score separates toxic comments from clean ones, where 1.0 is perfect and 0.5 is a coin flip.¹²

3. The model returns a probability

   For any new sentence, the calibrated model returns a number between 0 and 1: its estimate of the chance a reader would find the sentence toxic.¹

4. A threshold makes it yes or no

   RealToxicityPrompts counts a sentence as toxic at 0.5 and above.¹ Where to put that line is a choice, not a fact. Move it and text that was toxic becomes clean, with nothing in the text having changed.

5. An average makes it one score

   Generate many continuations for many prompts, run each through the classifier, and report the aggregate. That average is the toxicity score that ranks the models.¹

The labelers' sense of what is rude becomes the training signal, the model's fit to that signal becomes the probability, and the threshold and the average become the score. Nothing in the finished number records whose sense of rude it started from.

The average that ranks the models

RealToxicityPrompts fixed how the averaging is done. Its authors drew 100,000 sentences from the OpenWebText Corpus, a body of English web text scraped from links posted to Reddit. They sampled 25,000 from each of four equal bands of Perspective toxicity, so the set spans clean to toxic.¹ By the 0.5 threshold, about 22,000 of the prompts count as toxic and the rest do not.¹ Each sentence is split in half: the first half is the prompt, the second the continuation, and both are scored.

To rank a model, the benchmark generates 25 continuations per prompt and reports two aggregates. The first is the expected maximum toxicity: the average, across prompts, of the single most toxic of a model's 25 tries. The second is the probability that at least one of the 25 crosses 0.5. Both are defined at exactly 25 generations, and both rise as that number grows, so a score means something only next to another score measured the same way.¹

As a ranking, the differences are slight: on non-toxic prompts the four models sit within seven hundredths of one another. The more telling number is the level. A neutral prompt still gives roughly even odds of a toxic continuation by this classifier's 0.5 line.

The Perspective model is not fixed. RealToxicityPrompts scored against the convolutional network just described; the current production Perspective is a different model, a multilingual character-level transformer Jigsaw described in 2022.⁴ A score taken from one endpoint on one date does not transfer to another, which is why every number in this lesson carries the study and the year that produced it.

The penalty the average hides

On average the score tracks human judgment fairly well. When RealToxicityPrompts' authors hand-checked 100 web documents themselves, their toxic-or-not calls agreed with Perspective's about 88% of the time, a correlation of 0.83.¹ The number is not noise. That aggregate agreement is exactly what hides the failures: they concentrate on a few kinds of text, and a strong average absorbs them.

The first kind is text that names an identity. Jigsaw's own researchers found that early versions of the Wikipedia-trained classifier gave clearly non-toxic statements like "I am a gay man" unreasonably high toxicity scores, a pattern they named false positive bias.⁵ The cause was in the training data. The word gay appeared in 3% of the comments labeled toxic but only 0.5% of comments overall, so the model learned the term itself as a signal of toxicity; "queer" and "homosexual" carried the same imbalance.⁵ Tested from outside Jigsaw in 2017, before that bias was mitigated, the public demo scored "I am a gay black woman" at 87% toxic while "I am a man" ranked among the least toxic phrases.⁶

The second kind is dialect. In a 2019 study, Maarten Sap and colleagues measured how often hate-speech classifiers, and Perspective itself, flag text written in African-American English, using a method that infers the dialect from word patterns.⁷ Markers of the dialect correlated with the labels in two widely used datasets at a Pearson correlation of 0.42 and 0.35. A classifier trained on one of them called almost half of the non-offensive African-American English tweets offensive: a 46% false-positive rate, against 9% for tweets in white-aligned English.⁷ Querying Perspective directly in December 2018, the authors found its TOXICITY score correlated with the dialect at 0.31 and 0.45, and negatively with white-aligned text.⁷

The gap is not one dataset's quirk. A separate team reached the same result independently. Davidson and colleagues trained classifiers on five widely used abuse datasets and compared how often each flagged black-aligned against white-aligned tweets. Every dataset showed the gap, with black-aligned text flagged between about 1.4 and 2.7 times as often.⁸

The bias enters at the first link of the chain, in the labels. When Sap's team told annotators which dialect a tweet was written in, the annotators marked African-American English tweets as offensive noticeably less often, dropping the mean rating from 0.55 to 0.44.⁷ The judgment the classifier learned to imitate was itself sensitive to who the writer appeared to be, and step one of the pipeline turned that judgment into a number.

What a lower score buys

Put the links together and the cost of ranking models by this number is specific. A toxicity score cannot separate what a model wrote from how the classifier reads it. A model that discusses being gay, or that renders African-American speech, collects the toxicity the classifier assigns to those features, so a leaderboard that sorts models by average toxicity scores that model as worse for doing it.⁵⁷ The same pressure runs the other way in training. A team lowering its model's toxicity score is optimizing against the classifier. Wherever the classifier is wrong, the cheapest way to score lower is to steer the model away from the text it overrates, which is text about identity.

Perspective's own documentation warns against this use. It states the tool is not intended for fully automated moderation, and that it detects toxicity in a statement, not anything about the person who wrote it.² Averaging its scores to rank or tune models is the automated, decontextualized use its owner cautions against.