Houston pushed out teachers over a score they could not check

Houston hired a formula to rank its teachers

In 2017 a federal court in Houston called it a matter of first impression: "the use of privately developed algorithms to terminate public school teachers for ineffective performance."¹ The formula was a value-added model, a statistical estimate of how much a teacher added to a class's learning.² It worked by predicting how far a class should grow on the year's standardized tests, then crediting the teacher with the gap between that prediction and the students' actual growth. Grow more than predicted and the teacher scored high. Grow less and the teacher scored low.

The Houston Independent School District did not build the model. It licensed one called the Education Value-Added Assessment System, or EVAAS, developed by the statistician William Sanders and sold by the analytics company SAS.¹ From 2011 to 2015 the district ran its teachers through it. For most of those years a teacher's value-added result counted for half of the official appraisal, dropping to 30 percent in the 2014-15 school year.¹

EVAAS did not report a raw growth number. It divided each teacher's score by its own margin of error to produce a Teacher Gain Index, then sorted that index into one of five labels at fixed cutoffs.¹ The label was the district's verdict on the year.

From a class's test scores to a lost job

Houston did not use the ratings only to advise teachers. In 2012 the district added low value-added performance as a stated reason for not renewing a contract. Its accountability system defined an "ineffective" teacher as one whose EVAAS index sat at or below -2.00, at the floor of the rating scale.¹ A district goal adopted that November set out to retain no more than 15 percent of the teachers rated ineffective.¹ Because the ratings were cut at hard lines, a hundredth of a point could decide which side of one a teacher fell on. The plaintiffs' expert offered one illustration, which the court recorded. A teacher at -2.01 and a teacher at -1.99 differ by nothing meaningful, yet the two fall on opposite sides of the -2.00 line that separated an ineffective rating from an acceptable one.¹

The exits followed. Across the 2011-12, 2012-13, and 2013-14 school years, the district reported moving out 20.3 percent, 24.4 percent, and 25.0 percent of its ineffective-rated teachers.¹ The Houston Federation of Teachers, the union that would sue, counted twelve of its members whose continuing contracts were terminated for low value-added scores between 2012 and 2014, four of whom received letters naming the scores as a reason.¹ Many more teachers resigned, retired, or were reassigned rather than face a formal termination, a route the court would later treat as a due-process problem of its own.¹

The score no one could recompute

In 2014 the Houston Federation of Teachers, a union of more than 6,100 members, and nine individual teachers sued the district in federal court.¹ Their central complaint was not that the model was bad science. It was that a teacher had no way to find out whether a given score was even calculated correctly.

The reason was structural. HISD did not compute the scores. SAS did, and SAS treated the equations, source code, decision rules, and assumptions behind EVAAS as trade secrets it would not divulge "to either HISD or the teachers themselves."¹ The district admitted that it did not verify or audit the numbers it received, and it conceded that any teacher's attempt to replicate a score with the information available "will necessarily fail."¹ Its own answer to its teachers said why.

One teacher's number could not be rechecked without re-running the whole district's, so no teacher could ask for an isolated recalculation. The plaintiffs' expert, given the same limited access, could not replicate the scores either. Without SAS's proprietary information, the court wrote, EVAAS scores "will remain a mysterious 'black box,' impervious to challenge."¹

On that ground the court ruled for the teachers. It held that HISD teachers "have no meaningful way to ensure correct calculation of their EVAAS scores, and as a result are unfairly subject to mistaken deprivation of constitutionally protected property interests in their jobs."¹ A public employee with a protected interest in a job is owed a fair process before losing it, and a score that cannot be checked is not one a teacher can contest. That, and not the quality of the statistics, is what the court found wanting.

The court was careful about what it did not decide. It granted the district summary judgment on the teachers' other constitutional claims. On whether EVAAS was accurate or its ratings stable, it declined to rule at all, calling that question "disputed" and pointing to other courts that had upheld value-added systems.¹ The ruling turned on verifiability and fair notice, not on a finding that the model was wrong.

The case never reached a trial. HISD had already ended its SAS contract in 2016.¹ In October 2017 the two sides settled, with the district agreeing to stop using value-added scores to terminate a teacher so long as the teacher could not independently verify the number, and to pay the union $237,000 in fees and costs.³⁴

Why an unseeable estimate cannot be appealed

Two features of this kind of scoring made the failure predictable. The first is that a value-added score is an estimate, and a noisy one. It does not measure a teacher's contribution directly. It infers that contribution from the gap between predicted and actual test-score growth, using data that swings from year to year and a classroom full of influences the model never sees. The American Statistical Association, the largest U.S. association of statisticians, issued a warning about these models in 2014. Most studies, it said, find that teachers account for only about 1 to 14 percent of the variation in test scores. The rest comes from students, families, and conditions outside any one classroom.⁵ Value-added models, it added, "typically measure correlation, not causation," and their scores carry "large standard errors" that "make rankings unstable, even under the best scenarios."⁵

The second feature is the one the court reached. An estimate carries uncertainty, and the ordinary way to handle uncertainty is to show the work so someone can check it. EVAAS could not be checked. Its computation was a trade secret, so a teacher facing a low rating could not point to the step that went wrong, because no step was visible. An estimate no one can inspect cannot be argued with. It can only be accepted.

Whether EVAAS was in fact accurate was never settled. SAS did not defend a crude tool.

The plaintiffs did not set out to disprove that. Their expert testified only that no one outside SAS could confirm it, because the equations were sealed. What would settle the disagreement, the court noted, is exactly what SAS withheld: the value-added equations, source code, decision rules, and assumptions.¹ Audrey Amrein-Beardsley, a researcher on value-added models who served as an expert for the teachers, later wrote the case up as a caution against staking employment decisions on ratings a district cannot open.⁷ The argument over the model's science stayed open. The due-process problem never depended on resolving it.

The same failure is now sorting job applicants

Hiring software now scores job applicants in much the way Houston scored its teachers. In 2019 the Electronic Privacy Information Center asked the Federal Trade Commission to investigate HireVue, a company whose software scored recorded video interviews and ranked candidates by an "opaque, proprietary algorithm."⁸ The complaint said HireVue gave candidates no access to their scores and no access to "the training data, factors, logic, or techniques" behind them. In some cases, it alleged, "even HireVue is unaware of the basis for an algorithmic assessment."⁸ By then more than 700 companies were using the product, along with public-sector clients.⁸

A person is ranked for a livelihood by a formula they cannot inspect, sold by a vendor that may not fully be able to explain it either. Facial expressions alone could account for as much as 10 to 30 percent of a HireVue score.⁸ After the complaint, in January 2021, HireVue said it would stop using facial analysis, while continuing to score candidates on their speech and other behavior.⁹ One input changed. The design that Houston's teachers ran into stayed in place.