ThePollsters How A Data Processing Error Changed Our Deluxe Forecast

Not open for further replies.


2022 Election

How A Data Processing Error Changed Our Deluxe Forecast​

By FiveThirtyEight

Nov. 18, 2022, at 2:11 PM

Soon after the midterm elections, we began our regular process of evaluating how FiveThirtyEight’s forecasts performed. We quickly discovered an error: We were using out-of-date data for one important source used in the Deluxe version of our forecast. Although this had little impact on the topline numbers for each party’s chance of controlling a chamber of Congress, it had modest-to-medium-sized effects on some individual races in the Deluxe forecast. It had no effect on the Lite or Classic forecasts.

The Deluxe forecast differs from the Classic and Lite forecasts in that it accounts for race ratings published by three groups: The Cook Political Report, Sabato’s Crystal Ball and Inside Elections. After adding new Inside Elections ratings for House races in late September, we noticed what we thought was an anomaly in the forecast. To investigate, we disabled automatic updates for that site’s House ratings. We determined that the election model was running correctly, but we neglected to re-enable automatic updates from Inside Elections. As a result, Inside Elections ratings for House races were frozen in time as of late September. (To be clear, this was FiveThirtyEight’s error and there is no fault whatsoever with Inside Elections or their ratings.)

If we had run the model with the updated ratings, the final forecast would still have shown Republicans with a 84 percent chance of winning the House, the same as our final forecast with the out-of-date ratings. And Republicans would have had a 55 percent chance of winning the Senate, instead of 59 percent. (Even though Inside Elections ratings for Senate and gubernatorial races were being updated, because of the way that the model works, there were some very minor, indirect effects on Senate and gubernatorial Deluxe forecasts as well.)1

Only one individual race forecast shifted by more than one category as a result of the error (e.g., a race shifting from “lean Republican” to “lean Democrat,” skipping over “toss-up”), and a number did have a one-category shift, as listed in the table below.

Races where ratings would’ve shifted if we corrected our error​

2022 midterm races where race rating categories changed after correcting for missing data in our final preelection Deluxe model

as publishedCorrected
forecast▲▼ race▲▼ rating▲▼ Dem odds▲▼ rating▲▼ Dem odds▲▼ Diff in Dem odds▲▼
HouseVA-02Toss-up47.8%Lean R33.1%-14.7
HouseTX-15Toss-up54.1Lean R39.9-14.2
HouseIA-03Toss-up42.3Lean R28.3-13.9
HouseWA-08Lean D72.4Toss-up58.8-13.7
HouseCT-05Lean D60.7Toss-up47.3-13.5
HouseIL-17Lean D62.2Toss-up49.3-12.9
HouseOR-05Toss-up42.3Lean R29.9-12.4
HouseAZ-02Lean R34.2Likely R22.2-12.0
HouseCA-13Lean D66.6Toss-up54.8-11.8
HouseNY-17Lean D70.1Toss-up58.5-11.5
HousePA-07Toss-up43.9Lean R32.4-11.5
HouseMN-02Likely D80.0Lean D68.8-11.2
HouseCA-49Likely D81.8Lean D71.4-10.4
HouseNJ-07Lean R28.4Likely R18.2-10.2
HouseMI-07Lean D65.3Toss-up55.4-9.9
HouseNV-03Lean D61.5Toss-up51.8-9.7
HouseNY-03Lean D68.3Toss-up58.9-9.4
HouseNH-01Lean D67.0Toss-up58.2-8.8
HouseME-02Lean D66.9Toss-up59.3-7.6
HouseNY-04Likely D77.7Lean D70.5-7.2
HouseCA-47Likely D79.7Lean D72.6-7.1
HouseTX-28Likely D75.9Lean D70.3-5.6
HouseOH-09Likely D77.8Lean D72.3-5.5
HouseCA-41Solid R5.3Likely R6.0+0.7
GovernorNVLean R38.9Toss-up41.1+2.2
HouseNY-02Solid R3.6Likely R6.6+3.1
HouseAZ-01Solid R5.4Likely R10.7+5.3
HouseCA-45Likely R19.3Lean R27.4+8.1
HouseNY-01Likely R22.6Lean R31.7+9.1
HouseCA-27Lean R36.6Toss-up49.2+12.6
HouseCA-22Lean R39.1Toss-up52.7+13.5
HouseOH-01Likely R16.1Lean R29.9+13.8
HouseNM-02Likely R22.4Lean R37.2+14.7
HouseOH-13Likely R18.6Lean R33.9+15.3
HouseNC-13Likely R23.4Lean R39.1+15.8
HouseNY-22Lean R35.8Toss-up52.3+16.5
HouseMI-03Toss-up59.1Likely D77.8+18.7

Not listed in that table is the House race in Washington’s 3rd Congressional District, which did not see a change in its categorization. It was won by Democrat Marie Gluesenkamp Perez, who was listed with only a 2 percent chance in the forecast. If updated Inside Elections ratings had been used, she would have had a 4 percent chance instead. So the race was a major upset either way — although one should keep in mind that when a model issues forecasts for 435 House districts, some low-probability upsets are to be expected if the model is calibrated properly.

We are reviewing our internal processes for how to better identify errors of this nature. One lesson is that smaller errors are sometimes harder to detect than larger ones. If our forecast in a high-profile race such as Pennsylvania’s U.S. Senate election had differed dramatically from the consensus, we would quickly have investigated it. Small anomalies in a series of mostly low-profile House races are harder to detect with the “eye test,” however. We also strongly appreciate reader feedback, including alerting us to potentially anomalous forecasts. While our models are fairly complex, the forecasts should still follow logically from the inputs. If a given forecast is hard to explain, it may reflect a problem with the underlying data or with the way that we’re processing it.

In evaluating how FiveThirtyEight’s forecasts did — for example, comparing our performance against other forecasts — we would recommend that you use the original, as-published forecasts, even though they were using outdated Inside Elections ratings. We of course would have preferred to use the updated ratings, but we don’t think we should get credit for a mistake that we only identified after the fact. In conducting our own assessment of our forecast once all race calls are finalized, we will show you four versions instead of our usual three: Lite, Classic, Deluxe (as published) and Deluxe (corrected).

A complete set of files showing what our final Deluxe forecast would have shown given updated Inside Elections ratings can be found here.

FiveThirtyEight regrets the error. We appreciate the time you spend on the site, and we hope that you found our midterm elections coverage valuable despite it.

Continue reading...

Not open for further replies.

Similar threads

Nate Silver
Geoffrey Skelley
  • Locked
Politics – FiveThirtyEight