PPFA Learns From Its Mistakes (Maybe)

Since starting this website in 2015, I’ve made my fair share of predictions. Some of them have been right. Some have been wrong.

With the midterms just eight days away, another set of predictions looms. Before I lock them in, I thought I might be able to learn from past mistakes and potentially implement those lessons into this year’s predictions.

So far this cycle, I’ve made predictions at the “six months out” mark, the “one month out” mark, and the “two weeks out” mark:

These predictions aren’t final. Next week, I’ll be calling upon all my knowledge to give you the best final predictions I can. To that end, today I’ll be reflecting on my past mistakes. I want to find what I was saying at points of past cycles, including what I said on the morning of the election, and then compare those predictions to the actual results. I’ll then ponder two general questions: 1) Where I was wrong? 2) Can I learn anything from those mistakes?

So let’s take a look…

(Author’s note: although I started this website in 2015, I didn’t make any Congressional predictions in the 2016 election. In those early days, I wrote about “presidential politics” and nothing else — hence the name of the website. Only later did I branch into history and other kinds of politics. Therefore, a 2016 postmortem isn’t terribly helpful, so we’ll start in 2018.)


2018

For the 2018 Midterms, here were my predictions at several points of the cycle, followed by the actual results of the election:

Takeaways:

  • Although I correctly predicted the chamber majorities, I underestimated the margin for the winners.
  • I’m fine with being off by 12 House races. There are just so many! And I don’t take the time to learn about individual districts. (You pay for what you get here at PPFA.)
  • The Senate results hurt a bit, though. In my final Senate prediction column, I correctly predicted 32 of the 35 Senate races, including three flips: Arizona and Nevada flipping blue, North Dakota flipping red.
  • However, all three PPFA losses were predictions for Democratic holds in states where Republicans won: Florida, Indiana, and Missouri. Ouch.

Now let’s juxtapose that performance with that of…

2020

In the presidential race, I correctly predicted 49/50 states, including the weird districts in Nebraska and Maine that cut against the rest of the state! The only miss was Georgia, which, with a margin of less than a quarter of a percent, was the closest result of the election. None of this has anything to do with Congressional predictions, but it’s my website and I can write whatever I want.

As for the 2020 race for Congress…

House Takeaways:

This really ticked me off. Although I correctly predicted which party would win the House majority in both 2018 and 2020, the margins were off and in no perceivable pattern.

  • In the 2018 House prediction, I underestimated the winner’s margin. In 2020, I overestimated it.
  • That means, in 2018, I underestimated Democratic performance (slash overestimated Republican performance). In 2020, I overestimated Democratic performance (slash underestimated Republican performance).
  • In sum: there was no trend in my House errors from 2018 to 2020, and therefore there’s nothing to learn from those mistakes.

This exercise has not helped so far and I’m regretting it.

Senate takeaways:

Ah, but in the Senate, there has been a pattern across the two data points.

  • In 2018, remember, I had 32/35 Senate races correct, but all three misses were in the same direction: I picked Democrats, but Republicans won.
  • In 2020, I did even better, picking 33/35 Senate races (if we include correctly predicting that both Georgia Senate races would go to run-offs). The two I missed were my “low confidence” Democratic pickups in Maine and North Carolina. Both went to Republicans.
  • The pattern: I underestimated Republican results in the Senate both times.

This is actually kind of jarring. To have just five total Senate misses in 70 races across two elections is pretty good, but to have all five be wrong in the same direction is not so good. It is perhaps evidence of something systemically wrong with my analysis. If each prediction were a coin flip, the odds of all five going wrong in the same direction is 1/2 to the fifth power, or 1 in 32, about a 3% chance.

This mathematical improbability demanded that I look more into it, but there were no observable patterns in the cause of the misses. And trust me, I looked into it! Check out this useless Excel chart that took me like a half hour to make:

In the four columns of data on the right side of the chart, the first two columns show the Real Clear Politics polling averages for each of the races (Florida 2018, Indiana 2018, Missouri 2018, Maine 2020, North Carolina 2020) at 30 days before the election and on Election Day. The next column (Momentum) shows the difference in polling averages across that time. The last column shows the actual result of the election. Note that Maine did not have an RCP average due to a limited number of polls (which helps explain the biggest polling miss of these five races). In place of an RCP average in the 30 Days Out and Election Day columns, I listed the margin of the poll closest to 30 days before the election, and then I averaged the last two polls before the election.

Aside from the Result column — which we already know contains the problem I’m trying to solve — are there any patterns? Incumbents won twice, challengers won three times. In the bottom three races, we can say that late momentum pointed toward the Republican win despite Democrats still leading a couple of those Election Day polling averages, but in the top two races Republicans both trailed in the polls and did not have momentum in the final month.

And yet, it’s not as if we “just can’t trust the polls.” It’s polling that helps me and others successfully predict so many Senate races, to say nothing of presidential election states. (Have I mentioned getting 49/50 right in 2020? Because I got 49/50 right in 2020.) I suppose we can’t “trust the polls” sometimes, but most times we can. The trick is finding out which is which. When does the exception push back against the rule? It’s easy to see in retrospect, but not in advance.

Anyway, these misses, however inexplicable, loom large over my upcoming predictions. There’s a good chance that if I determine a state or two as a tossup, this pattern might have me hedge toward Republicans.

But I’m not locking in any predictions yet. Here’s the upcoming schedule:

  • Next up: a final deep dive into the House.
  • Then, I’ll explore the possibility that the Democrats will shock us all.
  • This weekend I’ll take my final deep dive into the Senate
  • Monday, 11/7: Back by popular acclaim: your “Election Night Viewing Guide”!
  • Tuesday, 11/8 (Election Day!): Final predictions

Five posts over the next week? You better believe it. ‘Tis the season.


Today’s featured image came from Auntie P at Flickr. I don’t know what the “P” stands for, but I like to imagine it’s Pasto.

2 thoughts on “PPFA Learns From Its Mistakes (Maybe)”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.