2024 MLB Regular Season Projection System Scoring
I compare major projection systems from the 2024 season with what really went down to see which systems were most and least accurate
As a projection salesman, I find this a bit scary. I started this post before running the analysis, and I’m committed to publishing it no matter what I find. But if it turns out that my projection system is way worse than the other systems that you can get for free... well, that would be a bit embarrassing and- not good for business!
An online promise is not worth much, I understand, but I’m promising you anyway. No funny business is involved to make my projections look better. You’ll just have to take my word for it!
Only one way to find out! The projection systems I’m reviewing:
JA (mine)
Steamer
ATC
Bat X
Hitters
Categories I checked
Runs
Homers
Runs Batted In
Steals
Batting Average
On Base Percentage
Slugging Percentage
How I did it
Filter to every batter who got at least 450 ABs this year. I want to try to avoid punishing projection systems for injuries, so I use 450 ABs to only use guys that played mostly full season to grade with.
For each player, take each category and find the difference between actual and projection for each system (doing a standard subtraction and also doing an absolute value subtraction).
Look at the average errors for each category and each system.
Runs
Here’s an example just so you can see how these scores work. Looking at Anthony Volpe for runs:
He scored 90 runs. The closest projection was The Bat X, who had him at 76.5 runs scored. So, I took that “AbsDiff” column for each system and averaged every player out.
The results:
JA 13.17
The Bat X 13.53
ATC 13.81
Steamer 15.36
Hot Diggity Dog, look at that; JA takes a win right away! Some guys my system nailed:
Zach Neto (70 projected runs, 70 actual runs)
Lourdes Gurriel (72 projected, 73 actual)
Taylor Ward (73 projected, 72 actual)
The guys I missed big on:
Jurickson Profar: 45 projected, 94 actual
Matt Vierling: 40 projected, 80 actual
Yandy Diaz: 95 projected, 55 actual
Willi Castro: 89 50 projected, 89 actual
Masyn Winn: 48 projected, 85 actual
I’m not going to do that for every category, but those names would repeat themselves if we did. A lot of the misses will have to do with projected playing time and lineup spots. Few of us foresaw Masyn Winn playing every day and eventually leading off for the Cardinals. And we all projected Profar as a part-time player who was a mediocre hitter, and then he turned out to be one of the best hitters in the league, somehow.
Anyways, I get a win.
RBIs
JA 13.74
The Bat X 14.44
ATC 14.46
Steamer 14.98
Two for twooooo! And I won this one by a lot. Let’s go!
Homers
The Bat X 5.20
ATC 5.24
JA 5.42
Steamer 5.62
The Bat X takes this one, and I fall to third. Derek Carty is the best in the business, so it’s no surprise to see his projections at the top.
Steals
JA 5.32
ATC 5.35
The Bat X 5.56
Steamer 5.60
I’m feeling really good right now, but we’re about to get humbled with the average stats.
Batting Average
The Bat X .0165
ATC .0168
Steamer .0170
JA .0197
This is a bad loss for me; I’m well behind the rest. That’s true with the final two as well:
On Base Percentage
ATC .0196
The Bat X .0196
Steamer .0207
JA .0210
Slugging
The Bat X .0382
ATC .0406
Steamer .0429
JA .0451
If I do myself a favor and just take the standard five categories (this is my main focus in the lead-up to drafts anyways!) and find the average rank, I tie The Bat X:
So I feel pretty good about that. The biggest takeaway here might be that Steamer is pretty bad compared to the other systems.
My action point is to tweak how I’m projecting the triple-slash line of AVG/OBP/SLG. There’s a lot of room for improvement there.
Pitching
Looking at these categories:
ERA
WHIP
Wins
Saves
Strikeouts
K%
ERA
ERA might be the toughest of all of the categories to project. So these scores are pretty rough for everybody.
ATC .536
Steamer .591
JA .595
The Bat .704
We had some multiple-run misses on guys like Reynaldo Lopez, Tobias Meyers, Ronel Blanco, Hunter Greene, and Chris Sale. Each projection system were way too high with the ERA projection on those guys. The other way around, everybody was much too optimistic about guys like Miles Mikolas and Griffin Canning. So at least I wasn’t alone on Canning, although my projection system’s 4.06 projected ERA was the lowest. I’ll take third place here.
WHIP
ATC .109
JA .121
Steamer .126
The Bat .127
Domination from ATC in the two main pitching ratios. Nice job. Again, I’m happy to just not be in the basement here.
Wins
This was filtered to only pitchers with at least 25 starts.
Steamer 2.81
ATC 3.02
JA 3.03
The Bat 3.04
Saves
This was filtered to pitchers with at least 5 saves to get rid of all the exactly correct zero-save projections.
JA 8.46
The Bat 8.57
Steamer 8.59
ATC 8.59
Strikeouts
Back to starters who pitched close to a full season (25+ starts)
JA 26.4
Steamer 31.5
ATC 35.3
The Bat 39.5
K%
JA 2.21%
ATC 2.22%
Steamer 2.49%
The Bat 2.50%
BB%
ATC 1.37%
The bat 1.39%
JA 1.39%
Steamer 1.41%
Depending on how filtered (between including relievers or not…), these results could change. But with the way I happened to do it (and it was not manipulated, I once again promise), I take the crown here:
I’m not going to go out bragging about that, because there are a bunch of different ways you could do this - and I’m sure I didn’t do it in the most proper way.
What we see pretty clearly here is that Carty isn’t too good at pitcher projections. I’m sure those will improve once he finally gets the statcast-powered pitching projections done. I think he’s still using the old system and hasn’t integrated new data like I have.
In summary:
My projections are at least to be considered competitive with the big boys, and that makes me feel good.
If not using mine, I’d recommend The Bat X for hitter projections and ATC for pitcher projections next year.
My 2025 projections will come out in late December or early January for paid members of this SubStack. Hopefully, I’ll be able to improve it a bit with what I found here.
Well done, congrats on a very good year of projections. Love the integrity before you even knew the results. BTW, I'm enjoying your football write-ups as well.