Go straight to the Projection Update!
Go to the previous projection evaluation update
It's July 23rd and it's time to look at how the various projections are doing. As you may or may not remember, at the beginning of June, I thought it would be kind of fun to see how my projections had done so far, compared to some of the more high profile names out there. I cautioned that it wass just June, and lots could happen to the stats of various players.
I also decided to post some more projections of players who have, during the season, managed to obtain a position where they are currently getting some playing time. This happens usually when someone gets injured, but sometimes they just win the job. On the same page, I've also finally added the projections for the hitters lucky enough to play their home games in Coors. Yes, I now have the projections up for all of the Colorado regulars. These projections do not use any of the current season performance (minors or majors) as a basis. They appear the same as if they were posted before the season started. They can all be found here.
I'll also add that I won't use any of these projections when evaluating how well the projections have done (neither now, nor at the end of the season). Obviously if they're not up by the time the season is underway, it's hard to fairly include them in the analysis.
As of games of Julu 22nd, I decided I would take a quick look at how my projections are doing compared to others. I'm going to do a quick comparison of mine, with those of Stats, Inc. (as published in their yearly handbook), CBS Sportsline, Diamond Mind Baseball (A computer simulation game) and Baseball Prospectus. In this update, I've now added the projections from John Mosey's Site. I am simply going to compare OPS at this point (though when the season's over, I'll get much more in depth and maybe use other sources as well).
What I did was keep track of the projected OPS for each player from each source, and use the least squares method to obtain a correlation coefficient between the projected OPS for each source and the actual current OPS. (For all of you who don't speak math: I looked at what was projected and used a valid statistical technique to compare it with what has happened). I have decided that from here until the end of the season, I'll only do the comparison for the top 170 players in At Bats plus walks. This will make it easier to compare how much the projections improve as the sample increases. The results:
| Source | Corr. Coeff. | Average Error |
|---|---|---|
| Vörös | 0.707 | 0.079 |
| Stats Inc. | 0.697 | 0.082 |
| CBS Sportsline | 0.598 | 0.085 |
| Diamond Mind Baseball | 0.652 | 0.083 |
| Baseball Prospectus | 0.693 | 0.079 |
| John Mosey | 0.645 | 0.081 |
The first thing I'd like to state is that I think all of these are pretty good, and better than I'd thought the systems would do. All of the systems have designs on reaching an r of .70 or greater, and mine has reached that already. Now, I'll state very clearly here that between all of the various systems, those aren't very big differences (they shouldn't be as the projections were all reasonably close to each other). I'll also state that CBS Sportsline's projections pretty clearly look to be just simple estimations based on fairly recent performance without taking into account minor league numbers. That fact makes it pretty difficult for their's to compete (and frankly I don't think they much care about their projections anyway).
But I will say that even at this early stage, I'm very pleased. I know both Stats, Inc., Baseball Prospectus, DMB and John all put a whole bunch of effort into their projections, and I was pretty sure that for this first year they'd spank mine pretty hard. That still could very well happen, but for the time being I'm extremely happy to simply be holding my own with them.
It's worth noting that the average projected OPS of the group for each source was .811 (me), .802 (Stats), .818 (CBS), .815 (DMB), .815 (BP) and .839 (Mosey). Whereas the actual average OPS of the group is .846. Credit goes to John for pegging the level of offense better than the rest. I'm not sure whether this is due to increased offense this year, or shaping, or whatever. If I had to guess, I'd say it's partially increased offense, and partially that there's a minor bias due to the fact that hitting well early on probably increases playing time.
As the year goes on, I'll check back here and make more comments. Thanks for reading along.
Vörös McCracken