MEASURING RUNS CREATED: THE VALUE ADDED APPROACH

The following appeared in The 1987 Bill James Baseball Abstract.

— Gary R. Skoog

INTRODUCTION

One of the major interests of baseball research has long been the attempt to measure how many of a team's runs are created by each player. This article discusses how runs created can be measured from event data (that is, looking at each event in the context of play) rather than from cumulative data, as has been necessary in the past when only category totals were available. For all regulars from the 1986 season, we compute both proposed and older measures of runs created. This paper expands on results reported by the author at the 1986 SABR convention and presented to an American Statistical Association meeting.

With only a season's totals available for each player ("aggregate data") the subject of the proper attribution of runs created continues to receive refinement and controversy. Two methods, Bill James's runs created and Pete Palmer's linear weights, have defined the present state of the art. Both methods attempt to construct an index that measures runs created by each player using aggregate data. Both men have attempted to design runs created methods that are not situation dependent, as are runs scored and RBI counts, but the act of scoring runs itself is situation dependent, and so its removal per force creates a measure that varies from the ideal: We want a statistic that doesn't penalize a player for batting in fewer run producing situations than another, but at the same time rewards players who perform well in those situations. Our methodology cuts this Gordian knot by directly measuring the object of interest, the improvement or deterioration in the run expectation of the player's team at the moment of his contribution.

Given the precise event data, our first statistic, RC1 (read, runs created, version 1; marginal runs created) is appropriate for many comparative purposes in the same way that marginal cost is the appropriate cost measure in economics. Like Pete Palmer's linear weights, it is essentially mean-corrected, so that zero denotes average performance, and players are measured relative to the average. A second variant, RC2 (read, runs created, version 2) is presented, which is more descriptive in that it is generally non-negative and adds to the team's actual runs scored when aggregated over a season. As such, it is comparable to the James runs created, which will be referred to below as RCJ. Indeed, 94% of the variation in RC2 is explained by RCJ in the American League 1986 data, and 91% in the National League.

Although we don't emphasize it here, our approach unifies the sabermetric study at the micro, or event, level. It opens up a potentially more powerful and precise approach to the assessment of runs allowed by pitchers, runs created or lost on the bases, runs cost by errors in the field. Or even runs lost by bad coaching decisions or umpires mistakes. Of course, for starting pitchers ERA does approximately the same thing, but the biases of this statistic for relief pitchers are purged with the value added approach.

VALUE-ADDED

When a batter steps up to the plate, there may be 0, 1, or 2 outs, and any one of 8 runner on base situations. Thus there are 24 initial game states, abstracting from other characteristics such as the score, which teams are playing, who the players on base are, etc. When he finishes his turn at bat, he will have put his team in any one of 25 possible states (the extra state is "3 outs" and for our purposes here, no loss of generality is incurred in ignoring the configuration of the men left on base at the end of an inning). Let us denote the beginning and ending states by "s" and "t," respectively. For each state, from his team's and the league's data, we may accurately measure the distribution of runs scored in an inning, conditional on a team being in that state. Denote the means of these random variables (technically, stopping times on a specially constructed sigma algebra) by E(s) and E(t) and let R denote the runs scored during this transition Then this at bat produced R + E(t) - E(s)=actual runs scored on at bat plus expected team's runs in inning after player bats minus expected total team's runs in inning before player batted. There are refinements, some of which will be discussed below, but this is the basic idea.

In words, the RC1 in a plate appearance is positive to the extent that the batter advanced his team's cause more than an average amount, and similarly for negative contributions. It is measured in units of runs. For the general manager confronted with a - 30 run player, this statistic tells him how many runs his team would improve if he could bring this position up to the league average. The extra runs then could be converted into extra wins by Pythagorean theory.

The transitions as a team bats through an inning must, as is shown below, sum to the actual number of runs scored, minus the expectation of the state which leads off each inning of .454 (see Table 1). This is due to the telescoping nature of the sum, and the fact that there is an absorbing state, "3 outs," to which almost all innings converge An example below will make this clear. The exceptions are games won in the bottom of the last inning, and games suspended in the middle of an inning and not resumed.

Since, if a hitter does not increase the out count, his contribution must be positive (we haven't yet discussed errors) there can be at most 3 negative contributions in an inning.

We might prefer that the decrements of .454 be redistributed among the batters in the inning, so that the runs created becomes a total measure, calibrated so as to give the actual number of runs scored. We call this kind of total measure RC2, and briefly consider ways of doing this.

To fix ideas, consider an inning in which the leadoff hitter homers, and the next 3 batters make outs. Using Table 1, RC1 gives the measures l 000, -.205, -.154, and -.095, summing to 1 -.454=.546. Suggestions to redistribute the .454 and get exactly one run produced include 3 philosophies:

1. Add .454/4 to all batters appearing in the inning.
2. Add .454/3 to those 3 batters who increased the out count, making obvious modifications for double and triple plays.
3. Add total runs scored in league/total plate appearances to each batter. For the 1986 AL this was 10449/ 86XS2=.120308 and 8096/74006=.1093965 in the NL in 1986.

An advantage of l and 3 is that they yield the same differential contributions as RC1. The leadoff home run in the example above left the team on average 1.205 ahead of where they would have been with an out, which uses up .205 run when it is the first out. This argument has much appeal. A drawback is that it gives the leadoff man more than one run created for his home run, which after all does return the team to the beginning state but with an extra run-all of which should yield precisely one run created. Another advantage is that runs created equals runs scored in every (half) inning, so a fortiori for every game for every team-season, and for the league-the various levels of aggregation. Note that the entries of Table 1 are estimated from an entire season, and so are average in this sense. (We have not preserved the distinction between population averages, the E(s), and their sampled counterparts-a reader sophisticated enough to look for the difference will not be confused.)

The drawback above suggests 2, which redistributes the decrements among those players most likely to have negative runs created. It maintains one run for the solo home run, and implicitly suggests a non-negativity of runs created per plate appearance desideratum: Since runs are negative, why not extend this same property to runs created? This method does more to move the negatives toward zero than 1, although it can't totally succeed, without causing further difficulties. To see this, note that to bring all batters to non-negative numbers, we'd have to overcompensate by adding .205 times 3=.615, and we'd have to take .615 - .454=.161 off the home run-and this for a scheme which awarded .205-.095=.110 of a run for making the final out! Another objection is that outs are already taken into consideration by RC1, so an adjustment based on them would result in "double count

ing." This method shares the advantage of having the runs balance out over every half-inning.

Both submethods 1 and 2 divide .454 explicitly; instead, we could use 3 above and take the total plate appearances divided into the total runs for a league season and add this to each at bat; this would give correct runs created on average, although inning totals wouldn't necessarily be correct. The argument is, there is unnecessary noise introduced by requiring them to add, along with a mixing of the level of aggregation. This is the method used below in the RC2 calculation. The author is not adamant in its use, however, and encourages discussion on this point in the sabermetric community before the next edition of this book.

From Palmer's simulations reported in The Hidden Game, we report his table giving the E(s) entries for the 24 states:

Table 1
Expected Future Runs In An Inning,
Conditional On The State

We have added the Project Scoresheet notation for the states. The idea of using these states, incidentally, goes back at least to the fundamental 1963 paper in Operations Research, "An Investigation of Strategies in Baseball," by George Lindsey, and is implicit in the work of anyone having done serious study in any branch of science. The RC measures proposed here are similar in spirit to the Mills's "player win average," although the measures address quite different questions.

Rather than simulate, we will in the future estimate these expectations from the 2106 x 80 - 168,480 or so such situations which arise over a major league season. There will be some statistical subtley here, for we are doing inference on realizations of a Markov chain with no ergodic events and with obvious statistical dependences. Variances, rather than our estimates themselves- means-will be affected by the fact that the same inning, say, with a leadoff home run, will have the 0000 or "a" state occurring at least twice, followed by the same events for the rest of the inning entering into the sample. In theory, one could estimate a Markov half inning transition matrix and derive estimates for the entries in Table 1. This method has two drawbacks. First, the standard errors are very complicated functionals of the model parameters. Worse, model specification error would enter, and would be avoidable with the direct, nonparametric approach suggested above. The parameters of the transition matrix nevertheless are of independent interest, however, and will be estimated for various subsets of the data.

The measures of runs created reported below use Table 1. We will sometimes refer to a state not by its letter but by four numbers, as the 0000 above. The first is O. 1, or 2 and gives the outs; the next 3 are 0 or 1, depending on whether the base is unoccupied or not.

We do expect to see league differences in our estimated versions of Table 1, since pitchers bat in the National League. Consequently the relevant sample size will be smaller by roughly half. In fact, RCI for National League pitchers have been computed (but not reported below) and are uniformly negative, as expected.

DETAILED EXAMPLE OF THE CALCULATION

A runner is on first, nobody out. The batter singles, the runner on first stopping at second. The third batter follows with an RBI single, leaving runners at first and second. The next batter grounds into a 6-4-3 double play, the runner advancing to third. A strikeout ends the inning.

The official statistics give the second batter a hit only. He didn't score the run or bat it in, yet he was as instrumental in manufacturing the run as the players who received the RBI or run scored. The value added approach (refer to Table 1 above) gives him 1.380 - .783=.597 runs. The leadoff leadoff hitter gets .783 - .454 - .329 and the third hitter gets 1 run, since the runners ended up at first and second, the same state he found them in. The double play gave the fourth batsman 382 - 1.380=- 998, and the strikeout stranding the runner on third was - .382 The team earned 1.926 runs and lost 1.380 runs, giving a total of .546 above the initial state or league average of .454.

If the total decrements of .454 are added by redistributing them among the batters in the inning, we get an RC2 measure of exactly 1.

FURTHER DEVELOPMENTS: BATTING, RUNNING, AND FIELDING

For each transition, we know whether the batter's turn at bat terminated or not. In Project Scoresheet these are referred to as batting events and non-batting events, respectively. If the leadoff batter walks and steals second, (the latter is a non-batting event), then the second batter's initial states is 0010 - 0 outs, man on second, not the 0100 - 0 outs, man on first-that prevailed when he came to the plate. The man who stole second earned 1.285 - .783=.402 of a run (RC1) for his stolen base, and baserunning runs created may be kept as a separate category in this way. Similarly, errors create runs for the opposition, and may be accounted for by introducing a fictitious state of errorless play between the events involving the error. Another example will make this clear.

Say the leadoff man reaches on an error. Just as in batting average calculation, we may act from the batter's perspective as though he had been put out. The fictitious state here is 1000 - I out no one on. Now the transition 0000 to 1000, worth .249 - .454=-.205 is awarded the batter, and the transition 1000 to 0100 worth .783 - 249=.534 gives the runs created by the error. If the next 3 batters strike out, the team run potential is again reduced to 0, and their RC1 decrement is .783. Thus, the team has an RC1 total of -.205 + -.783=.988; they were given .534 of a run by the opposition, bringing us back to the familiar .454. Since they scored no runs, to get an RC2 to equal zero, there were in effect 4 "outs" inflicting negative runs created, and the "gift" of the error might be redistributed along with the .454. Errors are not so treated in the results given below, although further refinements may incorporate them.

Observe that the 1000 to 0100 transition causes outs to decrease, and so is impossible according to baseball rules. Nevertheless, there is nothing stopping our evaluating this contrafactual state transition, and indeed there is a necessity to do this to properly evaluate the error.

Present Project Scoresheet data structures, and doubtless others as well, will make this decomposition difficult for some errors, notably errors allowing runners to advance on a play. Errors allowing the batter to reach are more adequately represented. Unfortunately, we need in both cases a set of heuristics to guess the result of errorless play. Here as in many areas, theory runs ahead of practice.

ELIMINATION OF SITUATION DEPENDENCE

Besides measuring precisely and directly our objective, the value added approach has a reasonable chance at correcting for "situation dependence.'' Several factors point to this conclusion, although ultimately a minor refinement may still be in order.

A player who bats with many men on base will have high E(s) values for leaving lots of men on base to subtract from the high R and E(t) values he earns. In the example. the three singles were worth .329, .597, and 1 run and not equal amounts, reflecting the obvious fact that run production is situation dependent. The batter who hit with 2 men on base also had most to lose by not producing, as the next paragraph shows.

To see the way the value added approach corrects for situation dependence while properly acknowledging it. consider a player batting with the bases loaded and 2 out. A walk credits him with an entire run, whereas a leadoff walk in an inning is only worth .299. But had he struck out with the bases loaded and 2 out, he would have cost his team .798 - 0 (expected runs after 3 outs!)=.798 of a run, whereas a leadoff strikeout costs .454-propitious situations will amass high totals of the traditional count data (runs and RBI) but these should have subtracted from them many runs destroyed from his failures.

At a higher level of sophistication, consider a hitter, say Wade Boggs (our 1986 AL RC leader), batting in the highest E(s) state, 0111, from which 2.254 runs are expected, and the lowest state, 2000, from which .095 runs are expected. We can take Boggs' season totals and make educated guesses as to the transition probabilities from these states to any other states. This would let us compute conditional runs created from each state, for both an individual player and the league average. Then, for there to be bias for Boggs, two things must be present. First, there must be variation in the conditional runs created across the states, which the paragraph above argues (but does not prove) will be minimal. Second, Boggs must find himself with a distribution of at bats among the 24 states that is significantly different from the league averages. This may happen for pinch hitters, and to a lesser extent for leadoff hitters, who start off the game in the same state. It is an empirical question how large these discrepancies are, if any. If found significant, a further correction to RC2 is in order.

REMARKS

1. A sacrifice fly is always a fly, usually an out, but hardly ever a sacrifice, and not an official at bat. Conventional treatment thus seems dubious. In our scheme, it is properly evaluated, since its effect is the same as any other occurrence which changes the state in the same way. Clearly the concept of "state" is intended to be a statistically "sufficient" description, capturing all and only what is essential for analysis. For some purposes, mentioned below, it may be advisable to add other information such as the score, but that is not necessary for the issue at hand.
2. A ground out accomplishing the same thing as a sacrifice bunt is here given the same credit, unlike in the official statistics. A sacrifice bunt effecting 0100 to 1010 is worth -.084, explaining why some managers use it so selectively. Since it does create an out, it would get a net positive value after an RC2 redistribution.
3. A three-run home run should be worth less than three runs to the batter, as the runners have some likelihood of being driven in by a subsequent hitter. Our state change adjusts for this. The double counting here has bedeviled other methods.
4. Pitching, especially relief pitching, may be analyzed with the obvious use of the value added method. However, since the game is so often on the line, one may prefer a score dependent version in which we evaluate not expected runs in the inning but the probability of winning the game in place of E(s) and E(t)-the player win average.
5. Runner speed isn't properly adjusted yet: if a single sends a runner to third, the credit goes to the hitter and not the runner. With more (judgmental) data, this second order effect could be corrected.
6. Intentional walks are arguably not given special treatment. One place where this is clearly aberrant is in tie games in the bottom of the ninth inning or later, when the man being walked "means nothing." Then the run distribution is truncated, and from a different population than that used to estimate runs created. This is likely a third order correction, or higher.

EMPIRICAL RESULTS

The Tables below give (mean corrected) RC1, (total, positive) RC2 and the technical version of Bill James's runs created, listed under RCJ.

While we leave extensive comparison to another time, a few points may be made. First, our measure does not give "runs created or destroyed attempting to steal," which Bill's runs created method does allow for. A further refinement of RC1 and RC2 on this issue is obviously appropriate. This explains our understatements for Coleman. Henderson and Wilson. Second, the high percentage of explained variations of RC2 by RCJ-94% in the American League, 91% in the National League-have been noted. Third, the names of Boggs and Mattingly atop the AL and Schmidt and Raines atop the NL according to both methods is expected and reassuring. Finally, the diminution of agreement as one progresses toward lower RC2 and RCJ totals reminds us that RCJ was constructed on the basis of team aggregate data. Forcing it to apply to regular player totals-a sample of 600 or 700 plate appearances- is one thing; applying it to smaller totals requires its extrapolation outside the region in which it was fit. Statistical models always show such "out of sample" deterioration.*

*Editor's note: The runs created formula-technical version works with very small data samples, as is shown by the fact that it works well with games, and with very large ones such as leagues. I strongly suspect that the failure of agreement at low levels of plate appearances occurs because the failures of both methods are most apparent in small data sets where long-term randomizing factors have not acted to disguise them.

CONCLUSION

It is not surprising that a different sabermetric approach to runs created emerges when methodologies from statistics (regression, expectation, state space Markov chain framework) and economics (value-added, marginal and average) are combined with a vastly superior database, as has been made available by Project Scoresheet. As is always true in science, the new builds on the old, and will in turn be refined. It is hoped that the new methodology introduced here will be developed and incorporated into mainstream sabermetric analysis.

Excerpted from 1987 Bill James Baseball Abstract. Ballantine Books. 1987. James, Bill. "Measuring Runs Created: The Value Added Approach".

Back to the top of page | Tom Ruane's VAR Essay | BTF Essays Page | BTF Homepage | BaseballStuff.com