ezPM Compared with RAPM: Part I

An APBR forum member (back2newbelf) has recently started publishing regularized adjusted +/- (RAPM) data. It's like +/- data, but on steroids. If you want to learn more about how RAPM works, in general, see the paper presented by Joe Sill at the MIT Sloan Sports Analytics Conference in March, 2010. I thought it would be useful at this point in the development of the ezPM model to compare 1yr and 3yr averages with back2newbelf's RAPM data. What follows are the results for regression of total RAPM on total ezPM100 (both metrics are per 100 possessions), with some tables of best/worst players by average of the two metrics, to give some idea of the actual numbers. In a subsequent post, I will perform the same type of analysis on the offensive and defensive components of the metrics.

1 Yr RAPM

The 1yr RAPM data set can be found here. I used a 1000 possession minimum as my cutoff, which left about 200 players to compare. Here are the results in graphical form, followed by regression data from R:

1 yr. RAPM as a function of ezPM100.
Call:lm(formula = RAPM ~ EZPM, data = data.1yr)
Residuals:    Min      1Q  Median      3Q     Max
 -3.2659 -0.8885  0.0009  0.8501  4.3308 
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.20883    0.09066   2.303   0.0223 *  
EZPM         0.27176    0.03348   8.118 4.34e-14 ***
---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
 Residual standard error: 1.284 on 205 degrees of freedom
Multiple R-squared: 0.2433,	Adjusted R-squared: 0.2396 
F-statistic: 65.91 on 1 and 205 DF,  p-value: 4.335e-14

A few things to note here. 1) The regression result is highly significant (p=4.34e-14); 2) The slope of the regression is 0.27, which means that RAPM underestimates ezPM100, or put the other way, ezPM100 overestimates RAPM; and 3) ezPM100 explains about 24% of the variance (R^2=0.24).

Let's look at some of the best and worst players according to the 1-yr data by averaging the two metrics:

Top 20 Players by 1 Yr. Metric Average (ezPM100+RAPM)/2

RANK NAME RAPM EZPM AVG
1 LeBron James 3.3 7.37 5.34
2 Manu Ginobili 4.1 5.41 4.76
3 Chris Paul 2.7 6.63 4.67
4 Dwight Howard 1.7 7.48 4.59
5 Pau Gasol 2.9 6.04 4.47
6 Dwyane Wade 2.3 6.45 4.38
7 Paul Pierce 3.4 5.20 4.30
8 Steve Nash 2.5 5.67 4.09
9 Dirk Nowitzki 5.2 2.43 3.82
10 Kevin Garnett 4.0 3.45 3.73
11 Kobe Bryant 0.5 6.22 3.36
12 Tyson Chandler 3.4 3.25 3.33
13 Nene Hilario 2.4 4.12 3.26
14 George Hill 3.5 2.87 3.19
15 Ronnie Brewer 0.9 5.43 3.17
16 Kevin Durant 1.7 4.30 3.00
17 Brandon Bass 3.2 2.75 2.98
18 Lamar Odom 1.7 4.11 2.91
19 Rajon Rondo 2.2 3.45 2.83
20 Al Horford 1.1 4.51 2.81

Bottom 20 Players

RANK NAME RAPM EZPM AVG
207 J.J. Hickson -2.8 -4.99 -3.90
206 Andrea Bargnani -0.9 -5.89 -3.40
205 Goran Dragic -3.6 -3.13 -3.37
204 Eric Bledsoe -2.5 -3.57 -3.04
203 Sonny Weems -1.4 -4.47 -2.94
202 Jordan Hill -2.9 -2.93 -2.92
201 DeMarcus Cousins -2.1 -3.71 -2.91
200 John Wall -1.8 -4.00 -2.90
199 Travis Outlaw -2.8 -2.97 -2.89
198 Dante Cunningham -0.9 -4.80 -2.85
197 Spencer Hawes -0.8 -4.79 -2.80
196 Steve Blake 0.6 -6.03 -2.72
195 Richard Hamilton -2.7 -2.27 -2.49
194 Charlie Villanueva -0.1 -4.77 -2.44
193 Stephen Jackson -2.1 -2.33 -2.22
192 Linas Kleiza -0.1 -4.31 -2.21
191 Jeff Green -0.4 -3.99 -2.20
190 Antawn Jamison -1.7 -2.65 -2.18
189 Michael Beasley -0.8 -3.46 -2.13
188 Darko Milicic -1.0 -3.25 -2.13

Ok, both models are clearly wrong. Did you see that block J.J. Hickson made on Griffin last night?

3 Yr RAPM

The 3 yr. RAPM data can be found here. My data set goes back to the 2008-2009 season through the first week of February. As far as I know, the RAPM data weights each year equally, so I did the same to make the comparison fair. As before, a plot followed by numbers:

 

3 yr. RAPM as a function of ezPM100.

Call:lm(formula = RAPM ~ EZPM, data = avg.3yr)
Residuals:    Min      1Q  Median      3Q     Max
 -5.8064 -1.1009  0.0435  1.1527  5.9218 
Coefficients:            
Estimate Std. Error t value Pr(>|t|)    
(Intercept)  0.58592    0.10964   5.344 1.92e-07 ***
EZPM         0.43908    0.03725  11.786  < 2e-16 ***
---Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
Residual standard error: 1.703 on 273 degrees of freedom
Multiple R-squared: 0.3372,	Adjusted R-squared: 0.3348 
F-statistic: 138.9 on 1 and 273 DF,  p-value: < 2.2e-16

As expected, there is an improvement in both the slope (0.44) and R^2 (~0.34). Here are the player tables:

Top 20 Players by 3 Yr Metric Average

RANK NAME RAPM EZPM AVG
1 LeBron James 10.6 9.8 10.2
2 Dwight Howard 6.7 10.1 8.4
3 Chris Paul 6.7 7.7 7.2
4 Kevin Garnett 7.2 4.1 5.6
5 Manu Ginobili 5.8 5.4 5.6
6 Tim Duncan 5.0 5.7 5.4
7 Steve Nash 7.4 3.3 5.3
8 Dirk Nowitzki 7.5 2.3 4.9
9 Kobe Bryant 4.7 5.0 4.8
10 Pau Gasol 4.0 5.4 4.7
11 Yao Ming 4.0 5.2 4.6
12 Chris Bosh 5.0 3.4 4.2
13 Paul Pierce 4.7 3.6 4.2
14 Greg Oden 2.7 5.6 4.2
15 Andrew Bogut 4.5 3.8 4.1
16 Lamar Odom 5.0 3.2 4.1
17 Leon Powe 1.4 6.7 4.1
18 Amir Johnson 5.2 2.9 4.1
19 Marcus Camby 3.2 4.3 3.8
20 Nene Hilario 3.6 3.9 3.8

Bottom 20 Players by 3yr Average (Or the List You Really Don't Want to Appear On)

RANK NAME RAPM EZPM AVG
458 Gerald Green -1.9 -10.2 -6.0
457 Josh Powell -5.5 -6.4 -5.9
456 Bobby Brown -3.6 -7.4 -5.5
455 Adam Morrison -2.6 -7.7 -5.2
454 Ricky Davis -4.9 -5.0 -5.0
453 Darnell Jackson -2.8 -6.8 -4.8
452 Stephon Marbury -2.6 -6.8 -4.7
451 Brian Skinner -4.5 -4.6 -4.6
450 Brian Scalabrine -1.1 -8.0 -4.5
449 Johan Petro -3.7 -5.2 -4.5
448 Jannero Pargo -0.7 -8.1 -4.4
447 Marcus Williams -3.1 -5.6 -4.3
446 Malik Allen -2.2 -6.4 -4.3
444 Timofey Mozgov -1.8 -6.5 -4.1
445 Trenton Hassell -4.5 -3.8 -4.1
443 Rob Kurz -2.2 -6.0 -4.1
442 DaJuan Summers -1.2 -7.0 -4.1
441 J.J. Hickson -5.9 -2.2 -4.1
440 Oleksiy Pecherov -0.8 -7.3 -4.0
439 Brian Cook -2.6 -5.3 -3.9

Conclusions

This was definitely a worthwhile exercise. It's good to see how the ezPM model compares to RAPM. Of course, it should not be expected that the two models line up perfectly. That would be great, but in practice, we should be using multiple models to evaluate players. Some players may look better in one metric or the other. We should have more confidence in players that are highly rated by both an APM model and a box score metric, such as ezPM. For example, what I didn't show here are the players that were ranked in the top 20 by either metric alone. That would have showed that Derek Fisher is one of the best players in the league according to RAPM 1yr data (2.4), but not according to ezPM (-4.03). Kris Humphries looks great according to ezPM (4.52), but not RAPM (-1.4). (His new girlfriend always looks great!)

Anyway, this is a good stopping point, but also a good starting point. Going forward, we'll see if there are adjustments that can be made to ezPM that will make it even more consistent with RAPM. For example, why is Dirk rated so much higher in RAPM? Does it have something to do with usage? His teammates? It's also important to ask which model is a better predictor. If one or the other (or an average) is a better predictor, we probably want to know that, right? As always, to be continued...

29 thoughts on “ezPM Compared with RAPM: Part I”

  1. I believe 12 players are in both top 20s. That seems like a fairly high amount of stability and you'd expect some change.

    I think only 1 is in both bottom 20s. New guys and guys exiting the league play a role in that.

    I wonder if the 1 yr was compared to the 3 year which component varied more- EZPM or RAPM? For the top and bottom 20s and in general?

    That knowledge may be potentially helpful in refining the blend of the EZPM and RAPM components in the tails. Regress EZPM some and / or regress RAPM less?

    Maybe there would be value to others in a blend of the 1 yr and 3 yr EZPM and RAPM components into a higher blend?

    1. "I wonder if the 1 yr was compared to the 3 year which component varied more- EZPM or RAPM? For the top and bottom 20s and in general?"

      I should do this.

  2. Hey Evan - I know that at least at one point you were working on having ezPM predict +/- scores better (maybe that's still your ultimate goal?). Given that, would you call ezPM a box score measure? Or statistical +/- for that matter? Obviously they use box score numbers, but only as a stand-in for +/- data.

    1. Statistical +/- uses regression as a sort of black box to calculate the numbers. ezPM doesn't use any regression (currently). It's purely "theoretical". I would call it a "box score" metric nominally, but really more of a play-by-play metric, since a lot of the data can't actually be found in box scores.

      DSMok1's ASPM (advanced statistical +/-) regresses advanced stats (TS%, USG%, AST%, etc) onto statistical +/- data (i.e. adjusted +/-).

      So, there are basically three ways to do it. And of course, these could all be combined, if useful.

      1. Yes a 3 way comparison or blend make sense to look at.

        Arguably coach assigned minutes might be a 4th estimate of value worth having nearby to the other 3. Maybe salary too, though it is heavily affected / biased by the CBA and market forces for most.

      2. I regress onto Adjusted Plus/Minus data; Statistical Plus/Minus is a term that should be reserved for various methods of regressing box score data onto Adjusted Plus/Minus data. Just trying to clarify the terminology!

  3. For the 12 guys in top 20 on both lists, the difference in the 3 yr component - the 1 yr component is near 0 (-0.03) for EZPM while for RAPM it is +3.3.

  4. Maybe applying age curves for each component will help narrow the gap in the 3 yr components some. Do the age curves look different between the 2 component metrics?

  5. I have explained this general idea in the past and never got anywhere with it with previous APM authors but...

    what if you gave say 1/3 of "prior" credit to players on the court based on say their 3 year EZPM

    (or just 1/4th to 1/6th and then more to one or both of the upcoming components)

    and 1/3 rd of the credit for what occurs in a play stint to players on the court based on ASPM

    and then left "just" the remaining 1/3rd of credit to be assigned by multi-season RAPM?

    Rather than wait for an after the calculation blending of metrics? Would that help? Instead of starting off being completely agnostic about who was valuable / harmful to the team +/- while on the court and giving all teammates equal input credit or blame, say instead we think we can reasonably assign 2/3rds of credit and want to try to capture the remaining 1/3rd at play stint level accurately that might lie beyond or behind the boxscore using RAPM?

    I'd appreciate your judgment on this from an informed technical / theoretical perspective and perhaps also from a philosophical perspective, if you follow the description of the idea well enough and care to comment.

  6. Note that RAPM is biased towards 0 throughout, so the slope should be significantly below 1. In fact, I would recommend looking at "subsets" of the RAPM data based on how many possessions each player had. In other words, all RAPM players with the same numbers of possessions should be similarly biased towards 0.... well, except for a few cases.

    Take for example the case of Lebron's teammates last year in Cleveland. In actuality, probably Lebron was like a +10 and most of his teammates were averaging 0. RAPM, though, biases everybody towards 0, pulling Lebron significantly downward and his teammates upward (since the team still has to total to its efficiency differential.

    Crow: one of my long-term goals is to figure out how to use ASPM as a Bayesian prior for an APM regression. Since you already have a prior, theoretically you shouldn't need to regularize... I'm not sure how to do it in practice.

    1. I look forward to seeing your efforts to implement some such effort DSMok1.

      Regularization may still have a role but it depends on how much weight the Bayesian prior gets and how well it does the job.

      My suggestion used EZPM as the prior and ASPM's regression found stat values for part of the current scoring; but maybe some sort of blend of EZPM, ASPM and maybe RAPM could be in the prior and then let the APM or RAPM do the rest on its own or still use ASPM's regression found stat values for part of the current scoring.

    2. Daniel, just curious. ASPM is already based on regressing APM, right? Wouldn't it make more sense to use a prior that is not? Or is it that you just need to be "close enough" and ASPM could get you there as well as anything else?

      Also, as I said in the post, I plan to break this down into offense and defense. I'm expecting the defensive comp to be the most interesting part of all this. What little correlation I've seen of existing box score metrics to defensive APM...well, it was little. So, ezPM might be of more value on that side. We'll see, I guess.

      1. ASPM is based on regressing Advanced Box-Score stats ONTO APM. So I'm trying to predict what APM would be based on the Advanced Box-Score metrics.

        RAPM will add up to team efficiency, as will ASPM (both by definition).

        RAPM is at its core an application of a Bayesian prior consisting of a normal distribution of mean 0.

  7. I can't immediately find a file that bears on this issue but how do the correlations and overs / under the trend line patterns of EZPM and RAPM look by position or for a backcourt / frontcourt or perimeter / interior split?

  8. It could be that the metric blends would usefully vary by position to reduce error or arguably reduce error, if you were to take that step for real world usage.

  9. Nice work!
    A couple of things:
    1. The 1 year RAPM stuff is kind of old now and more data is definitely important. You should probably do this again when the season is 75% over or later.
    2. To see which is the better predictor we could do the following. We can both compute our ratings for all games except one, |games| times, then compute error on the left out games. I'd prefer to compute the error for all lineups of the game, instead of predicting the score difference between the two teams. Maybe hit me up mid-March for this

    1. Ok, or maybe when the season is over might be better. I'm going to be busy teaching from mid-March to the end of April.

      (If I understand correctly, though, we would use Aaron B's matchup files to do this, right?)

  10. Evan:
    It might be interesting to look at the two groups of players with the largest gaps: those valued much more by ezpm, and those valued much less by ezpm. Then look at the attributes of each group of players. If one group had a lot of big rebounders, for example, it would suggest that ezpm was valuing rebounds too much or too little (or at least, much more or less than RAPM). If PGs are consistently rated higher by ezpm, that might mean your assist coefficient is too generous. And so on.

  11. It is 50% PFs & Cs, that is probably close to equal representation to the population of players. Close enough to not be able to make that the clear issue in itself. Few great assist guys or great rebound guys. Probably heavy on weakness on one or both of these 2 criteria but I haven't checked them all. Maybe those weaknesses aren't hurting as much as EZPM suggest?

    I haven't checked counterpart defense by itself or it vs. signs of help defense strength or weakness.

  12. Amir Johnson so far has just missed the top 20 this season.

    He has a 3.60 ezPM100 and a 1.90 RAPM for a 2.75 average. Just below Horford's who is currently 20th - 2.81.

    For three seasons he is currently 18th.

    Is his new contract of $30 million guaranteed over five seasons too much like all the pundits said when he signed last summer?

    This year he is making $5 million. It seems to me that he is not being overpaid.

  13. I liked him and his contract too. That he has been passed around a lot probably means something (right or wrong or some of both) but I can't know what the insiders know / see direct away from the court.

Leave a Reply