How Sausage Is Made: Using Statistics to Compare Players Across Eras

Editor's note: once again, back is the incredible Brian Tung, guest author and friend of SS&R. As always, we're delighted to share his original and provocative mind-bombs with our readers. Enjoy! ~DF

A week or so ago, those kind folks at Hoopspeak, Ethan and Beckley, wrote a post comparing MJ and Kobe, and immediately the hackles of Lakers Nation were raised. (If you were wondering what your hackles are, they're those hairs on the back of your neck that stand up on end whenever someone writes a post comparing MJ and Kobe.) As you can probably guess from the fact that I'm writing this, they reached the conclusion that MJ was better than Kobe. Those amongst us cynical enough to cast the first stone might contend that they even reached that conclusion before writing the post. The horror!

Except that, of course, that conclusion is not really all that earthshaking. Nor was their approach: for novelty, they decided to approach the comparison statistically. This is me being stunned.

There are the usual problems. The first statistic to be trotted out is the dreaded six-for-24 Kobe managed to produce in the biggest game of his career. It was, let's face it, pretty awful. Fifteen rebounds and solid D aren't really going to save that. Bad as it was, though, it was a single game. Nothing says "made my mind up already" like anecdotal statistics. What long-term statistics there were, were per-game statistics, uncorrected for pace (or minutes played, for that matter).

Of course, who really knows whether Kobe would come out smelling better if they avoided statistics last defined in 1978. Honestly, I doubt it. And like many (though certainly not all) Lakers fans, I think Jordan was overall a better player. But having said previously that statistics are a bad way to measure greatness, let me come out again and say this: statistics are even worse for comparing players from different eras.

Oh look, it's The Jump!

Basic statistics like points, rebounds, assists, blocks, steals: these are flawed, but at least most people more or less understand how they work, and why you can't use any individual one -- even points, the most popular one -- as the consensus standard for player greatness.

As a result, folks like Hollinger, Berri, and Winston have gone to some trouble to define alternative statistics like Player Efficiency Rating (PER), Wins Produced (WP, or WP48), and Adjusted Plus-Minus (APM). In their proper context, these statistics are very useful and shouldn't be tossed aside lightly. Instead, I'm going to toss them aside deliberately. (Well, at least as a basis for comparing players from different eras.) To do that, however, one has to take a closer look at what goes into them, which I'm afraid is a bit like learning how sausage is made.

For the purposes of this discussion, I'm going to pick on Hollinger's PER, but a similar point could be made about Berri's WP48. I'm leaving aside Winston's APM, for now, because it's fundamentally different from the other two and deserves its own post. Hollinger publishes his formula for PER, which when written out is tremendously long, but essentially works as follows. (I will take a few liberties that don't greatly affect the interpretation of PER.) One begins with unadjusted PER:

uPER = (Points + Assists + FieldGoals + FreeThrows + NetPossessions - Fouls) / PossessionsPlayed

NetPossessions takes into account things like steals and rebounds, which gain possessions, as well as turnovers, which lose possessions. PER then scales everyone's uPER by a constant factor so that the league average is 15.

Right away, you might observe that points seem to be counted twice, once as raw points and a second time as either field goals or free throws. Plausibly, that's because a player has additional value if he's able to create free throw opportunities and unassisted field goals beyond what would be expected from just the raw point count. You might also observe that these various categories shouldn't be given equal weight. For instance, there's no reason, a priori, that points and assists should count exactly the same. And you'd be quite right. So in fact, each of the variables above has a weight, a coefficient, which represents the relative contribution each makes to the player efficiency rating:

uPER = (wP × P + wA × A + wFG × FG + wFT × FT + wNP × NP - wF × F) / PP

where each of the w terms is a weight for the corresponding variable.

That's it, that's all there is, really, about Hollinger's PER. Now, to be perfectly fair, although wP and wA are simple constants in Hollinger's formulation, the other weights (and variables) are more complex; in particular, they depend on team and league averages for the variables in question. But the main point remains: PER is essentially a weighted sum of basic basketball statistics. And now you know a little about how sausage is made.

Based on the above, one thing I hope you'll notice is that there isn't one universal PER, but rather a template, upon which you can build a PER. Any choice of weights, whether they're simple constants or more complex expressions, constitutes a specific PER. Despite what I said above about the different variables being worth different amounts, one could set all of the weights to 1, and obtain a perfectly legitimate PER. It just wouldn't be Hollinger's PER.

This makes it sound as though Hollinger's PER is largely arbitrary, and it is to some extent, but not entirely so. Hollinger has, one must assume, put plenty of time and effort into crafting the different weights so as to best assess player worth -- certainly better than setting all the weights to 1. And despite the prevailing sentiment amongst Lakers fans, he's not a Lakers hater or an idiot. Anyone venturing forth to define their own PER has much to learn from Hollinger's creation.

But what about the main topic of this post, which is using statistics to compare players across different eras? How can we tell what a given PER means historically? There's nothing in the definition of either PER generally or Hollinger's PER specifically that tells us how good any given PER is, except that the average player has a PER of 15.

Fortunately, Hollinger has provided a handy rubric for determining the historical significance of a given PER:

A Year For the Ages: 35.0
Runaway MVP Candidate: 30.0
Strong MVP Candidate: 27.5
Weak MVP Candidate: 25.0
Bona fide All-Star: 22.5
Borderline All-Star: 20.0
Solid 2nd option: 18.0
3rd Banana: 16.5
Pretty good player: 15.0
In the rotation: 13.0
Scrounging for minutes: 11.0
Definitely renting: 9.0
Next stop, D-League: 5.0

You might reasonably wonder how Hollinger knows this. What makes a PER of 25.0 a weak MVP candidate, and a PER of 27.5 a strong MVP candidate? Because, historically, that's what they've been. If you go back through past seasons and look at players with PERs of 25.0 (or 27.5), you find that they have, by and large, been weak (or strong) MVP candidates. Of course, there are exceptions, but those could just as easily be poor judgment on the part of MVP voters as an infelicity in the definition of PER.

But then, I have to ask, is it fair to compare the current crop of players with past players using something like PER? The one thing today's players in their prime have in common is that they're not done playing. No one has had the time to base statistical measures on their play. But the giants of the past -- greatness is defined by their performance. The statistics that we use to assess and predict the impact of today's players are calibrated, significantly, by how well they match up against our subjective impressions of yesterday's heroes. The weights given to the various components of PER, for instance, have the values they do in no small part because they coincide with what was relevant in the past. It is not a trivial question whether a player with a high PER is inherently, objectively, quintessentially great, as much as he is Jordanesque, or Chamberlainesque. In the Great Bar Bet of which player is the greatest of all time, those past heroes are playing with house money. You can't get much more Jordanesque than Jordan himself.

Well, I suppose sometimes you can, for different values of Jordan.

It's possible that, someday, someone will have the insight to define greatness statistically in a way that we can all agree upon and that is decoupled from calibration against the players of any particular era. We would then be able to decide, objectively and definitively, who the greatest players of all time are. But the astonishing variety of basketball players and their fans argues compellingly against that, and I wouldn't bet on it happening in my lifetime, even with substantial odds in my favor.