Incentive Stats & Goodhart's Law (HV Weekly: 10/2/2020)
How do coaches get players to care about the "right" statistics?
We are BACK on Fridays with the Hoop Vision Weekly, as we’re less than two months away from the scheduled start of the 2020-21 college basketball season.
Before we get into today’s edition, some quick programming notes:
On Monday, HV+ subscribers got their first taste of our subscriber-only preseason coverage with a comprehensive preview of Villanova, and how Jay Wright’s progressive, guard-heavy style has seeped into the rest of college basketball over the years. Next up: Illinois. Subscribe now for the full HV+ experience, featuring substantive preseason coverage you won’t find anywhere else:
I want to give some credit to Seth Partnow (NBA Analyst for The Athletic), whose tweet about Goodhart’s Law inspired this week’s newsletter. This isn’t the first time Partnow’s work has inspired some Hoop Vision content — so go ahead and follow him on Twitter if you haven’t already.
And as a call-back to our HV Weekly edition of two weeks ago — we tried to capture the incentives and tension that goes into non-conference scheduling — Louisville head coach Chris Mack captured it far more successfully in this two-minute video.
Incentive Stats
Accountability is one of the most important aspects of the coach-player relationship, and coaches employ many different styles to hold players accountable. Some emphasize individual meetings, some swear by group film sessions, many believe accountability can be manufactured by yelling during practice…and then we have statistics.
I refer to the application of statistics involving accountability as incentive stats.
In other applications, statistics and data can be used to help determine what to emphasize. Should we play small? Should we play fast? Should we force baseline?
Incentive stats serve a different purpose. We already know what we want to emphasize, now we just have to measure it as accurately as possible.
We hear a lot about the negatives of statistics turning players into robots or somehow dehumanizing the game. In my experiences working for teams, the act of checking and caring about statistics is an extremely human thing. In post-game locker rooms, you can’t hold a box score sheet in your hand for more than a minute without a player or staff member ripping it out of your hands to check their stats.
That’s just a natural part of being a competitive human being, but it’s important for the greater good of a program that when a player eagerly checks the box score, he/she understands and cares about the “right” statistics.
Take a gifted scorer, for example: the key stat for them to check in a box score could be as simple as shooting percentage — valuing efficiency as much as just raw points.
So how do you get players to care about the “right” statistics?
The answer is very simple in theory but potentially very difficult in reality…
Playing time. Playing time isn’t the only way to make a point, but it’s the best way.
Again drawing back to my experiences working for teams, when players knew how important a statistic was to our head coach and program — so important that it dictated playing time — they became highly invested in maximizing that statistic.
On the first ever episode of Solving Basketball, I told the story of how our players became invested in adjusted defensive efficiency — even if they didn’t necessarily know the exact details of the stat.
The problem with the playing time approach is you have to put your money where your mouth is. If one of your best players is performing poorly in an incentive stat that you have previously deemed important and preached daily, the system begins to lose its power if that talented player doesn’t receive consequences.
Plus-Minus… the foundational incentive stat
Apart from the “3 is more than 2, so avoid mid-range jumpers” philosophy, plus-minus is probably the second item most associated with the basketball analytics movement
(There’s some major irony that this supposedly non-traditional statistic is literally just the score of the game, but we’ll save that for another day.)
I’ve talked at length in the past about how plus-minus shouldn’t be used to make future decisions on playing time. Not only is it a very noisy statistic in small samples, but because of the nature of college basketball — short season length, 40-minute games, star players infrequently getting subbed out — there is little to no evidence that it is predictive, even over the course of a full season.
But from a player accountability perspective, plus-minus is seemingly foolproof. At the most fundamental level, every player should want the score to move in the right direction when they enter the game. That’s the whole point of keeping score.
Goodhart’s Law
So far we’ve mainly focused on the positives of incentive stats, but it wouldn’t be a Hoop Vision newsletter without discussing the trade-offs. In this case, the negatives come in the form of Goodhart’s Law.
"When a measure becomes a target, it ceases to be a good measure."
Just like it’s a natural human reaction to check the box score, it’s a natural human reaction to “game the system” — or whatever statistic is being targeted.
Let’s start with a very simple example.
NBA players are evaluated by coaches and general managers on shooting efficiency. The players know they are being evaluated. As a result, they purposefully let the buzzer go off before attempting an end of quarter heave. Twitter users chuckle about it when they see the videos, but it is a very real thing.
Here’s Kevin Durant even (somewhat) admitting to doing so:
“It depends on what I’m shooting from the field. First quarter if I’m 4-for-4, I let it go. Third quarter if I’m like 10-for-16, or 10-for-17, I might let it go. But if I’m like 8-for-19, I’m going to go ahead and dribble one more second and let that buzzer go off and then throw it up there. So it depends on how the game’s going.”
An end-of-quarter heave is just a mostly innocuous one-off example, but incentive stats can also have broader impact.
The overall goal of rebounding is for your team to get the ball. But on a player-by-player level, there can be different goals depending on context — like boxing out the opponent. A common incentive stat used to hold players accountable is “box out percentage” — how often an individual player boxes out their assignment.
I worked on staffs where we would regularly chart and display that information to our players, but it’s important to still remember the overall purpose of grabbing the ball. The box out is a skill that’s underrepresented by the traditional box score, but it’s not immune to Goodhart’s Law.
If players feel they are only being evaluated on boxing out, it creates an incentive where they become misaligned with the larger rebounding goals. Perhaps the ball comes off the rim right to a player, but he chooses to be technically correct (boxing out) instead of practical (grabbing the ball).
So how do you avoid Goodhart’s Law and balance incentives?
Some possible solutions…
Measure as precisely as possible. This is an area where the analytics movement has undoubtedly advanced the sport. Use rebound percentage not rebound margin. Use effective field goal percentage not field goal percentage.
It’s very easy to solve the NBA heave example by simply removing all heaves from your measurement. In fact, the NCAA default is to omit heaves from the official stats. (And to be fair, I’m sure NBA teams all omit heaves for their in-house analysis.)
Measure the actual result as much as the details. The details (like boxing out) are important, but only to the extent that they impact the result (rebounding). It’s important to measure both.
As previously mentioned, plus-minus (aka the score) is the ultimate result. Unlike a very specific skill or task, it’s mostly impervious to Goodhart’s Law. The only way I can think of a player (sustainably) gaming plus-minus would be something along the lines of refusing to go into the game depending on context.
Diversify your measurements. Goodhart’s law especially comes into play when there is a specific measurement being disproportionately emphasized. So instead, diversify your measurements.
For rebounding, you can still track box out percentage — but consider adding something like “out of area rebounds” into your measurements. Or something similar that captures grabbing the ball.
The players don’t need to know everything. A measurement can’t be gamed if the individuals being measured aren’t aware of it in the first place. If you’re looking for a truly impartial measurement, it might make sense to keep it behind the scenes.
For more on incentive stats, check out our tutorial on Individual Defensive Accounting.
That’s it for this week! As a reminder: this weekly newsletter — along with all Hoop Vision content across platforms — is presented ad-free and supported by subscribers.
If you’ve enjoyed this newsletter or any tweets, videos or podcasts over the past couple years, please consider subscribing to Hoop Vision PLUS for $10/month or $100 annually.