I have been thinking about this problem from an analytics persepctive since the beginning of the season, but the WSJ had recently an article that triggered some more thoughts. Here I am going to put some of my thoughts on the topic — bare with me because the thoughts will be rather unstructured at this point, and mainly aim at seeing what other people think about the problem.
Win probability added – vs – expected points added
In the WSJ article the importance of a challenge is measured through the win probability added from it. Win probability added is a natural choice of course, but are there any possible shortcomings using it?
First let’s compare the two objectives of maximizing win probability – vs – maximizing expected points. During the beginning of a game, there is more uncertainty with regards to each outcome, with win probabilities being closer to 50% (or in general the pre-game probabilities that are based on team strength). As a result, any given possession will not swing the win probability by a large percentage. During the end of the game, the uncertainty is smaller, which also means that there is smaller uncertainty on how a given possession impacts the win probability. For example, an offensive foul in the first possession of the game, most probably will not change the win probability at all, since there is high uncertainty during that time of the game. However, the same charge in the last 5 seconds of the game being down 1 point, most probably will have a swing in the win probability of about 20-30%. This is an artifact of the finite duration of the game.
Depending on which view one takes, the corresponding on-court strategy might be different as well. For example, early in the game maximizing the win probability or maximizing the expected points, most probably will yield similar decisions. In the extremely simplified version of decision making, a corner three at the first possession of the game is a better option compared to a long-2 both in terms of win probability and expected points. However, at the end of a tied game these two objectives diverge. A long-2 is the better option in terms of maximizing win probability, while a corner three still provides the best choice in terms of expected points. So it should be clear that game clock clearly impacts the leverage of each call in terms of win probability. However, in terms of expected points the value of each call is not impacted by the game clock. A challenge that is worth 1.5 expected points, will be so either if it is during the first possession or the last one of the game.
At the same, a point scored in the first possession of the game counts the same (i.e., 1 point) as one scored at the end of the game. Their (win probability) levarege at the time being scored will be different – just as explained above. However, their contribution to the final points of the game is the same (1 point). So in close games – where challenges have the potential to make a difference – a point scored or saved in the first quarter is equally important as one in the last 2 minutes. For example, the WSJ article ranks the challenge used by POR at the game against DAL as the one with the largest swing at win probability (which is true). During that challenge, the referees had called a shooting foul against POR with 9 seconds left in the game. DAL was down 1 point and the challenge was worth approximately 1.5 expected points (Finney-Smith is about a 75% FT shooter) if won (without considering the ensuing jump-ball result if the challenge was succcesful). However, if POR had used their challenge earlier in the game for a situation that was at least worth 1.5 expected points, they would be 9 seconds before the end of the game up by 2.5 (expected) points, which would give at that point a very similar win probability as the one after the reversal of the call challenged. Note that this is not any type of hindsight bias — 2 points scored in the last 10 seconds give you the same lead as if these points were scored during the first half. Of course this makes the assumption that the game would have proceeded in the same way/path if a challenge was called earlier in the game. However, given how small of a change any given basket provides in the overall score, this assumption might not be too much of a stretch (of course cases where a star player is charged with early foul troubles etc. might be different, but these are also things that are hard to be included in the expected value from a challenge).
So overall, the fact that the highest win-probability added comes from late game challenges is to be expected given the high leverage in terms of win probability that late game situations have. However, looking the issue from an expected points lense, might also give us more insight on how one can approach this strategically.
The secretary problem
The coach challenge fits well under problems involving the optimal stopping theory. The most popular example – and very relevant to our setting – is the secretary problem. You have one secretary position to fill and start interviewing (known before hand) canditates for the position. Candidates come in random order (in terms of their skills/quality). We have to make a decision on whether to hire the candidate immediately after the interview. Once we reject a candidate we cannot call him/her back again, while once a hire is made the process stops and no more candidates are interviewed. The question is when should we make a hire. This is similar to the coaching challenge problem (with some important differences we will discuss later); we get a series of calls that the coach can challenge (hire), and once you make a challenge no other challenges (hires) can be made.
In the case of the secretary problem, the optimal strategy is the following stopping rule: interview the first candidates without hiring anyone. Assume also that the best candidate among them has a “skill rating” of . Then interview the rest of the candidates and hire the first that has a “skill rating” better than (or the last candidate if this never happens). This simple rule selects the best candidate with a probability approximately 37% – in other words, the best candidate is chosen in 37% of the times.
Now the coach challenge problem has a few differences. For one, if we use the win probability as our metric of qualifying calls (candidates), then the order with which these calls arrive is not random. Based on the above discussion, one can argue that earlier calls might hold less value in terms of win probability compared to later calls (e.g., last 5 minutes). However, in terms of expected points the order can be assumed to be random. The expected points of a challenge needs to consider the probability of the call being wrong (which might not be trivial in subjective calls — e.g., block – vs – charge). The expected call from a call reversal then depends on the type of call. For example, an out-of-bounds reversal essentially will provide the expected points per possession for the opponent. A shooting foul will provide the expected points of two free throws from the player to take them. It should be clear that there is an upper limit on what is the maximum expected point from a challenge. For example, if a three-pointer was scored but an offensive foul was called on the shooter and the coach challenges it to be a shooting foul, a successful challenge will yield 3+ points (say 3.8 points). So there is a limit on what value you can get from the challenge in terms of expected points. These cases might not appear very often but still there is an upper bound on the value, which does not exist in the secretary problem. If there is an absolute maximum that you can obtain, then once you observe this maximum you know there is no way to find a better situation and you should make the challenge. So if we assume that the max expected points to be yield are 3 (again in reality getting this expected value from a call is extremely unlikely) and that the number of calls that a coach can challenge are 20 (this number can be learnt from data and we can also consider only calls that have say more than 40-50% probability of being overturned). Then we can use the following decision process:
- For the first “challengable” calls do nothing unless if one of them yields the maximum expected points. In this case, challenge.
- After the first 7 challenges, and with being the maximum expected points for the first 7 challenges, if or , challenge. Otherwise, continue.
Now this is of course just a roadmap on how one could start thinking of the problem on when to challenge. High-profile, high-levrage challenges like the one in the Portland – Dallas game, are outliers and personally I think that keeping the challenge in case a call/situation like this appears is a bad strategy (it is essentially based on the availability heuristic – it is easier to recall high-leverage successes than run-of-the-mill ones).
Incorrect calls – vs – incorrect non-calls
One of the limitations of the challenge process at the moment is the inability to challenge non-calls. There are practical reasons for this (i.e., continuation of play etc.) but next iterations of the rule (if it is to remain) need to consider this. This is particularly important, since based on the L2M since 2015, 92% of the incorrect ref decisions are non-calls, and hence, non-challengable !
Again this text is just a dump of ideas and philosophical-level thoughts about the win probability and expected points. I am looking forward to hearing to your comments.