CRS Pilot, Show me the Math! - 8/17/23
Nearing the Finish Line
We completed nine weeks of our Pilot. In dog weeks, that’s sixty-three weeks of events. Or so it might feel that way to you. I say that because we’re nearing the end of our Pilot, and your questions are increasingly changing to … “what’s next”? I’ll go to the home page of Court Reserve for more details:
“… leaves the board extremely encouraged that there is a role for a Computerized Rating System in our Club.”
No need to carefully parse that sentence for hidden meaning. It’s direct, it’s on the home page of Court Reserve. You’ve done the work, and your work is paying off. There is a role for a Computerized Rating System in our Club.
Bad Court Assignments
Do you remember the Seinfeld episode where Jerry notices that nobody is eating at Babu Bhatt’s restaurant? Jerry eats there, then Jerry gives Babu several ideas to improve the restaurant. Babu spends money to renovate the restaurant, nobody shows up. Babu must close his restaurant.
During the early stages of the episode, Babu tells Jerry how wonderful he is. Jerry agrees, he daydreams that he is wonderful. Then Jerry bankrupts Babu without intending to do so, and Babu suggests Jerry is a bad man.
When you run a Pilot, you think you are doing a good job. Then somebody reminds you that they are constantly assigned to courts six, seven, and eight … where there is no shade, there are no fans (wind-producing devices), and there is ample heat and humidity. You realize you are not wonderful, you are unrepentantly putting some of your players in the blast furnace every week.
It was not my intention to do that. I’ve never once thought about anything other than just starting the higher colors on Court 1 and moving to Court 2 from there. That was a mistake. I should have given more thought to where our players were playing each week. I should have mixed things up more. That’s on me.
The Player Resume
Now that we have a book of work to fall back on, it makes sense to review some of our players via what I call a “Player Resume”. With nearly four-hundred-sixty games in our pilot database, we can see how the computer evaluated players and compare the evaluation against won/lost records by color level of the opponent.
Let’s look at a Teal player.
Initial Rating = 2.500
Current Rating = 2.494
Record Against Orange Players: 0 wins, 3 losses
Record Against Purple Players: 1 win, 3 losses
Record Against Red Players: 5 wins, 28 losses
Record Against Teal Players: 18 wins, 14 losses
Can we agree that the computer evaluated this player in a fair manner? Sure looks like it. This player has a reliable resume (many games), and wins just over half of games within the color level of the player. Against better players, the player upsets better players about 15% of the time. This player is well-classified by our Club and by the computer.
Let’s look at a Red player.
Initial Rating = 2.750
Current Rating = 3.023 (i.e. gained one color level)
Record Against Maroon Players: 1 win, 5 losses
Record Against Orange Players: 4 wins, 6 losses
Record Against Purple Players: 3 wins, 1 loss
Record Against Red Players: 4 wins, 0 losses
The sample is small, but the computer quickly moved this player up one level (24 opponents = 12 games in the sample). The player deserved to be moved up one level, going 7-7 against players one or two levels above the player, hitting a hard wall against Maroon players.
Let’s look at a Purple player.
Initial Rating = 3.000
Current Rating = 3.443 (i.e. gained nearly two color levels)
Record Against Green Players: 1 win, 0 losses
Record Against Indigo Players: 2 wins, 2 losses
Record Against Maroon Players: 3 wins, 1 loss
Record Against Orange Players: 11 wins, 2 losses
Record Against Purple Players: 10 wins, 7 losses
Record Against Red Players: 13 wins, 4 losses
When we eyeball the resume, we’re kind of surprised that this player hasn’t jumped even higher than the 0.443 point gain assigned by the computer. The secret is within Purple players … this player went 10-7 against Purple competition while excelling against higher-level players (17 wins, 5 losses). As this player sees continued computer rating increases, this player is penalized by losing subsequent games to Purple players. That’s how a computer rating works. We will get used to “gray” scenarios where some metrics stand out positively while other metrics don’t look so great. Not every story is clear and easy to interpret.
Let’s look at an Orange player.
Initial Rating = 3.250
Current Rating = 3.219 (no change)
Record Against Indigo Players: 0 wins, 1 loss
Record Against Maroon Players: 4 wins, 8 losses
Record Against Orange Players: 5 wins, 0 losses
Record Against Purple Players: 0 wins 2 losses
Record Against Red Players: 1 win, 2 losses
Record Against Teal Players: 0 wins, 1 loss
What would a Club Rater do with this player? This player wins within the color level of the player, and upsets players at a fair rate against better competition … and has gone 1-5 against players at lower color levels. The computer says “no change”. Can we agree that the computer is being fair with this player? If the rater sees this player on days against lesser competition, the rater is not going to be impressed!
Let’s look at a Maroon player.
Initial Rating = 3.500
Current Rating = 3.282
Record Against Indigo Players: 0 wins, 2 losses
Record Against Maroon Players: 4 wins, 8 losses
Record Against Orange Players: 8 wins, 7 losses
Record Against Purple Players: 7 wins, 3 losses
Record Against Red Players: 4 wins, 3 losses
Record Against Teal Players: 3 wins, 1 loss
This is a classic example of a player being put in “harm’s way”, as I’ve repeatedly said during the Pilot. This player isn’t receiving a fair evaluation of skills. Only 14 of 50 opponents have been at/above the color level of the player. No good. The computer did the best job the computer could do … but in a world where we roll out computer ratings, this player will not be playing against Purple/Red/Teal players. This outcome is a failing of the match format of the Pilot.
Now, having said all of that, here’s something a lot of our Club Members are not going to want to hear. For every player that moves up via a computer rating, we’ll have a player who moves down. This is going to offend some of our players. Early in the implementation of a computer rating system, there will be significant swings up-and-down. Some of the downswings will be due to noise … random outcomes in the first 5-10 games. After 25 games, the downswings are going to reflect player performance. Nobody wants to lose 0.25 ratings points. Unfortunately, this will happen, and will happen to a significant minority of players. It happened in the Pilot, and it will happen in the real world. It is my opinion that we’ll need to protect players who have this happen to them to encourage continued participation in computerized ratings.
Let’s look at an Indigo player.
Initial Rating = 3.750
Current Rating = 3.848
Record Against Green Players: 4 wins, 2 losses
Record Against Indigo Players: 5 wins, 5 losses
Record Against Maroon Players: 0 wins, 3 losses
Record Against Orange Players: 5 wins, 2 losses
Record Against Purple Players: 3 wins, 1 loss
Record Against Red Players: 1 win, 0 losses
Record Against Teal Players: 0 wins, 1 loss
We see the shortcomings of the Pilot, don’t we? At 9:30 am in Summer, you don’t get to pick and choose games against players equal to you. We just don’t have enough parity to do that in this Pilot. Therefore, we must trust that the computer does a reasonable job with the information available to the computer. This player went 4-2 against better competition, 5-5 against equal competition, and went 9-7 against lower color levels. The computer looks at that mess of a situation and says “you’re better than your starting point”. Repeatedly, we see cases where players outperform expectations against better players and perform below expectations against lower color level players. That’s a topic for another day … and that topic won’t be an issue if we roll out computer ratings in 2023.
Let’s look at a Green player.
Initial Rating = 4.000
Current Rating = 4.294 (equal to moving up to Burgundy)
Record Against Burgundy Players: 4 wins, 0 losses
Record Against Green Players: 5 wins, 1 loss
Record Against Indigo Players: 10 wins, 1 loss
Record Against Maroon Players: 1 win, 0 losses
Record Against Orange Players: 1 win, 0 losses
Record Against Purple Players: 1 win, 0 losses
This player played just 11 games in the pilot, blitzed all competitors, and is essentially moved up one color level by the computer. The player went 9-1 against equal/better competition, the player went 13-1 against lower color levels. This outcome is REALLY IMPORTANT regarding our future. Can we agree that this is a good-looking resume? If a player beats everybody, the computer is going to reward the player.
Also notice that the player only got to play 2 games (times two opponents = 4 wins/losses) against Burgundy players. The computer does not care if you don’t get an opportunity to play against better players. The computer will reward you regardless! I probably spent 20 minutes this week talking to players who are concerned about Aqua/Burgundy/Green players locking them out of opportunities. The topic comes up every week, in every session. Yes, we will have players who lock us out of opportunities. THE COMPUTER COULD CARE LESS! Win the games in front of you. Take advantage of the opportunities presented to you. If you win, the computer will adjust you accordingly.
But … But … But … My Computer Rating Doesn’t Reflect My Ability!
It’s coming, folks.
If we move forward with computer ratings, there will be players who feel that the computer is not being fair to them. Some will be frustrated with a partner (“why did the round robin card assign me to play with Heather, that’s not fair, Heather is awful”). Some will be frustrated that somebody called a ball out that was clearly in. Somebody will be upset that their drop shots hit the net because of a gust of wind.
The Pilot suggests computer ratings begin to stabilize after fifteen (15) games. Realistically we’d want 20-30 games to see ratings stabilize, but that’s not possible in a Pilot. In cases where I’ve analyzed our most loyal pilot players, the computer rating begins to stabilize after fifteen (15) games. External factors begin to equalize … you played with Heather once but you played against Heather twice … can’t complain about Heather anymore!
There are secrets to performing well in a computer rating environment (should we adopt a computer as part of a ratings process). I will share some of those secrets with you now.
Out Balls: Don’t hit them. There is no reason to freely give points to your opponent. You control whether you hit out balls or not.
Low Balls: Hit them! If you hit a third shot drop at shoulder level, the ball is coming back at you like a missile.
Lines: Stay away from them! Why would you try to paint a line, giving your opponent a chance to say “out”!???? Keep your balls 12 inches away from all lines.
Stacking: Do you feel like you have a weak partner? Then stack. Take the forehand, and control the court appropriately. This benefits both you and your partner.
Stacking, Part 2: Playing with a left-handed player? Learn how to stack so you have both paddles in the middle of the court.
Drop Shots: Aim your drops a few inches higher than you’d aim them in rec play. Will your former rater be happy with you? No. Will you be less likely to hit the ball into the net? Yes! This is important. When you hit a third shot drop into the net, you give up a 45% chance of scoring a point, and you give up all future points you’d score after winning the point. On average, this means that every drop/drive that goes into the net results in your team losing an average of 0.85 points when you make this mistake (I’ve done the math).
Deep Serves: It’s hard to return a serve deep if the serve itself is deep. You control your own serves. Since you control where you hit serves, hit a deep serve to a backhand to make it hard for the opposition to return your serve. If you are a 4.0/Green player or above, put some topspin on the serve to make it even harder to hit an effective return of serve. If you are a Purple/Orange/Maroon player reading that sentence, start thinking …
Don’t Hit Serves Out: Again, you are giving up 0.85 points every time you hit your serve out. Don’t do that!
Deep Returns of Serve: Deep returns (especially into corners or into the middle of the court) make it much harder for your opposition to hit an effective third shot. Make their job difficult!
Drive Short Returns of Serve: The math is highly in your favor if somebody returns the serve short (especially to the middle of the court). Rip a drive in these situations, give your opposition minimal time to react.
Keep ‘em Back! On your 4th/6th/8th shots, keep the opposition back. Where possible, hit the ball to a backhand foot. If you are a Green/4.0 player and above, put topspin on the ball to make it even more difficult for your opposition to hit an effective shot.
Dramatic Put Aways: Do this in your open play games. Do not do this in a competitive environment. You are inviting mistakes on shots where you are likely to win the point.
The items listed above are simple concepts, and all are things you directly control. You cannot blame your partner when you make simple mistakes that cost your team points.
Want to conduct a fun experiment? In your next game, count every time you make an unforced error during a game. Try to keep your total at or under five. It’s hard! The first game I tried this last week I hit eleven (11) unforced errors. It’s hard to blame your partner when you hit eleven (11) unforced errors in one game.
Focus on everything you control. You control more in a pickleball game than you think you control.
Based on everything I’ve observed during our Pilot, the players who control themselves … those who do not make sloppy mistakes … those are the players who have made progress with their computer rating.
If we move forward with computer ratings, some players are going to grumble … suggesting that the computer is cheating them. If we move forward, I am going to quiz the grumblers how they are performing on all the issues noted above. We all need to be accountable for our style of play. The computer is going to reflect our dedication to accountability.
Also – if a player has problems with a computer rating, we will be able to produce a “resume” of player performance. We will be able to see wins and losses by level. We will have data to address the problem.
Show Me the Math!
Ok!
While I call our rating “KPR” during the pilot, I am using the UTPR formula used by USA Pickleball in tournament play. The formula is freely available for use. It is derived out of the ELO-family of equations used in online gaming, chess, and other applications. As long as we have a good starting point (i.e. colors), a computer rating will work properly. Look to DUPR for the problems a computer has when it uses a poor starting point. While you (personally) might not like color ratings, color ratings are the secret sauce that allows a computer to properly adjust players in a reasonably rapid manner.
Let’s pretend that you are an Orange player, and you believe you’ve been misclassified.
Game 1: You (Orange) and your partner (Maroon) are paired against two Maroon players. You win the game 11-7.
Going into the game, you had a 24.5% chance of winning the game (in our Pilot, the probability was about 33% in this scenario). You win the game. You earn 0.075 ratings points, your opponents lose 0.075 ratings points.
Your rating becomes 3.325.
Game 2: You (3.325) and your partner (3.550) play a pair of Maroon players who both won their previous game (3.550). You win the game 11-9.
You earn 0.073 ratings points, your opposition loses 0.073 ratings points. Your new rating is 3.325 + 0.073 = 3.398.
In just two games, you went from 3.250 to 3.398. If you are truly as good as you think you are, your rating will adjust quickly … if you win.
Game 3: Your head is swelling now. You ARE a great player. You play a pair of Maroon players who split their first two games (3.500). Your partner is a Maroon player who won both games (3.600). You lose 11-3. Your head is swollen now for a different reason! How does your rating change?
The game was a toss-up. You lose 0.050 points. Your rating adjusts from 3.398 to 3.348.
In the fourth game, you run into a buzzsaw … both players won their three previous games and have a rating of 3.650. Your partner lost all three games – and now has a rating of 3.350. You lose 11-5. You are angry with your partner. Your day is turning negative by the second. You rue the fact that a computer is evaluating your ability! What happened to your rating?
Look at that! You were supposed to lose (93.7% chance of losing), you lost, so you only give up 0.006 ratings points. Your 3.348 adjusts down to 3.342. In essence, your rating does not change. You are not penalized for playing a game you are supposed to lose.
Game 5: You play against two players who won all four games and are now rated 3.700. Your partner is a 3.45, you are a 3.348. YOU WIN!! YOU WIN!! 11-10. It’s a thriller, but you win. What happens here?
You gain 0.094 ratings points for your upset. Well done! Your computer rating just went from 3.348 to 3.442.
Game 6: You play against two players who are at 3.500. Your partner is at 3.450. You win the game by a score of 11-8. You’ve played six games, you are done for the day. What is your final computer rating after the day is over?
You add 0.062 ratings points, taking your rating from 3.442 to 3.504.
During six (6) games, your “Orange” rating of 3.25 has been adjusted by the computer to a “Maroon” rating of 3.504. And … and … you weren’t perfect, were you? You won four games, you lost two games.
The computer quickly adjusted your rating. If you truly beat players above your level, you will quickly adjust.
We’ve proven all through the pilot that even if you beat players at your level, the computer will quickly move you up within two weeks. The computer fixes injustices.
You have to win.
You don’t have to be perfect. If you lose a game, you lose a game. You are supposed to lose games.
If this is the direction our Club heads in, I am 100% confident that our players who choose to participate will move up-and-down … good weeks … bad weeks … good partners … bad partners. Over time, each player will end up where their results dictate. If you consistently win, the computer will reward you. If you lose, the computer will lower you to a point where you can begin to consistently win again. This will result in a reshuffling of the Club members who choose to participate. This will result in more competitive games for those who choose to participate. Those who choose to participate will be able to see, on a weekly basis, how their skills are improving in a real-world setting. Whereas a Ladder dictates slow movement, a computer rating can adjust players faster, benefiting players who are making good progress.
Can My Rating Go Up If I Lose More Games Than I Win?
Yes. 10 out of 154 players in the pilot lost more games than they won and saw a rating increase. This happens when the player consistently plays against high-rated competition and upsets high-rated competition more often than is expected.
Can My Rating Go Down If I Win More Games Than I Lose?
Yes. 18 out of 154 players in the pilot won more games than they lost and saw a rating decrease. This happens when the player consistently plays against lower-rated competition and does not perform to expectations.