CRS Pilot, We’ve Learned So Much - 7/13/23
We have completed five weeks of our Computerized Ratings Pilot.
June 6/8 = Practice Week
June 13/15 = 1st Official Week, Scores Entered into DUPR via Phone
June 20/22 = 2nd Official Week, Scores Entered into DUPR via Administrator
June 27/29 = 3rd Official Week, All Players Must Participate in DUPR
July 11/13 = 4th Official Week, Thursday Women’s Session Cancelled due to Heat
We have learned SO MUCH in a short a period of time.
If a player is mis-assigned to a color level, a computer can fix the problem within 1-2 weeks. We have proven this beyond a shadow of a doubt.
If a player is very close to the next color level, a computer can move the player one color level higher within 4-7 weeks. The player can have a bad day, but still make progress. We have proven this to be true.
More than 75% of the color assignments have proven to be reasonable, because the computer is evaluating players at +/- 0.15 points of where colors placed players. This will change as we get deeper into the pilot, but for now, the raters likely made mistakes with individuals (as will always be the case, not the fault of the rater) but the process was overwhelming accurate. Many will disagree with this assessment. You are allowed to disagree. The data prove this assessment is accurate.
DUPR changed their algorithm, making it hard to use for the majority of our Club Members. New players are assigned a 3.5 rating. It will take at least 30 games for DUPR to figure out how to properly assess new players. Our pilot is proving that 30 games is a reasonable estimate of the time needed to achieve an accurate DUPR. Our KPR (our parallel to the UTPR used in tournaments) allows for a starting point (color), allowing KPR to iterate to an accurate rating faster.
We continue to await a Court Reserve ratings system. If Court Reserve develops an “ELO-based” system (similar to DUPR or the UTPR tournament rating that we are also piloting, called “KPR”), there will be myriad advantages to using a rating that is integrated within Court Reserve.
We are not testing a “King of the Hill” format. We are using a “King of the Hill” format to create interesting matchups for our pilot. Should we roll out Computerized Ratings later this year, we will leverage formats that are potentially “more fair” than “King of the Hill”.
We are not evaluating your specific rating. In other words, we want to see what a computer does with your rating. We occasionally place you in favorable spots or unfavorable spots to see what happens. This is not a test of your ability, this is a test of the ability of a computer to evaluate what happened to you. For this reason, we are not moving people up/down based on the result of a pilot.
The computer does not care about players being targeted. The computer does not care about “bad partners”. The computer is able to average these challenges out over time and arrive at an accurate rating for a player.
Upsets happen. Upsets are supposed to happen! On average, two players one color level below two other competitors will win about 20% - 33% of the time. In our current world, we would ask to be “moved up” because we “beat” players better than us, meaning we feel we are “equal” to the players we beat. In a Computerized Rating world, we would earn 0.09 ratings points, and if we pulled off a few more upsets we would automatically be “equal” according to the computer. Of course, it is hard to sustain that level of success. But if we can sustain success, the computer will adjust our rating properly without having to ask to be “moved up”. We would earn the right to be moved up, and our rating would reflect our ability.
Upsets happen, part two. Computerized Ratings reward those who upset the opposition. Computerized Ratings penalize those who are upset. If we move to Computerized Ratings, we must become comfortable with the fact that we might have a 3.53 rating at the start of a day and end up with a 3.42 rating at the end of the day.
Players control their destiny. With Computerized Ratings, the player ultimately controls his/her progress. It takes a reasonable sample of games for this to happen, but the player dictates the future, not a rater that the player can be angry with. It will no longer be “somebody else’s fault”. If you complain because you lost 14 games out of 20 and the computer moved you from 3.50 to 3.39, it will be up to you to start winning games to reverse the trend.
One of the biggest complaints I hear is this:
“If we go to computerized ratings, the best players will not play. They will protect their rating at all costs, and we will not get opportunities to prove ourselves or compete against better players. Better players will lock us out of their world, and nothing will change.”
We can’t force players to participate. However, we have proven beyond a shadow of a doubt that a computer will adjust your rating properly regardless of who chooses to play. Your rating will increase if any of the following happens:
Win most games against inferior competition.
Win 60% of your games against equal competition.
Win 30% of your games against better competition.
Winning will allow your computer rating to increase.
Focus on what you control.
If we roll out computerized ratings, take advantage of the opportunities offered to you.
What Does a Computerized Ratings Journey Look Like?
We now have enough games to illustrate what the journey looks like. We need to become comfortable with “ups and downs”. We will not be a static 3.5/Maroon player for two years. Each week, our rating will change.
Here is the journey for one of our Pilot members. This player started at 3.25, and after four weeks of play the player is at 3.35 (and yes, this player started at a different level … I am standardizing all starting points at 3.25 to protect the identity of the player).
If you are on the journey this player is on, you’d be very excited with your progress. You went from 3.250 to 3.475. And then? Thump.
Blue arrow? The player made progress playing against his color level.
Red arrow? The player struggled playing against players one color level higher.
Over four weeks, the computer moved this player from 3.25 to 3.35.
If we move to Computerized Ratings, this is a version of the future we must become comfortable with. The computer will assign a number to us, and the number will move +/- 0.125 above/below who we are at that time. This player is probably a 3.375 player, moving between 3.25 and 3.50.
Let’s look at another journey.
This is another aspect of Computerized Ratings, one none of us want to deal with. Some of us must face the reality that we aren’t as good as the color assignment we received. This player had a good first week (3.25 to 3.32), and since then? Yuk! 3.32 to 3.11.
Every one of us will have a month-long stretch that looks like the graph above. Every one of us will have to become comfortable with the fact that we aren’t a 3.25 player ... instead, our rating is a snapshot of recent performance, and our rating can be changed (up or down).
If we implement Computerized Ratings, we will receive complaints.
“THE COMPUTER ISN’T FAIR. I’M BETTER THAN THE COMPUTER THINKS I AM. I SHOULD NOT BE PENALIZED!”
Wrong.
In the case above, the player went from 3.32 to 3.11 by winning one game and losing ten games.
It’s ok to have a won/lost record of 1-10. If you go 1-10 and only lose 0.21 ratings points, you weren’t penalized. Next week represents a new opportunity. Go win a few games and you will reverse the trend.
IMPORTANT POINT: Notice the first five games. Imagine if this player was being evaluated/observed for movement from 3.25 (Orange) to 3.50 (Maroon). The player would be winning games and could be moved up (incorrectly) to Maroon. There are problems with every ratings system (including a computerized ratings system). A system that moves players up and keeps them there creates inequalities (as observed in the table above). A computer reacts in a fluid manner, but the computer moves players down. It will take a lot of patience to help half (yes, half) of our club members deal with bruised egos when the computer takes a 4.10 player and moves the player to 3.95. It is going to happen often, and club members will feel frustrated. The 3.95 rating can be fixed by winning games. If the player doesn’t win games, the player is a 3.95 player, not a 4.10 player.
Let’s look at a final case study.
For most of us, this is what the story is going to look like. Up days and down days. This player had a day where KPR went from 3.37 to 3.26. This player had a day where KPR went from 3.27 to 3.35.
Over the course of 21 games, the player went from 3.25 to 3.38.
Heck, within just two games this player went from 3.37 to 3.24! That happens when you are upset in back-to-back games.
If we move to a Computerized Rating System, we will not operate in a static world where we have “earned something”. A player recently said “I earned the right to be at this color level, I am better than those players.” That mindset will become old thinking. Every one of us will have to become comfortable with “opportunities and consequences”. We will have the opportunity to play better players, something the old system suppressed. We will face consequences for losing games. We may have a week (or a month) where we are playing against players we perceive we are better than. This will be a consequence for losing to players the month prior. Our view of our performance will have to shift … from “I am an Orange player” to a world where we move in a fluid manner based on recent outcomes. Anybody who has lost in a Ladder understands this world. Anybody who has played in USA Pickleball sanctioned tournaments understands this world.
Upsets
The three case studies include upsets. Each player benefitted from winning a game the player should not have won. Each player suffered when losing a game the player should not have lost. In a computerized ratings world, you are rewarded when you upset somebody. You earn 0.08 or 0.09 ratings points.
Upsets happen 20% - 33% of the time when there is a one-color-level difference between players. If you don’t upset players, it will be harder for you to move your computer rating higher.
In all three case studies, upsets even out over time. Everything evens out over time (bad partners, good partners, being targeted, targeting players, out calls, wind, heat, humidity, balls trickling over the net, playing better players, playing weaker players). Once you get to twenty-ish games, the ups and downs of pickleball blend into a portrait of you as a player. You now have a resume. Your resume is your computer rating.