CRS Pilot, A Problem with DUPR - 6/23/23
We have completed three weeks of our Computerized Ratings Pilot.
June 6/8 = Practice Week
June 13/15 = 1st Official Week, Scores Entered into DUPR via Phone
June 20/22 = 2nd Official Week, Scores Entered into DUPR via Administrator
We continue to learn many things about Computerized Ratings. Some are good. Some are bad. Let’s explore what we learned in the second official week of our pilot.
A Problem with DUPR
There is an enjoyable aspect to DUPR. With the new algorithm, scores are entered, and you immediately obtain feedback. “My DUPR went from 3.48 to 3.53!!” We now have a handful of groups submitting scores to DUPR via phone after Open Play matches. We have always been able to do this, we just haven’t (as a Club) prioritized this aspect of Pickleball. If you want to enter your scores following Open Play games, you are free to do so. As an administrator, I cannot enter the scores of Open Play games. I need to be present to make sure that all games are played in a fair manner, with good intentions, or I cannot enter your scores.
Three weeks ago, DUPR made the decision to start all new players with a rating of 3.50. This creates a problem.
I won’t go into the details, but from a math/statistics standpoint, the decision to start new players at a 3.50 level is irrelevant once a player plays thirty-ish games. DUPR will adjust up/down and the player will eventually land on a correct rating.
The process of adjusting up/down at a Club level is challenging. Allow me to provide an example. We have one player assigned a 2.55 rating by DUPR following a tournament. This player is now playing in our pilot, playing with/against players who are new to DUPR and are assigned 3.50 ratings. Regardless of talent level, when this player wins, this player gains about 0.09 DUPR points. When this player loses, this player loses about 0.01 DUPR points. To get this player to where this player needs to be, this player has to win about eight matches without losing. To win the eight matches, the players who lose must take a 0.09 ratings hit to help the other player achieve an accurate rating. This is not fair to the other players. Those players should likely take a 0.05 ratings hit, in a fair and equal world.
From a Club standpoint, this situation creates challenges.
We had instances of RED players performing well, winning most games. Their DUPR values increased from 3.50 to 3.60. We had MAROON players perform at a good level against GREEN players, but lose. Their DUPRs decreased from 3.50 to 3.40. Again, all these fluctuations will self-adjust after 30-50 games depending upon the competition players compete against.
Within a tournament environment, DUPR will self-correct over time.
Within a Club environment, there will not be many scenarios where the RED player finally plays against the MAROON player, thereby starting the process of self-correction. The RED player could earn a 3.80 rating playing against RED/PURPLE competition.
DUPR appears to be a highly enjoyable tool that provides immediate feedback following games.
DUPR appears to work very well among players with equal skills. If you have a group of Open Play players with comparable skills, DUPR is going to be a fun tool to use. Use your phone, enter results, watch your rating change in real time.
DUPR created a challenge by starting all new players at a 3.50 level.
We are running DUPR-only events next week. We want to see if we can add matches to DUPR to accelerate the self-correcting process DUPR created by adding in new players at a 3.50 level. At this time, my faith in DUPR is not as high as it was previously because of changes they chose to implement.
Heat
We have been blessed beyond belief with our June weather. Without getting into too many details, our winds originated out of Alaska this month, meaning it hasn’t been 115 degrees.
Next week, that all changes. A portion of the “heat dome” over Texas slides into Arizona, and we will approach 115 degrees by the end of next week.
We will see what playing conditions are like on Tuesday and Thursday. Nobody is going to be forced to play if it is too hot. We will adjust courts and matchups as necessary, and we will gladly send players home if it is too hot. There is no obligation to play “x” games per session. There is an obligation to prevent heat exhaustion.
Once we get to July, we add humidity to the mix. It will feel hotter even if the temperature is the same. We will need to closely evaluate our Pilot, determining how far into the morning we can play under extreme heat/humidity.
If you are sensitive to playing in the heat, please do not sign up for these events. This is a Pilot, we are testing a computer. Your health is more important than the health of a Ratings Computer.
I was asked numerous times this week why we cannot run our Pilot at 6:00am or at 7:45am. Remember, this is a Pilot, it is not an official Program. We want courts for Open Play and Drop In at 6:00am, as this is the best weather of the day. We want courts for Round Robins at 7:45am. Round Robins are an important and established Club activity.
Positive Outcomes from Computerized Ratings
There are three areas where Computerized Ratings are providing value.
Adjustment of players assigned an incorrect Color Level.
Positive adjustment of players who are prevented from playing against better players but consistently win against comparable competition.
Minimal penalties for playing against players 2-3 color levels higher than you.
We clearly have players at the Teal/Red/Purple level who are better than the color level they reside in. Through two weeks, Computerized Ratings add about 0.20 – 0.30 to the ratings level of these players … in essence, proving that these players should be a minimum of one-color level higher than they currently are at.
Computerized Ratings reasonably rectify injustices.
Yesterday, I put up a firewall at Court 4 – you could not go higher than Court 4. This was done on purpose to test the theory that if our better players choose to not participate in Computerized Ratings then good players are not provided with an opportunity to increase their Ratings level. One player on Court 4 won three games and lost one game, jumping 0.30 ratings points. Any jump of 0.25 ratings points equals one color level. The phrase “tick tock, the game is locked” does not hold true in a world where Computerized Ratings are available to assist the player. Of course, in the future the player would have to play against better competition, so self-correction “could” happen. I tested this hypothesis on Thursday. The week prior, one of our players was put on a court where there was a firewall and could not progress. This player won three, lost one, and moved up about 0.15 ratings points. This week, the player was placed with much better players, losing 0.05 ratings points. In total, the system worked. Small sample size? Yes. But the data shows promise.
In the past week, I tested placing players 2-3 color levels below other players WITH much better players. We learned two things.
Play becomes very choppy when there is a three (3) color level difference.
A player playing at a two-level color disadvantage isn’t hurt by “playing up”.
An example. In one case, I had a MAROON/MAROON vs. ORANGE/PURPLE matchup. The difference in average rating was 0.336 ratings points between the teams. That is a big difference. MAROON/MAROON won the game 11-6. MAROON/MAROON earned 0.005 ratings points (going from 3.500 to 3.505). ORANGE/PURPLE lost 0.005 ratings points. In other words, there was almost no gain for the MAROON/MAROON team, and almost no harm to the ORANGE/PURPLE team.
Had an upset happened, ORANGE/PURPLE could have claimed 0.095 ratings points from MAROON/MAROON. That would have been a big deal!
But I Beat Better Players All the Time, Move Me Up!
We all beat better players!
Through two weeks, when there is a one-half to one level color difference between players, the lower-level team wins 34% of the time.
Through two weeks, when there is a one to two level color difference between players, the lower-leveled team wins 18% of the time.
These findings address a key concern I keep hearing from players. You keep telling me “I beat players at higher color levels all the time, therefore, I should be at the higher level.” I bet you are right, you are beating players at higher color levels. You are supposed to beat players at higher levels from time to time.
The issue is “how often are you beating players at higher color levels?” View it this way. If you are a RED player and you have a PURPLE partner, playing a PURPLE/PURPLE team, your team should win 1/3rd of the time. To you, it will feel like you are winning often, and you are. But you should win often … about 1/3rd of the time.
If you are a GREEN player playing with a GREEN player against BURGUNDY/BURGUNDY, you should win about 1/4th of the time. To you, it will feel like you are winning often, and you are. But you should win often … about 1/4th of the time.
The issue isn’t beating players at a higher color level … that is always going to happen.
The issue is “how do you perform over time vs. an expectation?” If you beat players at a higher color level 1/4th of the time or 1/3rd of the time, you are meeting your expectation and a computer will maintain your level. If you are beating players at a higher color level 61% of the time, a computerized ratings process will reward you.
I’ll show you an example. Here is one of our players from this week. This person played four games.
Game One: 0.126 ratings point higher, supposed to win. Player wins. KPR increases by 0.024 points.
Game Two: 0.281 ratings point higher, supposed to win. Player wins. KPR only increases by 0.007 points, even though the player beats a player at a higher color level.
Game Three: 0.230 ratings points lower, supposed to lose. Player loses against two players two ratings levels higher. KPR decreases by 0.011.
Game Four: 0.047 ratings points lower, supposed to lose. Player loses to a player two color levels higher and a player one color level below. KPR decreases by 0.040.
End Result? Player performed exactly as expected. This player won two, lost two, and beat a player one color level above. The KPR decreased … 0.024 + 0.007 – 0.011 – 0.040 = -0.020. If this player was an ORANGE player, the rating would change from 3.250 to 3.230.
In other words, this player remains about the same in terms of KPR … AND this player beat a player a color level above the player. The player can honestly say the player beat somebody at a higher level … but the player did exactly what the computer said the player should do … 2 wins, 2 losses, no change to the KPR.
Through two weeks, a whopping 103 players played in at least one of our two events. Think about that! How many out of 103 players saw their KPR increase by at least 0.20 ratings points (nearly one color level)?
Seven.
What were the color levels of the players who made large jumps?
Purple, Teal, Red, Red, Purple, Orange, Indigo.
In other words, these players might be assigned to the wrong color level, and the computer is correcting the issue via their won/lost record. These players won 33 games and lost 10 games over the two weeks.
Eight players experienced the opposite situation … and in many cases I caused that to happen by placing them in very difficult situations. It was not their fault, it was MY fault!
The other 88 players … 85% of those who participated … did not see a large rating change (to date). We are doing what we are supposed to do – even if that means beating players at higher color levels from time to time.
I digressed into this line of argument for a good reason. All of us are supposed to beat better players from time to time, that’s the nature of competition. As of writing this, the Oakland A’s baseball team has a record of 19 wins and 58 losses. Oakland wins 25% of their games, similar to a RED team going up against a PURPLE team or ORANGE team. Oakland will win from time to time. It doesn’t mean Oakland deserves to be moved up in the standings.
Two-Week Summary
It is my belief that Computerized Ratings are doing two things well, so far.
Addressing injustices.
Measuring/Rewarding progress over time.
If we ever got to a point where our Club used Computerized Ratings, I would not care about weekly performance (as weekly performance will be noisy). I would care about making progress on a monthly basis. If I could increase from 2.75 to 2.85, and the next month increase from 2.85 to 3.02, I would know that I was improving. It would then be our job as a Club to recognize progress.
In my case, I’m confident that I would have seen the opposite happen in the past three months. My feet have been sore, and I can’t move like I’d like to move. My Computerized Rating would likely have dropped from 4.25ish to maybe 4.00ish. It should drop – I haven’t been able to play as well as I’d like to play.
We continue to learn via our Computerized Ratings Pilot. After next week we will have a better read on whether DUPR and their decision to start all new players at a 3.50 level can be overcome. We will have three weeks of KPR data to evaluate longer-term trends among players who play all three weeks.
Thank you to all 103 players who have selflessly played in warm temperatures to test Computerized Ratings. You have given up your time trying to help our Club assess something. Not every Pickleball Club is willing to test a new idea. Not every Pickleball Club is willing to be patient while going through growing pains. Not every person is willing to pick up a broom and clean up a mess, some would rather point out where the dust bunnies are hiding and expect somebody else to do the work. You are picking up a broom and are doing the work.
I’m proud and appreciative of all of you for the work you’ve done to help us this summer!