CRS Pilot, How Important is Winning? - 7/27/23

We have completed seven weeks of our Computerized Ratings Pilot.

• June 6/8 = Practice Week

• June 13/15 = 1st Official Week, Scores Entered into DUPR via Phone

• June 20/22 = 2nd Official Week, Scores Entered into DUPR via Administrator

• June 27/29 = 3rd Official Week, All Players Must Participate in DUPR

• July 11/13 = 4th Official Week, Thursday Women’s Session Cancelled due to Heat

• July 18/20 = 5th Official Week, Team-Based Events

• July 25/27 = 6th Official Week

• Upcoming August 1/3 = 7th Official Week, PICK YOUR PARTNER WEEK

We have learned A LOT this summer.

• 139 players have played at least one game between June 13 and July 27.

• If a player is mis-classified, the computer can move players at 0.25 point increments within 1-2 weeks.

• It takes the computer 3-7 weeks (12-28 games) to move other players 0.25 ratings points. This doesn’t mean the player stays at the higher level.

• About 75% of our players are moving +/- 0.15 ratings points (i.e., they are staying approximately where they started off).

• We continue to measure DUPR, but DUPR ratings are not adjusting fast enough for broad use by a club our size.

• The purpose of our pilot is to test the computer. Many of you are put in challenging spots, thereby artificially hurting/helping your computer rating. After 5-6 sessions, these biases start to even out. Once a player hits about 20 matches, we begin to see a more reasonable story of how that player compares to other players.

• Upsets continue to happen at a consistent rate. Upsets have happened at consistent rates throughout the pilot.

• Lower Color Level / Higher Color Level vs. Higher/Higher = 30% chance of an upset.

• Lower Color Level / Lower Color Level vs. Higher/Higher = 20% chance of an upset.

It Is SO HOT Out There!

We checked the temperature on the courts this week, at 10:00 am. Court temperatures were 120 degrees. Court temperatures averaged about 20 degrees warmer than the ambient temperature.

This has been the warmest four-week stretch ever measured in Phoenix history. We cut back on the number of games played to help our volunteers. Even in this heat, we had 30 men and 14 women play between three and four games in the past week. That’s impossible!

We look forward to a cooler week next week. However, the long-range forecast shows us returning to 111-115 degree temperatures the following week. If you are bothered by the excessive heat, please do not sign up to play. I will keep our schedule “light” in response to the weather, with no more than 3-4 games in a session and plenty of rest.

Also – thank your Board for getting additional shade in front of Court 2. That’s been a positive aspect of our summer play.

How Important is Winning to a Computer Rating?

Winning is important. Winning against better players is really important. Losing against better players doesn’t really hurt you.

I looked at all players who have had KPR increases of 0.20 or more in our pilot. Look at the wins/losses accrued by our players.

• 6 wins, 6 losses

• 8 wins, 7 losses

• 15 wins, 12 losses

• 15 wins, 11 losses

• 11 wins, 8 losses

• 9 wins, 5 losses

• 8 wins, 4 losses

• 7 wins, 3 losses

• 7 wins, 2 losses

• 4 wins, 1 loss

• 11 wins, 1 loss

• Average = 9.2 wins, 5.5 losses

Players who boosted their KPR by at least 0.20 ratings points played an average of about 15 games, winning 9.2 games and losing 5.5 games.

This is an important finding. The player does not have to be perfect to see an increase in the computer rating. If you lose a game, you lose a game. In fact, the players with large increases in KPR won just 63% of their games, on average. This is a “Win 5, Lose 3” pace. It is a pace that results in a player scoring an average of 9.01 points per game. Clearly, the players who saw their KPR increase by at least 0.20 points weren’t perfect (wins/losses), but they were very competitive.

Is The Pilot Biased by “Winning” Players?

A little bit, yes.

The data suggests that some (not many) players get discouraged when their KPR decreases too much, while other players play more often as their KPR increases. Here is what the relationship looks like:

Players playing 20-25 games saw their KPRs increase by an average of about 0.035 points. Not much. But they were seeing, on average, positive results.

Meanwhile, players playing 1-5 games saw their KPRs decrease by an average of about 0.02 points.

There is a potential drawback to a computer rating … for some players to increase their computer rating there have to be players who lose games and suffer a computer ratings decrease. If we decide to implement computer ratings, we need to encourage participation among a large cross-section of players. One can envision a world where a player loses four out of five games, sees their computer rating drop from 4.00 to 3.88, and stops participating in competitive play in an effort to “protect” their computer rating.

When colors were implemented, we learned that many players wanted to move “up”. This constant desire to improve skills and be rewarded with a higher rating should keep participation at reasonable levels. Once a player is happy with the alignment between skills and rating and social inclusion, there won’t be an incentive to participate in a computer rating process.

Kevin, You Sure Don’t Talk About DUPR Anymore!

You are correct.

I was enthusiastic about DUPR, in large part because our ratings would be equalized across communities. In other words, a 3.77 DUPR at PebbleCreek would have the same meaning as a 3.77 DUPR over at the former Bell Bank Park.

Unfortunately, DUPR changed how they rate players. They decided to introduce all new players at a 3.5 level. Imagine if we had 1,350 players all evaluated at the same level? It would be a hot mess.

Many of our players are not participating in DUPR this summer, so we are not getting a lot of DUPR-entered games to compare progress of their ratings system. In a perfect world, their system would quickly move a 3.50 player to 2.75 or to 4.25 … getting the player to the right spot.

To simulate what might happen if every one of our games were DUPR eligible and assigned a starting point of 3.50, I simulated your KPR through 7/27 assuming you’d start with a KPR of 3.50. Below is a histogram of results.

95% of our players ended up somewhere between 3.25 and 3.75. However, by playing competition similar to your ability, you can see your simulated DUPR move up to 3.75 while a 4.25 player can see their simulated DUPR move down to 3.25. That’s not how we want the process to work.

The key appears to be a “fair” (not perfect) starting point. I realize many of you detest color levels in a deep and personal manner few have previously considered. Trust me, I’ve heard your thoughts. Duly noted. But in this pilot, you may have noticed that we used colors as our starting point.

• 2.50 = Teal

• 2.75 = Red

• 3.00 = Purple

• 3.25 = Orange

• 3.50 = Maroon

• 3.75 = Indigo

• 4.00 = Green

• 4.25 = Burgundy

• 4.50 = Aqua

What’s interesting about having a “fair” starting point is that most players stay somewhat close to the fair starting point. Fifty-six (56) players have played ten or more games in our pilot.

• 8 of the 56 players … 14% … moved up at least 0.25 ratings points (one color level) since starting the pilot.

In other words, the pilot is demonstrating the value of a computerized rating system. All of your hard work is paying off!

If you were properly classified by the color system, the pilot is proving that you’ll wobble +/- 0.20 from where you started.

If you believe the color system incorrectly classified you, the pilot is proving that a computer rating gives you control over your situation. With the appropriate level of talent, you should be able to move up. Players in our pilot have done just that – and on a small number of games, no less.

If you believe the color system incorrectly classified you and the computer is not moving you up, well, that means you might have a more ambitious view of your skills than is reflected via game play against comparable competition. You won’t have a rater to yell at. You will have to win games. If you aren’t winning enough games after 10-25 attempts, it will not be the fault of partners, or balls hitting the tape, or wind, or anything else. It will mean that the computer is properly evaluating your play. These will likely be uncomfortable situations for some of our members. I know I’d love to be a 4.5 player. I am not a 4.5 player. If I were playing in this pilot, the computer would not reward my play. I’d have to come to the conclusion that my perception of my skills did not match my actual skills.

Our pilot is verifying an awful lot of positives, and a few concerns. I am confident that a computer rating will give our members more control over their advancement. With additional skills gained, our members should be able to use a computer rating to quickly move up where appropriate. Your hard work gives me confidence in this process. I’m so appreciative of the 332 games you’ve volunteered to play. You gave us a lot of information to verify if a computer rating is appropriate for a club our size.