News & Features

My Players

|

My Lists

|

My School

|

My Notifications

Team Home

Since 2015, Tennis Recruiting has analyzed the performance of tennis ranking and rating algorithms for 17 different tennis tournaments - ranging from USTA junior sectional championships to professional Grand Slam events. These tournaments include over 20,000 match results, and a summary of our findings can be found here. Our most recent analysis came from the 2016 Easter Bowl in April. These studies investigate how well the various tennis rating systems predict winners and upsets.

When we release one of these analysis articles, the most common question we field is, "What do you mean by an upset?" There is some confusion as to which matches are labeled upsets in these studies - and also what constitutes a real or true upset.

This article tries to address the question of "What is an upset?" with help from the data we have collected.

Simple Definition

One simple definition of an upset is any match where a lower-rated player beats a higher-rated player. This is a meaningful position, and our analysis articles make use of this definition because it provides a common framework for evaluating the same results across different ranking and rating systems.

While this definition looks easy to understand, it becomes more complicated when we realize that there are different rating systems that rank players differently. A player who is highly-rated by one system might be rated lower in another.

When addressing the question, "What is an upset?" subjectively, we believe that this simple definition is too broad and includes matches that most people would not think are real upsets.

With that in mind, let's dig deeper ...

Perspective from March Madness

To frame the discussion, consider the first round games from the "March Madness" basketball tournaments. The tournament has four regions with 16 teams that are fully seeded - from #1 down to #16. There are first-round matchups between evenly-matched opponents (like when a #8 seed plays a #9 seed) and games intended to be lopsided (#1 vs. #16).

At one extreme, no #16 seed has ever beaten a #1 seed - the #1 seed is a perfect 128-0 in 1v16 matchups. When a #16 seed finally defeats a #1 seed, it will be big news - virtually everyone in the country will consider it to be a huge upset.

On the other hand, no one is surprised when a #9 seed beats a #8 seed. In fact the all-time win-loss record of #9 seeds against #8 seeds is 72-80 (47.4%). The #8s and #9s are evenly matched - and hardly anyone considers it an upset when the lower-seeded team wins.

What about something in the middle? Say, the #6 seed vs. the #11 seed? Most people would consider a win by the #11 seed to be an upset, but it is not all that uncommon. The #11 seeds are 52-96 all-time against #6 seeds, winning more than a third of the time. The #11 seed has had lots of success recently, winning five of the eight matchups with the #6 in 2015 and 2016. Since such upsets are common, many people spend time in March on their brackets guessing which double-digit seeds might advance.

In the NCAA basketball tournament, there are:

• some matchups where there is a clear underdog,
• some where you would flip a coin,
• and some matches in between.

We will use these same three categories when we look at tennis ...

The Heavy Favorite Versus the Clear Underdog

Let's take a look at the tennis data we collected - starting with matches that have a clear underdog. The Tennis Recruiting rating system gives win probabilities for each matchup, and there are some matches where the favorite has a 90% chance or better to win. Most people agree that there is an upset when the underdog wins such a match.

Of the 20,000 matches in our studies, more than 6,000 fall into this "clear underdog" classification - where one player is expected to win 90-100% of the time. Looking specifically at these 6,641 matches, our system actually expected the favorite to win 96% of the time, and we call this figure the Expected Win Percentage or EWP. Here's how the various rating systems did for these 90-100% EWP matches:

EWP Range	Matches	How Often We Expected Favorite to Win	How Often Favorite Actually Won
			TRN	UTR	USTA
90‑100%	6,641	96.0%	95.4%	95.2%	89.3%

The table shows how the Tennis Recruiting rankings (TRN), Universal Tennis Ratings (UTR), and USTA Points-Per-Round (USTA) systems performed when predicting the winners of these matches. All three systems make similar predictions, and all three systems call the matches correctly the vast majority of the time.

Let's also take a look at the UTR definition of an upset. Universal Tennis considers a match an upset whenever a player defeats an opponent rated more than one full point higher on the UTR scale. We again think most people would agree that these are genuine upsets. Looking at the same data, 7,947 matches fall into this category. Take a look at how the three systems performed when predicting these matches:

UTR Difference	Matches	How Often We Expected Favorite to Win	How Often Favorite Actually Won
			TRN	UTR	USTA
> 1.0	7,947	92.5%	91.9%	91.7%	83.3%

Both the TRN and UTR systems predicted more than 91% of these matches correctly, and the USTA system chose the winner more than 83% of the time as well.

However you choose to slice the pie, wins by the lower-rated player in these situations don't come easy - or particularly often. When considering the total number of 20,000 matches, these upsets only occur about 1.5% of the time. There seems to be consensus that these wins are upsets.

Flip a Coin

So there is strong agreement on matches between players with very different skill levels, but what about players whose levels are very close? Let's consider the round of 64 match-up in the 2015 US Open between Steve Johnson and Thomaz Bellucci. Here is how the various systems rated these two players:

Player	ATP	UTR	TRN
Steve Johnson	1,240 Points	15.44	51.3% EWP
Thomaz Bellucci	1,105 Points	15.52	48.7% EWP

While all three systems rate these players closely, they disagree. Johnson was rated higher by the ATP and TRN, but Bellucci was rated higher by Universal Tennis. So at least one of the systems was going to be wrong - independent of who won the match. Steve Johnson did end up winning, but almost no one would consider this result an upset. You might as well toss a coin. This scenario is very similar to the #8 versus #9 basketball games discussed above.

So let's take a look at all the matches where the higher rated player had a 50-60% chance of winning. A little over 3,000 matches fell into this category, and our EWP for these matches was only 55%.

EWP Range	Matches	How Often We Expected Favorite to Win	How Often Favorite Actually Won
			TRN	UTR	USTA
50-60%	3,038	55.0%	54.8%	53.8%	52.1%

None of the systems reached even this low 55% threshold. To put that into perspective, picking winners randomly would result in correct predictions about 50% of the time, so none of the systems did well for this pool of matches. Even though the lower rated player won almost half the time, not many would view these wins as upsets.

We see similar results if we look at players rated similarly by the UTR. The next table shows results for players rated within two-tenths of a point of each other on the UTR scale.

UTR Difference	Matches	How Often We Expected Favorite to Win	How Often Favorite Actually Won
			TRN	UTR	USTA
< 0.2	2,793	63.8%	60.9%	59.3%	55.8%

While this pool of matches does not seem to be quite as hard to call as the group listed above, none of the systems excelled at predicting the winners. Again, not many people would view these matches as upsets.

The Other Half

The above classifications by expected win percentage and UTR difference account for about half of the overall matches we examined. What about the other 10,000 matches or so in between? Do some of those matches fall into the upset category?

One of the best examples that a subscriber pointed out to us last year was Serena Williams' US Open semifinal match in her quest to achieve the calendar-year grand slam. Our model gave Williams a 86% chance of beating Roberta Vinci, and she was not rated a full UTR point above her opponent. This match does not fall into either the 90-100% EWP or >1.0 UTR categories listed above, but many viewed this win to be one of the biggest upsets in women's tennis history. So ... are the 90-100% EWP and >1.0 UTR classifications too exclusive?

Let's take a look at the data for these matches in the middle. First consider all the matches where the favorite has between a 60% and 90% chance of winning.

EWP Range	Matches	How Often We Expected Favorite to Win	How Often Favorite Actually Won
			TRN	UTR	USTA
60-90%	10,363	75.9%	73.9%	71.7%	63.4%

We expected that our algorithm would pick the correct winner 76% of the time, but notice that all of the systems fell a little short of this mark.

We also look at the numbers for matches between players with a UTR difference of between 0.2 and 1.0 points.

UTR Difference	Matches	How Often We Expected Favorite to Win	How Often Favorite Actually Won
			TRN	UTR	USTA
0.2-1.0	9,300	72.9%	71.5%	70.3%	63.0%

The results look similar to those from the 60-90% EWP group.

With all this in mind, what do we think? What is the best definition of an upset?

One Verdict

Many answers are possible when defining upsets, but if we have to pick one, we would probably settle on the following definitions:

An upset occurs when:

• the winner has an EWP of 0-30% - or less than a 30% chance of winning the match.
• the winner has a UTR more than 0.5 points lower than his/her opponent

Why 30%? That number expands the pool of potential upsets to more than 13,000 matches, and the lower-rated player wins just under 2,000 of these matches. With this definition, the total number of upsets goes to just under 10% of the 20,000 total matches.

Upset Definition	Matches	Upsets	How Often Underdog Wins
			(Subset)	(Total)
< 30% EWP (TRN)	13,871	1,873	13.5%	9.3%
> 0.5 UTR Difference	13,334	1,954	14.7%	9.7%

An upset occurs often enough to be noticeable - but not so infrequently as to be a shooting star. Again, there is no real right or wrong answer to this question, but this our best attempt at an answer.

What do you think? Where do you draw the line between an unexpected win and an upset? We would love to hear from you in the comments below.

All the data used to compile the information in this article is available from our Analysis of Junior Rankings and Ratings.

Leave a Comment