Monday, April 8, 2013

Empirically Verifying Course and Slope Ratings


Most golfers have played a course where they thought the Course Rating and/or the Slope Rating were too low.  These ratings errors would explain why their scoring differential was so much higher than their index.  The golf association responsible for the ratings, however, would argue the ratings are accurate and the player’s poor performance is more properly explained by the random nature of scoring.[1] 
Course and Slope Ratings are determined through the use of estimating models.  Estimating models in both the hard sciences and the social sciences are validated by how well they predict.  There are models, for example, that attempt to predict the winning team in games of the National Football League.   If the predictions of a model are better than chance over a long period of time, the model is said to be empirically validated.  Interestingly, the United state Golf Association (USGA) has never published any evidence validating it Ratings models.
There are probably two reasons for the USGA’s lack of diligence.  First, a research finding that the models are inaccurate would tarnish the image of omnipotence  the USGA likes to project. Secondly, there are severe theoretical and logistical problems in designing an experiment to verify the ratings. These problems have been discussed in a previous post – How Accurate is the Slope System? [2]  In theory, a large group of scratch players could play a course many times and it would be possible to solve for the Course Rating that would make these players scratch at this course.  It is not known with any certainty, however, that these players are actually scratch or came about that handicap due to an error in the Course Rating at their home courses.  The logistical problem is getting a reference group (i.e., the scratch players) at least 400 (a minimum of 20 players having 20 rounds each) starting times at the test course. Because of these problems, golf associations cannot be easily challenged on the accuracy of their Ratings. 
There is one special case, however, that eliminates the logistical problem and allows for the examination of Rating errors.  That case is where a golf club has two or more courses.  By examining the performance of the same golfers on different courses, it should be possible to estimate the size of any errors in the Ratings.    Slope Theory assumes the average differential ((Average Score – Course Rating) x 113/Slope Rating) at each course should be the same for a player.  To test the hypothesis that a player’s differentials are equal, data were collected on 34 four players who alternate playing two courses under competitive conditions.  The courses are named Course1 and Course2 in this analysis.  A player’s Average Course1 Differential is plotted against his Average Course2 Differential in Figure 1 below.


  Figure 1 – Average Course1 Differential versus Average Course2 Differential

If Slope Theory is correct the regression line should have a zero intercept and a slope of 1.0.  In other words the best estimate of the Average Differential on the Course 1 is the Average Differential on the Course 2. The regression line fitted to the date shown in Figure 1 was estimated to be:
                1)            Avg. Course1 Differential = 2.1 + .85 · Avg. Course2 Differential
The regression does not have a zero intercept.  The constant “2.1” in the regression equation, indicates a possible error in the Course Ratings.[3]
Rewriting the equation 1,
                2)            Avg. Course1 Differential - .85 · Avg. Course2 Differential = 2.1
To eliminate the constant, 2.1, the average differential on Course1 must be decreased or the average differential on Course2 increased or some combination of the two.  A reasonable assumption is that the error in the fixed component should be split equally between the two courses.  Therefore, the Average Course1 Differential is reduced by 1.0 (i.e. Course1 is more difficult than it is rated), and the Average Course2 Differential is increased by 1.2 (i.e., Course2 is easier than it is rated).  The plot of these adjusted differentials for each player is shown in Figure 2 below.
Figure 2 – Average Course1 Differential Reduced by 1.0 versus Average Course2 Differential Increased by 1.2
The new regression equation is:
                3)            Average Course1 Differential = 0.03 + .85 Average Course2 Differential
The fixed component of the relationship is now close to zero.  The ratio of the two differentials, however, should be 1.0 and not .85.  To achieve that end, the Slope Rating of Course2 would have to increase or the Slope Rating on Course1 decrease -- or some combination of the two.  There are a large number of new Slope Ratings that would equalize the ratio of adjusted differentials.[4]  Again, it is assumed the errors in Slope Ratings are approximately the same size though in different directions.   The postulated Slope Ratings under this assumption are 115 for Course1 and 132 for Course2.  The plot of the differentials adjusted for the new Slope Ratings is shown in Figure 3.  The new regression line is:
                4)         Average Adjusted Course1 Differential = 0.2 +.98 Average Adjusted Course2    Differential
The new regression line eliminates the fixed component, makes the ratio of the two differentials approximately equal to 1.0, and therefore satisfies the requirements of Slope Theory.


Figure 3 – Plot of Differentials Adjusted for New Slope Ratings

Given these adjustments, what would be the estimates of the new Course Ratings?  To reduce the Average Differential on Course1 by 1.0, the Course Rating must be increased by 1.0 x 124/113 or 1.1 strokes.  The new Course Rating on Course1 would be 71.1 (from 70.0).  To increase the Average Differential on Course2 by 1.2, the Course Rating would have to be decreased by 1.2 x 122/113 or 1.3 strokes.  The new Course Rating on Course1 would be 68.3 (from 69.6).[5]  A summary of the estimated and existing Ratings are presented in the table below.

Table
Estimated and Existing Ratings
Course
Course Ratings
Slope Ratings
Estimated
Existing
Estimated
Existing
Course1
71.1
70.0
115
124
Course2
68.3
69.6
132
122

The estimated ratings in Table 1 are purely speculative.  There are a large number of Ratings that would equalize player differentials between the two courses.  It does appear, however, that the current Ratings are not correct.  There are three possible explanations for the apparent errors:
1. The Ratings are in Error – As discussed in How Accurate is the Slope System?, the Course Rating models of the USGA are seriously flawed.  The models are deemed so imperfect the results can be changed by the Course Rating Review Committee at each golf association. [6]
2. Sampling Error – The small size of the sample (34 players) makes any conclusion tentative at best.

3. Slope Theory Is Not Valid - Under Slope Theory, a players’ average score at a course is a linear function of his index. All players have their index increased (or decreased) by the same percentage (Slope Rating/113) to find their handicap. But what if a player’s average score is a curvilinear function of his index? In that case, even if the Course Rating, Bogey Rating, and Slope Rating were estimated with precision, the average differentials at the two courses would not necessarily be equal.
There is anecdotal evidence that reason 1 is the most plausible explanation.  Individual and team (2 best balls of 4) net scores are generally lower on Course2 than on Course1.  The current Ratings would not predict such a disparity in net scores.   Reason 2 can only be eliminated by expanding the sample size.  It would be helpful if a golf association with access to the General Handicap Information Network (GHIN) would undertake a similar study.  Since this is unlikely, the most reasonable course of action is to repeat the experiment each year and examine whether the results are consistent across time.    




               



[1] The USGA maintains the standard error in the Course Ratings is .285 strokes, and therefore an error of more than .6 strokes is extremely unlikely.  (See Knuth, D, “A two parameter golf course rating system,” Science and Golf, edited by A.J. Cochran and M.R. Farally, E & FN Spon, London, 1990, p. 145.)
[2] See Ongolfhandicaps.blogspot.com, October 8, 2012.
[3] The constant 2.1 is only statistically significant at the 90 percent level of confidence. The variable coefficient (.85) is statistically significant at the 95 percent level of confidence.
[4] The current Slope Ratings are 122 for Course2 and 124 for Course1.  The ratio of the adjusted Course1 Differential to the Adjusted Course2 Differential is .85.  To make the ratio of the adjusted differentials equal to 1.0, the ratio of the new Course2 Slope Rating to the new Course1 Slope Rating must be approximately 1.16.
[5] The errors seem large and the USGA would argue they are not within the realm of reason.  What is often forgotten is that a small error in the Course Rating can lead to what looks like a substantial error in the Slope Rating.  Let’s look at the Course2.  The Slope Rating is given by the formula:
                Slope Rating = 5.381 (Bogey Rating – Course Rating)
If the Course Rating is overestimated by 1.3 strokes, the Slope Rating will be underestimated by 7 rating points.  That would put the Slope Rating at 129 which is not too far from the 132 used in this analysis.
[6] USGA Handicap System, 2012-2015, Unites States Golf Association, Far Hills, NJ, p. 98.

Monday, March 25, 2013

The USGA Bogeys Its Explanation of the Slope System


Previous posts (e.g., see The Mathematical Underpinnings of the "New" Slope Rating, February 4, 2013) have documented the confusion at the USGA about the Slope Rating.  This post takes another whack in the spirit of flogging a dead horse.  Let’s start with documenting the USGA’s arguments as put forth by Dean Knuth, former USGA Director of Handicapping:

The USGA coined the term which numerically describes the difference in course rating difficulty between the bogey and scratch players as the “Slope Rating.”…Slope Rating is the slope of the regression line of total score versus USGA Handicap for a particular golf course.  The Y-intercept is the USGA Course Rating which is the better half of score average of scratch golfers… The slope of the scores line of an average course has been observed to be 1.13 and the USGA Slope Rating is referenced as 113 to deal with whole numbers.  A scratch golfer will score proportionally fewer strokes than a 10, 20, and 30 handicapper.  The slope of the line through their score indicates the difficulty of the course, and the slope is steeper for more difficult courses and less steep for a less difficult course.[1] (Note: USGA Handicap refers to a player’s handicap before the introduction of the Slope System).

 A minor error is that Knuth uses the regression line of “total” score versus USGA Handicap.  He probably meant to use “average” score as shown in the work of Stroud and Riccio.[2]  (The term “total” score has no obvious meaning, so “average” score is substituted for the purposes of this post.)  The larger error is the assertion that the slope of the regression line of average score versus USGA Handicap will be the Slope Rating.  This is not the case.  The slope of this regression line at any and every course should be 1.13 if Stroud and Riccio are correct.

Why isn’t the slope of the regression line steeper at more difficult courses as Knuth maintains?  The answer is straightforward.  If a player only plays his home course, his handicap is independent of the Slope Rating.  The relationship between average score and USGA Handicap is only a function of the procedures for reducing an average score to produce a handicap (i.e., taking only the ten best scores out of twenty, equitable stroke control, and the 96 percent factor).

If Knuth was correct, a Slope Rating would have an absolute value.  Raters could simply use the regression line relating average score and handicap in order to estimate the Slope Rating.  But the Slope Rating only has relative value.  The Slope Rating is dependent on the Slope Rating of the reference course.  What Knuth should have done is taken players from the reference course and had them play the course in question.  He should have then plotted the player’s average score versus his handicap on the reference course.  The Slope Rating would then be the slope of the regression line relating average score versus the handicap of the player from the reference course.

Knuth compounds his error by arguing the Y-intercept of the regression line is the Course Rating.  The Y-intercept, however, is the “average score” of a player with a scratch handicap.  The Course Rating however is the average of a player’s ten best scores out of the last twenty.  The Y-intercept is clearly not an unbiased estimate of the Course Rating.

Several conclusions can be drawn from this critique of USGA research:

1. The research papers appear to have been dashed off in order to qualify for a conference at St. Andrews in July of 1990 (e.g., greek symbols were entered by hand).  I suspect the authors were more concerned with locking down their tee times on the Old Course than quality control.  Why else would members of the USGA Research Team (Knuth, Riccio, and Stroud) arrive at such disparate reasons for the ubiquitous number 113 that pervades the USGA Handicap System?  

2. The Conference editors (Cochran and Farrally) should have been responsible for the quality of the papers presented.  They overlooked this duty, probably for monetary reasons.  I suspect the USGA, a major financial sponsor of the conference, could have submitted the menu from Rusack’s Hotel and it would have been accepted.

Well that’s enough flogging for now. The next post will examine the difficulty in empirically verifying Course and Slope Ratings.





[1] Knuth, D, “A two parameter golf course rating system,” Science and Golf, edited by A. J. Cochran and M. R. Farrally, E & FN Spon, 1990, pp. 142-143.
[2] Stroud, R.C. and Riccio, L.J., “Mathematical underpinning of the slope handicap system,” Science and Golf,  edited by A. J. Cochran and M. R. Farrally, E & FN Spon, 1990,  pp. 135-140.

Thursday, March 7, 2013

What a Difference an ESC Makes


Introduction - A previous post detailed the various equitable stroke control procedures (ESC) implemented by the United States Golf Association (USGA) over the past twenty years--See What Does the Equitable Stroke Control Procedure Mean for Golf? February 13, 2013.  This post presents an analysis of the impact of different ESCs on handicaps and the overall equity of competition.  A simulation model was used to estimate the effects of different ESCs on adjusted scores and handicaps.  It was found that the latest ESC (ESC97) reduced the advantage of the low-handicap player over the high handicap player.  This conclusion was affirmed in an empirical test using a small sample of players. 

Simulation Model - The first step in creating the simulation model was to estimate the probabilities of hole scores relative to par for golfers of varying ability.  These probabilities were estimated first by drawing from a small sample of scorecards, and then reviewing and adjusting these estimates after discussions with numerous players.  The results are dependent on these estimates.  The overall conclusions of this analysis, however, were found to be fairly insensitive to moderate changes in the estimated probabilities.  The probabilities for a low-handicap (5-handicap), a medium-handicap (15-handicap) and a high-handicap (25-handicap) are shown in Table 1.
Table 1
Probability of Hole Score Relative to Par 

Score Relative to Par
Low
Medium
High
-2
.005
.000
.000
-1
.140
.060
.010
0
.450
.250
.150
+1
.310
.430
.380
+2
.070
.200
.300
+3
.020
.040
.100
+4
.005
.020
.050
+5
.000
.000
.010


Each player had 1000 simulated rounds. His score was adjusted after each round under the three ESC formulas: 1) OLDESC, 2) ESC93, and 3) ESC97.  The players’ average score and adjusted scores are presented in Table 2.
Table 2
Average Scores 

Player Handicap
Average Score
OLDESC
ESC93
ESC97
Low
78.77
78.26
77.98
78.26
Medium
89.36
87.91
88.86
88.86
High
99.43
98.09
99.02
99.02
 

As Table 2 shows, the adoption ESC93 lowered the adjusted score of the low-handicap player (.28 strokes), but raised the adjusted scores for the medium-handicap player (.95 strokes) and the high- handicap player (.93 strokes).  After complaints by low-handicap players, ESC93 was dropped and ESC97 was adopted.  (ESC93 limited the low-handicap player to a maximum of 6-strokes, while ESC97 allowed him to take a double bogey on any hole).  This brought the low-handicap player back to where he was under the OLDESC.  The low-handicap player, however, still lost approximately one-stroke vis-à-vis the medium and low handicap player. 
Handicaps were computed after each 20 rounds.  The average handicap for each player under the three ESCs is shown in Table 3.
Table 3
Average Handicap (Course Rating = 70.0) 


Player Handicap

OLDESC

ESC93

ESC97
Low
5.34
5.14
5.34
Medium
14.60
15.19
15.19
High
23.92
24.52
24.52
 

Comparing handicaps under the OLDESC and ESC97, the low-handicap player lost approximately .6 stroke advantage in handicap to both the medium- and high-handicap player.

One measure of over-all equity is the difference between a player’s average differential (average score - course rating of 70,0) and his handicap. These differences are shown in Table 4.

Table 4
Difference in Average Differential and Handicap

Player Handicap
OLDESC
ESC93
ESC97
Low
3.43
3.63
3.43
Medium
4.76
4.17
4.17
High
5.51
4.91
4.91

As Table 4 indicates, the low-handicap player has a handicap closer to his average differential, and therefore has an advantage over the medium- and high-handicap player.  ESC97, however, has reduced the size of the advantage.  The relationship between average differential and handicap can be represented in equation form for each ESC procedure:[1]

OLDESC                Average Differential = 2.90 + 1.11 x Handicap

ESC93                    Average Differential = 3.25 + 1.07 x Handicap   

 ESC97                   Average Differential = 3.01 + 1.08 x Handicap

The equations show the adoption of ESC97 decreased the penalty against the higher handicap player by 3 percentage points.

Empirical Test - To test if the results of the simulation model  are confirmed by real world results, a sample of 21 golfers was used. This sample was not random.  It consisted of players known to the author and celebrity players such as John Paul Newport, golf columnist of the Wall Street Journal, George Peper, former editor of Golf Magazine, and Dean Knuth, former USGA Senior Director of Handicapping.  The average differential ((score -course rating) x 113/Slope Rating) of each player was regressed against their index (2/15/2013) as shown in the figure below. (Note: If the ESC equations above are multplied by 113/Slope Rating, they become identical in form to the equation shown in  the figure below).


This empirical test shows the low-handicap player has a 9.5 percent edge as opposed to the 8 percent edge predicted by the theoretical model.  Both the theoretical and empirical models, however, predict a smaller edge for the low handicap player than the 13 percent edge assumed in the development of the Slope Handicap System when OLDESC was in force.  Clearly, the small sample size limits the confidence in the estimate of the coefficient (i.e., .095).  Since ESC97 increased the cap on scoring for medium- and high-handicapper players,  a reduction in the edge given to the low-handicap player was inevitable.  Therefore, it is reasonable to conclude ESC97, as opposed to OLDESC, made competition more equitable, but not equitable. How golfers have reacted to this inequity will be the subject of a forthcoming post--In Defense of Sandbagging.

 

 



[1] These equations were determined by finding the trendline among three data points.