Monday, October 8, 2012

How Accurate is the Slope System?



 Introduction - The introduction of the Slope System to golf handicapping has given the illusion of scientific accuracy.  Players have an index calculated to the first decimal.  Rather than being a 10-handicap as in the old days, one is now a 10.4 index.  The decimal gives the impression of increased accuracy which may not be deserved.

When a player finds his or her handicap on a slope conversion table, another level of deception occurs.  The table emits an aura of authority with it rows and columns of neatly printed numbers.  It may not occur to the player with a 10.4 index why he should get a ten percent increase in handicap when the slope of a course goes from 114 to 115.  The player may not ask if a course can be rated with such precision to distinguish between a 114 and a 115 Slope Rating.  Has the Slope System contributed to the precision of golf handicaps or merely added another layer of mathematical obfuscation?

The research presented here attempts to measure any contribution the Slope System has made in decreasing the error in estimating golf handicaps.  The Slope Rating of a course is estimated by combining estimates of the Course Rating and the Bogey Rating.  To understand the errors in the Slope Rating, the errors in estimating the Course and Bogey Ratings are examined in turn.

 Error in the Course Rating - To estimate the uncertainty stemming from errors in estimating the Course Rating, it is necessary to examine the USGA course rating model.  Since the USGA has steadfastly refused to release either statistical information on model estimation or the raw data,[1] much of what follows has to be based on informed guesswork.

By definition, the Course Rating should be the average of the better half of 20 scores submitted by a “scratch” golfer.  This suggests that if a statistically significant number of rounds by scratch golfer were submitted, a very good estimate of the Course Rating could be made.  Unfortunately, you cannot tell if a player is scratch without knowing the Course Rating.  And you cannot determine the Course Rating without knowing if a player is scratch.

            The USGA gets around the “chicken and the egg” problem by assuming that players in the United States Amateur Championship are scratch players.  The model is estimated by using the average of the better half of their scores.[2]  This methodology underestimates the Course Rating.  The USGA is taking the low 144 scores out of 288 competitors.  But handicaps are based on the best 10 out of 20 scores.  To more accurately reflect the handicapping system, the USGA should have randomly selected groups of twenty players, and then taken the lowest ten scores from each group.  This sample selection method should have been used to estimate the coefficients in the USGA Course Rating Model discussed below.[3]

            The USGA course rating system is based on the following model:

Scratch Average Score = a + b•Yardage +c1(R1-S1) + c2(R2-S) +

                                                …C10(R10-S10)

Where,

                   Scratch Average Score =   Average of the better half of the scores

                                                                  of 288 players in the U.S. Amateur

                                           Yardage =   Hole length

                                                      Ri =   ith Obstacle factor

                                                      Si =   Reference value for the ith obstacle

                                                                  factor

The USGA used a form of least squares regression analysis to estimate the coefficients of the Course Rating Model.[4] 
                  

            The USGA model assumes yardage makes the same contribution to scoring over all ranges of hole length.  This implies that adding 50 yards to a 180 yard hole would lead to the same marginal increase in the average hole score as adding 50 yards to a 500 yard hole.  An alternative hypothesis is that yardage has a different effect depending on hole length.  One test of this hypothesis would be to take the USGA data and estimate separate equations for par 3, par 4, and par 5 holes.  If the coefficient of the yardage variable was approximately the same in all three equations, then the USGA model would be validated.  If the coefficients were different, then a specification error of unknown proportion has been introduced into the estimate of the course rating.

The USGA claims an unbiased estimate of the hole-by-hole error of predictions is:[5]

                                                                   s =   ((SSE/(N-12)).5

                                                                    s =   .067

Where,

                                                                   s =   Hole-by-hole standard error of prediction

                                                             SSE =   Sum of squares of the difference

                                                                           between the average hole score and the

                                                                           estimate of the average hole scores

The USGA claims the “root-mean-square error” in rating a golf course is only .285 strokes ((s·(18).5).  The USGA does not report either the value of the model coefficients or whether they are statistically significant. The USGA may not have even tested for statistical significance, since multiple regression programs were not readily available at the time the model was estimated. There is no public record of the USGA revisiting the Course Rating Model now that more data and powerful analytic techniques are easily accessible. Interestingly, the yardage variable has not changed in the past 20 years though players are hitting the ball much further.  It is also doubtful that all ten obstacle coefficients are significant.  The inclusion of so many variables has given the illusion of precision that probably cannot be justified by the data.[6]

            The variance is also underestimated because of the colinearity among the yardage and obstacle values.  This colinearity stems from both design (architects tend to design more obstacles in long championship courses) and definition (the “green target” obstacle for example is related to the length of the hole.  This implies the variables are not independent which is one of the key assumptions of regression analysis.

            In summary, the course rating model may have specification, measurement, and colinearity errors.  The size of these errors is unknown.  To be conservative, these errors are neglected, and the USGA error estimate of ±.3 strokes will be used in estimating possible errors in the Slope Rating.

Error in the Bogey Rating - The Bogey Rating is the expected average score of a bogey golfer who is defined as a player with a USGA Handicap Index of between 17.5 and 22.4.  The USGA has taken a group of 60 golfers with indices within the required range to create a norm reference group.  How golfers with these indices were found before the slope methodology was developed had never been documented by the USGA.    In a correspondence, however, a USGA official stated that the standard deviation of the bogey rating estimate was ±.5 strokes.[7]  The estimate of the standard deviation is subject to all of the errors discussed previously for the Course Rating.  The USGA is given the benefit of the doubt, however, and this estimate of the standard deviation is accepted here for the purposes of this paper.


Error in the Slope Rating -  The Slope Rating is a measure of the course difficulty for a bogey golfer.  It is calculated by the equation:

                                                 Slope Rating =   5.381• (BR - CR)

Where,
                                                                BR =   Bogey Rating
                                                                CR =   Course Rating

The estimated equation for the Slope Rating can be found by substituting the equations for the course and bogey rating into the equation above:

                                                 Slope Rating =   52.7 +.00917•Yardage +

                                                                             5.381(CBOV - CSOV)

Where,
                                                          CBOV =   Bogey Obstacle Value for 18 holes
                                                          CSOV =   Scratch Obstacle Value for 18 holes

           The Slope Rating is not very sensitive to yardage.  A five hundred yard difference in course length would only mean a difference of 5 points in the slope rating.  What drives the difference in Slope Ratings among courses is the last term that measures the difference in obstacle values for the bogey and scratch golfer.

            An example of two courses will demonstrate the impact of the obstacle values on the variation in slope rating among courses.  The contribution to the Slope Rating of yardage and obstacle values for the Stadium Course at PGA West and for Rancho Park, a municipal course of moderate difficulty are shown in Table 1.


Table 1
Contribution to the Slope Rating
Course
Constant
Yardage
Obstacle Value
Total
PGA West
7265 Yards

52.7

66.6

31.7

151
Rancho Park
6271 Yards

52.7

57.5

6.8

117

       
When only the constant and yardage contributions are considered, the slope rating for PGA West is only 8 percent higher than for Rancho Park.  When all contributions are considered, however, the slope rating for PGA West is 29 percent higher.  The point is that the Slope Rating is heavily impacted by the most subjective, and hence the more prone to error, part of the rating system.

            The variance in the estimate of the Slope Rating is:

                                Variance(Slope Rating)=   (5.381)2 (BRV + CRV)

Where,
                                                 BRV =   Variance in the estimate of the Bogey Rating
                                                 CRV =   Variance in the estimate of the Course Rating

             Now the variance is just the square of the standard deviation of the two ratings.  Using the USGA values of .5 and .3 for the standard deviations, the variance of the slope rating becomes:

                               Variance(Slope) =   28.96 (.25 +.09) = 9.85

            The standard deviation of the estimate of the Slope Rating would be the square root of the variance or approximately 3.1 rating points.

Impact of the Slope System on the Accuracy of Handicaps - Does the Slope System enhance or degrade the accuracy of handicaps?  The answer is “It depends.”  Let’s examine three cases to judge the efficacy of the Slope System.

Case 1: No Slope Effect - The existence of the slope effect (i.e., bogey golfers doing relatively worse on a course with a high slope rating) has never been empirically demonstrated.  In this case where there is no slope effect, possible errors associated with the estimating methodology are not a key concern.  It is not the errors in estimating the slope, but the entire model that needs to be re-examined.  This is not the purpose here, however, so we go on to other cases where the validity of the Slope System is not challenged.

Case 2: Small Range in Slope Ratings - In this case, it is assumed the slope effect does exist but the range of true slope ratings is quite small.  For our purposes, it is assumed that courses have a range of ten Slope Rating points.  Such a small range is not as unlikely as it may appear.  In a survey in Southern California, 48 percent of all courses with a Slope Rating over 100 (i.e., a rough measure of being a regulation length golf course) had Slope Ratings from the regular tees between 114 and 123.

            When courses have about the same true slope, measurement errors become more important.  Assume for example, that two courses have the same true slope.  The probability of estimating the true Slope Rating at any one course is only around .13--i.e., the USGA estimated Slope Rating will only equal the true Slope Rating in about 1 in 7 tries.[8] 

More important, however, is the difference in the estimate of the two slopes since this will partially determine the size of any portability error.  The standard error of the estimate of the difference in two slope ratings is approximately 4.4 rating points.  The probability that the difference in two slopes will be estimated correctly is about .09.  The probability of making errors of various size in the estimate of the difference in two slope ratings is shown in   Table 2.

Table 2
Probability of an Error in Estimating
the Difference in Two Slope Ratings
N= Difference in Slope Ratings
Probability of an Error of More than N Rating Points
0
.91
1
.73
2
.57
3
.42
4
.31
5
.21
6
.14
7
.09
8
.05
9
.03
10
.02

The table indicates there is a one in five chance the difference in Slope Ratings will be off by 5 or more rating points. In this example, the Slope System can actually lessen the equity of competition when the true slope of courses are closely clustered.

Case 3: Large Range in Slope Ratings - The value of the Slope System is more evident where there is a large difference in the true slope among courses.  If the true difference were 10 rating points, for example, the more difficult course would almost always have a higher Slope Sating.  That is a player’s handicap (as long as it not below scratch) will stay the same or increase while going to the course that is tougher for the bogey golfer.  The increase may not reflect the true difference in difficulty because of measurement errors.  Measurement errors, however, may be small in comparison to the true difference in the Slope Rating.


Summary - The overall assessment of the efficacy of the Slope System rests in part on the range of slope ratings a player encounters.  Are the errors involved in rating similar courses outweighed by the more accurate handicap at courses with a substantially higher or low slope rating?  The answer to this question lies within data controlled by the USGA.  It is only with this data that the size of other possible error can be estimated.   Remember, this analysis used the best possible case for the Slope System.  And most importantly, USGA data could reveal whether there is indeed a slope effect large enough to justify the Slope System.

The USGA Responds

            The USGA was not pleased with this paper.  They responded with  little understanding of the statistics underlying their own model.[9]  The USGA denied there could be such large errors in the Slope Ratings:

In the USGA Rating System we admit that scratch rating are only accurate to within ±.3 stokes and bogey rating to within ±.5 strokes.  Assuming the widest spread… the resulting error in the slope rating is .8•(5.381) or 4.3.  So the certainty that 130 is more difficult than 125 is a little better than the certainty we assign to course ratings.

First, the USGA makes an error in assuming the widest spread can be .8.  The standard error does not mean that the estimate cannot be off by more that ±.3.  The standard error only means that the course rating methodology has at best only a 68 percent chance of bracketing the true mean to with ±.3 strokes.  This is an elementary mistake in statistics and reflects the level of sophistication behind the USGA ratings model.

            Second, he USGA response only deals with the error at one course.  What is important to the success of the Slope System is the error in the difference in Slope Ratings between courses.  If you only play one course, any error in the slope rating will have no effect on your handicap.  The errors between courses, however, can be substantial and distort the computation of a player’s handicap and lead to inequitable competition in inter-club matches.



[1] Letter from David Fay, Executive Director of the USGA to the author, May 31, 1991
[2] It is assumed the USGA used the better half of the 18 hole scores rather that the better half of the scores on each hole.  Using the latter scores would lead to an underestimate of the course rating.
[3] In order to estimate the magnitude of the difference in the two different sampling methods, 280 scores from the 1991 Atlanta Open were examined.  Using the better half of all scores, the average score was 68.3.  Taking the 10 best scores out of sets of 20 scores (players were grouped alphabetically into groups of 20), the average was 68.5 strokes.  The method of selecting the sample led to a difference in the course rating estimate of .2 strokes.  The difference is likely to be larger in the USGA estimate of the course rating since there is probably a greater variance in scores at the USGA Amateur than there is at most professional tournaments.
[4] One of the major assumptions of this type of regression analysis is not met.  Least squares regression techniques assume there are no errors in measuring the independent variables (i.e., the yardage and obstacle values.)  This is not the case with obstacle values, however.  For example, with what level of precision can a water hazard be rated a “2.”  Would other raters put the value at 3.  Is the hazard really a 2.45, but rated at 2 because the rating methodology does not allow for such fine distinctions.?  The model proposed by the USGA belongs to the class of problems termed “errors in both variables.”  The USGA has made no attempt to estimate the size of this error.  Estimating techniques for such models can be found in Wonnacott and Wonnacott, Econometrics, Wiley and Sons, New York, 1970, p. 164.
[5] Knuth, D., “A Two Parameter Golf Course Rating System,” Science and Golf: the Proceedings of the First World Scientific Congress, Rutledge, Chapman and Hall, London, 1990
[6] Another problem with the methodology employed by the USGA is the sample of holes selected.  The USGA had data for 126 holes, but chose to use data from only 74 holes.  In essence over 40 percent of the sample was not used.  The criteria for the selection of holes has not been given.  The omission of so much data, however, could lead to vary different estimates of the coefficients or much larger error estimates than the USGA has presented..
[7] Letter from Warren Simmons, Executive Director of the Colorado Golf Association to Dean Knuth, Director of Handicapping of the USGA. February 14, 1991
[8] To estimate true slope, the estimate must be off by less than .5 of a rating point.  The standard deviation of the estimate of the mean is 3.1 rating pints.  Therefore, the estimate must be within .16 stand error of the true mean.  Assuming a normal distribution of the error, such an estimate should occur only 13 percent of the time.
[9] Letter from Warren Simmons, op. cit.

Wednesday, September 26, 2012

The Francis Scheid Affair



            Nothing illustrates the bankrupt nature of the USGA handicap program better that what I term the “Francis Sheid Affair.”  It exposes the dearth of research behind handicap policy, and the unwillingness of the USGA to address criticism of the handicap section.

            The affair started innocently enough when I read a paper by Francis Sheid of the USGA, presented at the Second World Scientific Congress of Golf (1994).  The Congress guidelines required papers to be original.  The organizers of the Congress properly believed that if papers presented were merely rehashes of old work, the Congress would suffer in prestige. 

Since Sheid was a member of the USGA Handicap Research Team, I was expecting an article on the cutting edge of research on handicap issues.  Scheid’s article was entitled “The Search for The Perfect Handicap.”[1]  In this article, Sheid evaluated various handicap systems.  He concluded that “the current official handicap systems do about has well as any.”  It was an interesting piece, but clearly not of overwhelming significance.

            I was struck, however that I had seen the title somewhere before.  I went back to the proceedings of the First Scientific Congress of Golf(1990).  In a paper presented at this conference, Scheid had included the following reference:
 
"Scheid, F. J. (1978) The search for the perfect handicap.  Proceedings of the Winter Simulation Conference."

         The reference is not well documented.  There is, for example, no indication which organization published the paper.  This had the effect, intended or not, of deterring anyone from finding this paper.  However, with the help of some of the best research librarians in the United States the article was uncovered.[2]

            For all intents and purposes Scheid’s 1978 and 1994 articles were identical.  This meant 1) Sheid violated the conference guidelines calling for original research , 2) Sheid, a lead member of the Handicap Advisory Team, had not done anything in the period 1978-1994 he thought worthy of publishing.

            Scheid’s 1978 paper was sent to Martin Farrally, the Congress chairman, along with an analysis of the similarity between the two papers.  I thought he would be outraged that the USGA had pulled such a stunt on his conference.  Unfortunately, he was not.  Instead, Farrally replied:

“Since I have no competence in golf statistics I am not able to comment on whether Scheid’s paper was a regurgitation of 16 year old piece of work.  As the Congress director, I have to trust the judgement (sic) of the theme co-ordinators.”[3]


            Farrally did not say that he would investigate the claims.  Here merely stated that if it got by his theme coordinators, that was good enough for him.  He did not question his theme coordinator if he/she was aware the 1994 paper was a re-write of an earlier work.  Since some of the coordinators were paid consultants to the USGA, it is not likely they judged USGA generated papers too harshly.  Moreover, the entire Congress depended upon support from the USGA.  Apparently Farrally, felt that if the USGA wanted to slap the Congress in the face that was all right, because they had paid for that privilege.


            Even though Farrally refused an independent investigation of the affair, he did send my complaint on to Scheid.  Sheid wrote to Farrally that he was innocent.[4]  Farrally did not send a copy of this letter to me but did send a copy to David Fay, Executive Director of the USGA.  Fay mistakenly believed that Sheid presented a credible explanation and forwarded Scheid’s letter to me. 

Sheid’s defense was incredulous, and Fay should have spotted it.  In Fay’s defense, he knows little about statistics.  To his discredit, he never made any effort to settle the dispute, and became the leader in the effort to protect Scheid from the punishment he deserved.

            The weakness of Scheid’s defense can easily be seen.  His defense is shown below in italics.  A rebuttal of each of his points follows.

The original research (i.e., the 1978 paper) was based on the limited data available to me at the time.  I completely repeated all experiments using an entirely new and much larger data base.

 A comparison of the size of the samples used in each paper does not indicate a larger data base in Scheid-1994:

· Scheid-1978 – “At each of the courses for which data was in hand, fifty pairs of golfers (at eighteen courses) were chosen…At least…80,000 matches were played at each club to produce these results.

· Scheid-1994 – “at each of several clubs fifty pairs of golfer were select…About 80,000 matches were simulated a each club.

Since in common usage “several” is less than eighteen, it appears the sample size in 1994 was actually smaller than in 1978. Unfortunately, Scheid did not even meet the minimal scientific standard of documenting his sample size in his 1994 paper, so his claim could be conclusively disproved.

Scheid, however, proves his own guilt in another paper written for the 1994 Congress.[5] Sheid references two papers entitled The Search for the Perfect Handicap (1979), and the Search for the Perfect Handicap Part II, 1980. If he really did the research in the 90’s as he claims, why wouldn’t he reference the later work? The obvious answer is that there was no later work.

The results were reassuringly close to the original ones though not identical. Scheid tries to make the argument that since the results were different, there must be two different research studies.  His argument does not stand up to even a cursory inspection.

            Tables 1 and 2 present a comparison of the error tables contained in Scheid’s two papers.  Tables1 and 2 only report handicap types that are presented in both Scheid-1978 and Scheid-1994.  Not one handicap type reported in both of Scheid’s papers has a different error.  Where a comparison can be made, all the results are identical!

            The probability that the median error for fourteen different handicap types would remain exactly the same in two different samples is infinitesimal (the probability of this result is approximately .0002).

 Table 1
Median Errors, Match Play

 
 
Error
Handicap Type
Scheid-1978
Handicap Type
Sheid-1994
.4
Mean, (2-19), (3-18), (4-17), (8,13)
Mean, (2-19), (3-18), (4-17), (8,13)
.5
(1-15), MEDIAN,(6-15), (B15), (8)
(1-15), MEDIAN, (6,15), B15,(8)
.7
B12
B12
.8
USGA
USGA
1.0
(2), (1-5)
(2), (1-5)
>1.0
PT12
PT12

 

Table 2
Median Errors to Equalize Percentile 1 Scores

 
Error
Handicap Type
Sheid-1978
Handicap Type
Scheid-1994
1.2
(1), (1,5), Norm*
(1), (1-5),NORM
1.4
(2), (2-19), USGA, (1-15)
(2), (2-19), USGA, (1-15)
>1.4
PT12, MEDIAN,
PT12, MEDIAN

·         NORM is reported as 1.3 in Scheid-1978.  The error category “1.3” is not reported in Scheid-1994.  It is assumed that the error in NORM was rounded down to at “1.2” error.

 

After all, reproducibility is important in science and I was delighted with the confirmation that the larger experiment brought.
 
To get exactly the same results 16 years apart is indeed a spectacular finding.  But if it was so important, why did not Scheid report it in 1994?  The only reasonable conclusion for omitting such a spectacular finding is that Scheid did not want to draw attention to his previous paper.

 The earlier paper…was offered about twenty years ago.  I was anxious that the scientific golf community hear of its major result.  The Congress seemed the ideal place to do this.

 If Scheid was so anxious, it has to be asked why he did not submit his paper to the 1990 Congress rather than wait until 1994?  It is even more curious that his 1990 Congress submission (“On the Normality and Independence of Golf Scores”) contained a reference to Scheid-1978, but Scheid-1994 did not.  In 1990 paper he did not allude to his “major “ finding.  He simple listed the paper in the reference section even though it had nothing to do with the body of work.  Simply put, the arguments Scheid makes in his defense are not consistent with the facts in the case.  

The paper also included a new result, that even a two-dimensional system based on the normal model would not bring a significant improvement in accuracy… 

This is not a new result but appears in Scheid-1978.  In Scheid-1978, he writes “The normal model also does as well as any…(But) official USGA and British handicaps take a close second place.”  Scheid-1994 draws the same conclusion

Scheid does add an analysis of team play in Scheid-1994.  This is a minor addition, however, and has no effect on the overall conclusion of the research.


            The USGA had steadfastly refused to admit that Scheid violated the rules of the Congress.  David Fay Executive Director of the USGA, wrote: “we will not censure Francis Scheid or offer an apology.”[6]  Never, however, has the USGA offered an affirmative defense of Scheid’s conduct.  This has been consistent with the USGA policy of dealing with criticism – attack the messenger and ignore the message.

            The concern here is not that Scheid submitted old research, but that his sixteen-year old work was the best the USGA had to offer.  The USGA should be the recognized authority on handicapping as it is on equipment testing.  The USGA has not shown the breadth and depth of research on handicapping issues to deserve such status.

            During the past ten years, major changes in the handicap system have been implemented – the Slope System, a revised and re-revised equitable stroke control, and the reduction in index for exceptional tournament performance.  Yet the USGA has not published any empirical research demonstrating these revisions have had a positive effect on the equity of competition. 

            The poor performance of the USGA is due in part to its monopoly status.  Members of the handicap research team appear to be selected because they had some commercial tie to the USGA or its executives.  There is no record of the USGA seeking to hire the best talent in the country to study handicapping problems.  Instead the USGA has relied on the “old crony” system and the results show it.  As documented throughout blog, the USGA has made numerous mistakes and questionable judgment calls on handicap policy.  Unfortunately, there are no forces on the horizon that could change the USGA.  The USGA Executive Committee will continue to rule by its inherent motto, “We’re God, and you’re not.”

 



[1] Sheid, F.J., “The search for the perfect handicap,” Science and Golf II: Proceedings of the World Scientific Congress of Golf,  Edited by A.J. Cochran and M.R. Farrally, E & FN Spn, London, 1994.
[2] Scheid, Francis, the Search for the perfect handicap, 1978 Winter Simulation conference, IEEE, 1978
 [3] Letter from Martin Farrally, Director of the World Scientific Congress of Golf to the author, December 13, 1994
[4] Letter from Francis Scheid to Martin Farrally, April 13, 1995.
[5] D.L. Knuth, F.J. Scheid, and F.P. Engel, “:Outlier identification procedure for reduction of handicaps,” Scheid can Golf II: Proceedings of the 1994 World Scientific Congress of Golf, E & FN Spon, London, 1994
[6] Letter from David Fay, Executive Director of the USGA to the author, December 7, 1998.