Monday, June 25, 2012

The USGA Responds II


A senior official at the USGA made the following response to the criticism detailed in the previous post (Far Hills, We Have a Problem):

"Thank you for the thoughtful attention and work. After meeting with the Handicap Committee, Course Rating Committee and the Handicap Procedure Committee, we are now scheduled to meet with the HRT next week with this being one of the discussion topics. We clearly understand the importance of our roles in this area relating to the USGA Handicap System and are indeed taking a more aggressive position with our R&D this year. As communicated previously, I will likely be reaching out to you for help where appropriate.
Again, thank you for taking the time to clearly outline your observations and recommendations. They will be given careful thought and consideration by all involved. I will reach back out to you once we have some actionable items. Until then, please feel free to contact me with any questions or concerns." 

I believe this is USGA speak for "You will never see or hear from us again."

.

Monday, June 4, 2012

Far Hills, We Have a Problem


IntroductionThis paper explores problems with the United States Golf Association (USGA) Handicap System.  These problems stem from a failure to understand the theory behind the Slope Handicap System, and the dearth of USGA-sponsored research on handicap issues for the past 20 years.  This in turn has led the USGA to make changes in the handicap system that reduce the equity of competition. 

The device chosen to illustrate these problems is a case study involving Section 10-3, Reduction of Handicap Index Based on Exceptional Tournament Scores, of the Handicap System. The following example and question were posed to the USGA: 
Player A plays two tournaments at a course with a Slope Rating of 150. His two net scores are five strokes under the Course Rating.
Player B plays two tournaments at a course with a Slope Rating of 87. His two net scores are also five strokes under the Course Rating.
Assume these T-scores are the only ones in each player’s file. Under Section 10-3, Player A receives no reduction in index.  Player B’s index is reduced by 4.8. Why is the player on the "easier" course treated so harshly?

The problems are examined in three parts.  First, the USGA’s response is analyzed for its understanding of the Slope Handicap System. The USGA’s answer is found to be unresponsive at best and invalid at worst.  Second, a flaw in the Slope Handicap System is demonstrated.  This flaw is partially responsible for the counter-intuitive result of the above example, and its existence has been steadfastly denied by various officials at the USGA and state golf associations. Third, the mistakes made in the development of the Handicap Reduction Table of Section 10.3 are documented.  The USGA changed the vocabulary used in the table to be consistent with the Slope Handicap System (e.g., “strokes under the course rating” became “index minus average tournament differential.”  The USGA didn’t know or didn’t care that these two concepts of exceptional performance are not synonymous.  This mistake has led to inequity and the counter-intuitive result of the example.  In a concluding section, the state of research at the USGA is assessed and recommendations for improvement made.   

The USGA Response – Scott Hovde, Manager of Course Rating and Handicap Education at the USGA, made the following response in an attempt to answer the questions posed in the example:[1]

As Slope Rating is a relative measurement of difficulty (not absolute), essentially an indication of how scores are spread out…a performance of 5 under the Course Rating on a Slope Rating of 87 is more significant.  On a course with an 87 Slope Rating, scores are expected to spread out at a rate of .77 (87/113…rounded) per difference of 1.0 in the Handicap Index.  To beat a scratch golfer (who is a very good player) by 5 strokes on a course where scores are very tight for players of all handicap levels, is more difficult than doing it on a course with a 150 Slope Rating, which spreads out scores by 1.32 per 1.0 in the Index.

The advantage of a player with a lower Index grows as Slope Rating goes up and shrinks as Slope Rating goes down, regardless of which side of scratch a player is on.

Hovde argues scores are very tight on courses with low slope ratings and therefore performances of 5-under the Course Rating are more difficult to achieve.   There is no empirical or theoretical basis for such a claim. Papers proposing the Slope System only argue a player’s average score increases with the Slope rating.[2]  These papers do not argue the Slope Rating affects the distribution of a player’s score.    

Hovde next argues the advantage of a player with a lower Index grows as Slope Rating goes up and shrinks as Slope Rating goes down.  This is true by the definition, was never in question, and is irrelevant to the question posed in the example.  The question is whether indices are assigned properly, and not whether the handicap calculation based on those indices is correct.  Why does the USGA consider the player who scores 5-under the Course Rating at the PGA West Stadium Course (Slope Rating = 150) less skilled than a player scoring 5-under the Course Rating at the Oasis (Slope Rating = 87)?  This question is examined next.

Flaw in the Slope Handicap System – To understand the flaw in the theory behind the Slope Handicap System, it is important to review its theoretical underpinnings.  The Slope Handicap system was introduced by the USGA in order to bring more equity to the game.  The previous handicap system had computed handicaps by measuring the difference between a player's score and the course rating.  It was believed, however, the course rating did not properly measure the difficulty of the course for the higher handicapper.

The theory behind the Slope System is illustrated in the figure below:[3]   

Figure - Player’s Score at Panther Mountain and Perfect Valley Versus Perfect Valley Handicap


The figure plots players' expected scores (defined here as 96 percent of the average of the ten best out of the last twenty rounds) on a tough course named Panther Mountain (slope rating = 155) against their handicaps on a course of average difficulty named Perfect Valley (slope rating = 113).  For example, a 10-handicap at Perfect Valley would be expected to have a score of 83.7 at Panther Mountain or 13.7 strokes over the course rating of 70.  Also shown in the figure is the score on Perfect Valley plotted against a player's Perfect Valley Handicap.  For convenience the course rating at each course is assumed to be the same (70), though that is not necessary to the argument.
Therefore, as illustrated in the figure, players from Perfect Valley with handicaps greater than scratch (i.e., 0 handicap), will score higher at Panther Mountain than they will at their home course.  The Slope Handicap System increases these players' handicaps at Panther Mountain reflecting the difficulty of the course.  While the USGA has never presented any empirical evidence of this “slope effect,” it does make intuitive sense.  Obstacles at Panther Mountain such as length, water hazards, and rough may cause problems for bogey golfers that are not incorporated in the Course Rating.

 Perfect Valley players with plus handicaps (i.e., less than scratch), however, are expected to score lower at Panther Mountain than at Perfect Valley.  Therefore, these players will have their handicaps reduced when they play at Panther Mountain.  The Slope Handicap System assumes a course with a high slope rating is relatively easy for players with indices less than scratch.  There is no empirical evidence indicating this assumption is valid.  Nor is there any theoretical reason to believe a player who scores    5-under the Course Rating at Perfect Valley is a much better player than one who scores 5-under at Panther Mountain.

The problem arises because the USGA chose to set the scratch golfer as the standard.  Had the standard been set at the “perfect golfer” (e.g., a +10) there would be no negative differentials and all players at Perfect Valley would have their handicap stay the same or increase when traveling to Panther Mountain.[4]   
Even if negative scoring differentials were eliminated, however, the peculiar result of the example would still exist.  It is a mistake made in the development of the Handicap Reduction Table that leads to the much harsher treatment of the player at the course with the low Slope Rating.  We turn to this next.

 Handicap Reduction Table – An abbreviated Handicap Reduction Table is shown below:

Handicap Reduction Table (Abbr.)

Avg. of 2 Lowest T-Score Differentials Below Index
Number of Eligible Tournament Scores
2
3
4
5-9
10-19
4.0-4.4
1.0




4.5-4.9
1.8
1.0



5.0-5.4
2.6
1.9
1.0


5.5-5.9
3.4
2.7
1.9
1.0

6.0-6.4
4.1
3.5
2.8
1.9
1.0
6.5-6.9
4.8
4.3
3.7
2.9
2.0

The table shows the higher the tournament differential (Index - Average of 2 lowest tournament differentials), the greater the possible reduction in index.  The problem is there is no research indicating the reduction should be a function of the tournament differential.  The basic research was done by F.P. Engel who used Bogevold’s data in 1983 to determine the likelihood of a player recording a low negative differential defined as net score minus the course rating.[5]  Dean Knuth, former Senior Director of Handicapping at the USGA, developed a probability table for a player’s two best scores to beat his handicap.[6]   Even though Knuth is credited with inventing the Slope Handicap System, he used strokes below course rating and not index differentials in his analysis.  In a recent interview with the Wall Street Journal,[7]  Knuth said “the odds of a mid-range player shooting eight strokes better than his handicap are 1 in 1,138.”[8]  In other words, the handicap research done under Knuth--which practically speaking is all of the research done by the USGA--viewed exceptional scores as strokes below the Course Rating.  Knuth’s work estimates the probability of being 5-under the course rating is independent of the Slope Rating.  This makes intuitive sense as well.  A player’s score and handicaps are not affected by the Slope Rating if a player only plays at home.   If scores are normally distributed with a constant standard of deviation, a player has an equal chance of scoring n-strokes under the course rating whether the Slope Rating is 87 of 150. 

So why did the USGA change “Strokes Under Course Rating”  to “Index - Average Tournament Differential ” as the measure of exceptional performance?   The two culprits were probably sloth and ignorance.  Sloth kept the USGA from doing any new research on the problem.  When the current Appendix E, Exceptional Tournament Score Probability Table, of the Handicap System is based on         30-year old data, it is a good indication no one in the Handicap Section of the USGA is working overtime.[9]

Ignorance probably guided someone at the USGA to make the change thinking it made little difference.  According to Appendix E, a net differential 8.0 has a 1 in 1138 chance of occurring.  A player would have a net differential of 8.0 if his net score was 10.6 strokes under the Course Rating at a course with a Slope Rating of 150 or if his net score was 6.2 strokes under the Course Rating at a course with a Slope Rating of 87. To believe those two events are equally likely, one has to suspend all judgment and disregard the bulk of USGA research.  Apparently, that was no problem for some USGA staffer and the Handicap Procedures Committee.

Conclusions and Recommendations – This paper has identified a flaw in the Slope Handicap System and problems with Sec. 10.3 of the Handicap System.  The flaw is not a major impediment to equity since:

1.       The flaw primarily affects players with plus indices who constitute a very small minority of all golfers,

2.       Plus indices are typically not concerned about their handicap in inter-club net scoring events,

3.       The flaw does not affect intra-club events where players’ handicaps are based primarily on scores from their home course.

Moreover, to correct the flaw would be very difficult, and only confuse the golfing community that is already sufficiently addled by the present Slope Handicap System.

 It is recommended, however, that the USGA 1) update the research on exceptional performance, and 2) re-examine Sec. 10-3 for efficacy and effectiveness. Both tasks need to be completed to ensure an equitable handicap system.


Updating Research on Exceptional Performance - Both Appendix E and the Handicap Reduction Table appear to be based on data and analysis that are 30 to 40-years old.  Due to limits in computing power back then, the sample size was probably small.  With the advent of GHIN and modern computers, research no longer has such restraints.  It would not be a difficult task, for example, to find the percentage of golfers who have a score of 7-under the course rating in their last 20 scores.  This empirical finding could be used to revise the probabilities presented in Appendix E.

Re-Examine sec. 10.3 - As discussed above, equivalent performances (i.e., n-strokes under the course rating) are treated differently depending on the Slope Rating.   It is doubtful the USGA can demonstrate Sec. 10-3 as presently constituted is equitable.  Armed with new results on exceptional performance, the USGA would be in position to construct an equitable Handicap Reduction Table.  This re-examination should also review if Sec. 10-3 is serving its intended purpose (e.g., Is the lack of uniformity in the application of Section 10-3 affecting the equity of competition?  Is the present method of measuring tournament performance equitable?)[10]

Research on golf performance has undergone resurgence recently.  Pope and Schweitzer have analyzed success rates when players are putting for par and birdie.[11]  Minton has explored the effect of randomness in golf and the mathematical strategies behind the sport.[12]  Wells and Skowronski examined the evidence on whether players choke under pressure.[13]  What all of these studies have in common is that they were performed with data made available by the PGA Tour.  There is no similar set of studies dealing with equity in amateur golf or using data from the USGA.

The dearth of research on handicapping is due to policies of the USGA.  The USGA has not directed much attention or resources to research in years.  The present policy of restricting public access to its research has also not served the USGA well.  Without external peer review, the probability of mistakes similar to those made in Sec. 10-3 and Appendix E will continue to be high.  The USGA needs something equivalent to perestroika to breakdown the insular barriers between it and the research community.  If history is any judge, such an “opening” is decades off.





[1] E-mail from Scott Hovde to the author, April 16, 2012.
[2] R.C. Stroud and L.J. Riccio, Mathematical underpinnings of the slope handicap, in Science and Golf: The Proceedings of the First World Scientific Congress of Golf, Rutledge, Chapman and Hall, London, 1990, pp. 135-140.  D. Knuth, A two parameter golf course rating system, Ibid, pp. 141-146.
[3] The figure is a slight modification from that found in R.C. Stroud and L.J. Riccio, loc. cit.  Stroud and Riccio wisely did not show or discuss the effect on plus handicaps at Perfect Valley.
[4]The USGA did not believe it had a problem even when the Slope Handicap was first adopted. Dean Knuth, Senior Director of Handicapping wrote “The hurdle of negative differentials could be bypassed completely by setting the course rating at a low enough level to ensure their non-existence…The effect would be exactly the same as what we have and want now.  What really is happening is that ability differences are being magnified or contracted, as appropriate.”  (Letter from Knuth to Russ Palmer, Executive Director of the Connecticut State Golf Association, Sept. 13, 1994.)  The effect, of course, would not be the same.
[5] D.L. Knuth, F.J. Scheid, and F.P. Engel, Outlier identification procedure for the reduction in handicap, Science and Golf II: Proceedings of the World Scientific Congress of Golf, E & F Spon, London, 1994, p. 230.
[6] Ibid, p. 231.
[7] John Paul Newport, Fighting Back Against Sandbaggers, Wall Street Journal, July 2, 2011. Also see John Paul Newport, The Genius of Handicapping, Wall Street Journal, November 1, 2008 where Newport cites the odds(sic) of beating your handicap by three strokes is 1 in 20. 
[8] Kelly Neely, a member of the USGA Handicap Procedures Committee and Senior Director of Handicapping of the Oregon Golf Association wrote “Your chances of beating your handicap by 10 strokes in one round is (one in) 37,000.” See www.oga.org/index.php/handicapping. Apparently, Ms. Neely relied on an old version of Appendix E.
[9] As an another example of sloth, Appendix E formerly showed the probability of a 13-21 handicap player having a net differential of 0 or better was 1 in 6.  For every other index range, the probability was 1 in 5.  When the USGA was alerted to discrepancy, it changed this probability to 1 in 5 for all players in the 2012 Handicap Manual.  No new research was done.  Someone just replaced a 6 with a 5—problem solved.  Equally disturbing, the 2012 Handicap System Manual contains typographical and editorial errors in Appendix E.  When these were brought to the attention of the USGA, Mr. Hovde blamed the errors on the printer. E-mail to author, February 9, 2012.
[10] Under the old Section 10-3(1994), the penalty for exceptional performance was a function of how well you played in tournaments.  Under the current version of Section 10-3, the penalty is a function of how well you played relative to your current index.  This leads to some peculiar and perhaps inequitable results.  See Dougharty, Laurence, Reduction in Index for Exceptional Tournament Performance: Some Conceptual Problems, Measuring Golf Performance, Golf Economics, Manhattan Beach, CA, 2005.
[11] Pope, Devin and Schweitzer, Maurice, Is Tiger Woods Loss Averse? Persistent Bias in the Face of Experience, Competition, and High Stakes, American Economic Review, 101 (February 2011), pp. 129-157.
[12] Minton, Roland, Golf by the Numbers, The Johns Hopkins University Press, Baltimore, MD, forthcoming June, 2012.
[13] Brett Wells & John Skowronski, Evidence of Choking Under Pressure on the PGA Tour, Basic and Applied Social Psychology, March/April 2012, Pages 175-182.










[