Wednesday, April 11, 2012

USGA Maintains Infallibility

The USGA continues to defend the indefensible.  Appendix E is clearly in error, but the USGA maintains, against all evidence, that it is not.   Below is a discussion of the errors in Appendix E with Scott Hovde, USGA Manager of Course Rating and Handicap Education.

Dougharty (2/1/2012): I see the typos in the web version of Appendix E were corrected as I suggested. You did not, however, change the row headings which I also believe are in error. Let me argue my case again. The probabilities in Appendix E represent the chances of a net differential less than some negative integer. When you make the row heading a range you introduce large errors. For example, the probability for a 5.0 index having a net differential of -8.0 or better is 1 in 20111. The current Appendix E assumes the probability of having a net differential of -8.9 or better is also 1 in 20111. It is actually closer to 1 in 45,000--i.e., a player is 2.2 times more likely to have a net differential of -8.0 or better than he is to have a net differential of -8.9 or better.

Here is what I suggest:

1. Change the row headings back to negative integers.

2. Change the example so it produces a negative integer as was done in the original Appendix E. Let the interested reader worry about interpolating between integer values.

I did receive my new handicap manual today. I was amazed no one caught the typos in Appendix E. I assume a draft was run by various committees, but the members did not take the time to read it. If you ever establish a Copy Editing Committee (CEC), please consider my application.
I am still examining the question of whether the probabilities in Appendix E are reliable estimates. I have one lead on their origin. Knowing that I have until 2016 to complete my task removes a lot of pressure.
Regards,
Larry

Hovde (2/9/2012):  When reviewing the entire Handicap manual for the 2012 update, we looked at the table in Appendix E to determine if it needed some tweaking or updating. The initial thought was to remove the table from the book altogether and only supply an online version once the table was fully reviewed. In consulting with a few of our HRT members (who were involved with the original table), it was decided to keep Appendix E in the book, but expand the rows to ranges, as the formula (and example) listed underneath the table to determine the net differential is taken to the tenth place. The values in each cell represented an average of the probability of net differentials within that range, and not just a single net differential. In reality each 0.1 difference in net differential will have a separate probability, but that table would be far too large to be of use.

In the interest of ease of use by our target audience, the ranges were there to keep them from having to interpolate and just plug in the net differential, and probability was listed in each cell as the divisor, instead of probability itself (so 1200 instead of .0008333…). We also felt probability (in the form of listing the divisor) was more relevant to the typical reader, as 4:1 odds is easy to mistakenly interpret as 1 in 4, as opposed to 1 in 5.

The typos in the manual were unfortunately made at the printer when they formatted the table for final printing, the table we submitted and that was reviewed by all involved had the correct values. Your offer to be part of a Copy Editing Review Committee in the future is noted. While 2016 may be the next version of the Handicap System manual, we do intend on continuing to research this topic (and others) and provide any updated information via the Handicapping section of the USGA website.
Thank you for your clear attention to detail and communicating your concerns with the intent improve the Handicap System. Feel free to pass along any further feedback relating to this topic.
Regards,
Scott

Dougharty (2/13/2012):  Your defense of Appendix E was both disappointing and unpersuasive.  The disappointment stems from seeing a great institution, one responsible for protecting the integrity of the game, blame a printer for an error clearly its own. Someone at the USGA should have signed off on the proof copy of the manual. If no one did, the error is still the organization’s fault and not the printer’s--see Rule 6-6.

You argue the values in each cell present an average probability of net differentials in that range. Your argument suffers from a number of weaknesses:

1. Originally, the cells were represented as the probability of shooting a net differential equal or greater than the number in the left column—let’s call this the integer probability. Now long-time members of the HRT have allegedly come forward and stated the cells represent the average probability of shooting a net differential in “some range” or better. The integer probability and the average probability have different values. Yet the values in the cells (with one exception) have remained the same in the revised Appendix. The HRT members were either wrong in the past or wrong now. (Note: You will now have to inform golf associations the probability of beating your index is no longer 1 in 5. It’s now closer to 1 in 4.)

2. I believe “ranges” were entered because someone mistakenly thought it gave the illusion of precision. Ranges actually make a mess of things. The revised Appendix states the probability of a -7.1 net differential is 1 in 1200. This is not correct. What you now have is the probability of shooting a net differential in the range of -7.0 to -7.9 or better is 1 in 1200. If this sounds clumsy, it is. Making reasonable assumptions, the probability of a -7.1 net differential or better is around 1 in 780. The revised Appendix is in error by approximately 35 percent.

3. If there was a conscious effort to have ranges, I would expect the text of the revised Appendix to reflect that change. It does not. The Appendix reads “The values in the table represent the probability of shooting a net differential EQUAL TO OR BETTER THAN the number (emphasis added) in the left column.” This sentence was not altered from the old Appendix. If there was a careful rewrite of the Appendix as you maintain, the description of the cell values would also have been changed.

4. In the development of the Handicap Reduction Table of section 10, members of the HRT (Knuth, Scheid, and Engel) used integer probabilities in the calculations. It seems unlikely they would go to the effort to compute average probabilities for which there is no obvious use.

To settle this matter, I am forwarding our correspondence to Dean Knuth, former Senior Director of Handicapping at the USGA. If Mr. Knuth believes the revised Appendix E is an accurate reflection of the work of the HRT, then I stand corrected and apologize for my nuisance.

Regards,

Larry

Note: As of this writing (4/11/2012), neither Mr. Knuth nor the USGA have responded. 

























Tuesday, March 6, 2012

The Reliability and Accuracy of USGA Handicap Research II

(Note: This post revises a previous post based on comments by the USGA and a paper written by the USGA's Handicap Research Team.)

The USGA Handicap Section is responsible for how a golfer’s performance is measured and recommends how handicaps should be adjusted under different tournament formats.   Golfers for the most part have put blind faith in the USGA and the legitimacy of its recommendations.   It appears that such faith is misplaced.  The research behind the recommendations has never been vetted by outside academics,[1] is dated,[2]and in some cases just plain peculiar.[3]
The problems in USGA handicap research are illustrated by two tables prepared by the USGA on the probability of a player making an exceptional score.  The table presented in Appendix E of the USGA Handicap System is entitled “Odds of shooting an Exceptional Tournament Score.”  Part of that table is shown as Table 1 below.  
Table 1
Appendix E - Exceptional Tournament Score Probability Table[4]
Net Differential
0-5
6-12
13-21
22-30
>30
0
5
5
6
5
5
-1
10
10
10
8
7
-2
23
22
21
13
10
-3
57
51
43
23
15
 The values in the table are the odds of shooting a net differential EQUAL OR BETTER than the number in the left  column.

The definition of net differential appears to have changed over time leading to significant errors. The original research defines net differential as the difference between a player’s net score and the course rating.[5] The current definition of net differential in Appendix E is the handicap differential for a score minus the player’s Handicap Index.  The two definitions are different as can be seen by an example.   Assume a 17.0 index player scores a net 64 on a course with a slope rating of 150 and a course rating of 72.0.  The player is a 23 handicap and records a gross score of 87.  His net differential under the old system is -8.0.     His net differential under the current definition is -5.7 -- (87-72) 113/150 – 17.0.   Using the original definition of net differential, a player would have a 1 in 1138 chance of such a score or better.  Under the current definition of net differential, a player would have about a 1 in 300 chance of such a score or better. I suspect the old definition is the correct one, and the current Appendix E underestimates the rarity of low scores for courses with a Slope Rating greater than 113.  

The USGA shows some confusion about the distinction between odds and probabilities.[6]  The title of appendix references “probabilities,” but the body of the appendix is supposed to represent “odds.”   The USGA apparently does not know there is a difference between these two concepts.  “Odds against” is the ratio of the number of times an event does not occur to the number of times it does occur.  Probability is the ratio of the number of times an event occurs to the total number of trials.[7]  If the odds are 6:1, a golfer will have a net differential of zero or better 1 in 7 (not 6 as shown in Appendix E) rounds.  I assume, however, that the USGA meant probability and will accept the numbers in the table as that. 
Looking at Table 1, an inconsistency should be obvious.   If a player has an index between 13 and 21, he is 25 percent less likely to score a net differential of zero or better than a player with either a lower or higher index.  This is the only case where the probability of any level of performance (e.g., -2,-3…-10 net differential) does not increase or stay the same as a player’s index is increased.  Why is this index range cursed with such poor performance? I suspect any difference in probability among groups for scoring a net differential of zero or better may not be statistically significant.[8]  
Exactly how these probabilities were derived is not known.  The USGA does not have access to raw scores, only adjusted scores through its GHIN system.  Since adjusted scores are lower or equal to actual score, their use would overestimate the likelihood of an exceptional score.  If the USGA used actual scores, where did it get them, and how large was the sample?  Without a description of the research methodology, little credence should be given to the table.  
The table “Probability of Two Best Scores Beating Handicap” which appears on the USGA’s website presents more problems.  The table is shown as Table 2 below: 

Is the Table Accurate? – Table 2 represents the probability of scoring exactly 2 net differentials of various values.  This methodology underestimates the probability of exceptional scores.  The errors, however, are small in the range of interest of the USGA.    It also appears that Table 2 is based on Table 1.  Errors in Table 1 translate into errors in Table 2.  While Table 2 may be accurate, it is misleading.  For example, the table shows the probability of two net differentials of -6 is 1 in 3385.  The probability of two net differentials of -6 or better is 1 in 573—i.e., Table 2 decreases the probability by restricting the scores to a narrow range.  A more complete discussion of this issue is contained in the appendix.
Probability of Equivalent Pairs Are Not Equal – The probability of two best scores being (a,b) should be the same as the probability of (b,a).  This is the case for most of the pairs in Table 2.  There is an exception, however.  The probability of (6,9) is 17109.  The probability of (9,6) is 17189.  This is clearly just a typographical error made by the USGA, but such errors seem to plague the USGA handicap publications.[9] 

Table 2
Probability of Two Best Scores Beating Handicap 

0
-1
-2
-3
-4
-5
-6
-7
-8
-9
-10
0
27
46
92
199
408
869
1808
2480
3871
9180
85779
-1
46
13
26
58
118
253
526
722
1126
2672
24987
-2
92
26
20
43
89
191
398
546
853
2023
18907
-3
199
58
43
59
121
258
537
737
1150
2728
25492
-4
408
118
89
121
200
427
888
1219
1903
4512
42163
-5
869
253
191
258
427
821
1708
2343
3667
8872
81030
-6
1808
526
398
537
888
1708
3385
4644
7249
17189
****
-7
2480
722
546
737
1219
2343
4644
6225
9716
23041
****
-8
3871
1126
853
1150
1903
3667
7249
9716
1492
35361
****
-9
9180
2672
2023
2728
4512
8672
17109
23041
35361
82951
****
-10
85779
24967
18907
25492
42163
81030
****
****
****
****
****

 Many of the questions surrounding USGA handicap research could be resolved if it were peer reviewed outside of the USGA.  The only plausible reason for the USGA’s reluctance to release research, however, is it knows it cannot withstand scrutiny. Once the curtain is pulled back, the USGA’s perceived omnipotence is lost.  This is would not be a bad thing for the game.  


Appendix: Probability of Having Two Best Net Differentials

The probability of a player having his two best two differentials equal –n in 20 rounds is found by the following equation:  
                                                          P(-n) =  (sum from k=2 to 20)(20!/((20-k)!k!)) xk a20-k
Where,

P(-n) = The probability the best two differentials out of 20 equal -n
x = Probability of a net differential of -n or higher (i.e., a worse score)
a = Probability to a net differential of –n
k = Number of –n differentials
The probability P(-n) covers the occurrences of 2,3,….20 differentials of –n in 20 rounds –i.e., even if a player has four rounds of with a net differential of –n, his two best differentials are still –n.

To find the probability of any particular net differential, we use the probabilities of scoring a net differential of various values or better.  These are taken from Appendix E of the USGA Handicap System and are shown in Table 1A below.[1] 

Table 1A
Probability of Net Differentials or Better (13- 21.9 Index Player) 
Net
Differential

0

-1

-2

-3

-4

-5

-6

-7

-8

-9

-10
Probability
1/5
1/10
1/21
1/43
1/87
1/174
1/323
1/552
1/1138
1/3577
1/37000

 The probability of a net differential between 0 and -1 is the difference between the probability of 0 or better and 1 or better (i.e., 1/5 – 1/10 = 1/10).  We call this P(0).  The probabilities of other net differentials are determined in the same way and shown in Table 2A.
Table 2A
Probability of Net Differentials
Net
Differential

0

-1

-2

-3

-4

-5

-6

-7

-8

-9

-10
Probability
.1000
.05238
.02436
.01176
.00575
.00265
.00128
.00093
.00060
.00025
na

 As an example, we estimate the probability of the two best differentials being equal to 0.  As shown in Table 1, the probability of a net differential higher than 0 is .8.  The probability of a differential of 0 is .1 (see Table 2A).  We can now use the equation for P(-n) to calculate the probability of two best scores being equal to 0 as shown in Table 3A.
Table 3A
Probability of k Scores with a Differential of 0 
k
Combinations
P(0)n·P(>0)20-n
Probability
2
190
.00018
.03427
3
1140
2.2518E-5
.02567
4
4845
2.81475E-6
.01363
5
15504
3.51844E-7
.00545
6
38760
1.15292E-8
.00044
7
77520
5.49756E-9
.00042


Total
.07997


The probability of having two best net differentials of 0 is 1 in 12. This differs from the USGA‘s estimate (1 in 27) since that estimate is based on only the occurrence of exactly 2 net differentials of 0. [2] 
The size of any error in the USGA calculations can be seen in Table 4A which presents the Equation estimate, the Exactly 2 estimate, and the USGA estimates. 
Table 4A
Probability of Two Best Scores 
Two Best Scores
Equation
Exactly 2
USGA
0,0
12
29
27
-1,-1
9
13
13
-2,-2
18
21
20
-3,-3
54
58
59
-4,-4
189
186
200
-5,-5
818
831
821
-6.-6
3349
3375
3385
-7,-7
6210
6245
6225
-8,-8
14855
14908
14912
-9,-9
82671
82545
82951

 All of the estimates are virtually the same for pairs -2 and better. Since this is the region of most importance in spotting exceptional scores, the USGA formulation is adequate.  It may, however, overstate how rare exceptional scores are.
The USGA presents the probability of the two best scores out of 20 being equal to -6 is 1 in 3385. But, using the same methodology, the probability of a player having two best scores of -6 or better is 1 in 572 or about six times more likely than having scores exactly equal to -6.[3]  
The rarity the USGA finds is due in part to the interval it has chosen.  Assume for example, the USGA had decimal intervals.  The probability of a player having a net differential of -3 is the probability of a net differential between -3 and -3.1.  The probability of two best net differentials of -3 (making simplifying assumptions) with this shortened interval is about 1 in 6800.   Basing any modification of a player’s handicap on this probability would clearly not be fair. 
The better method is to find the probability of two scores of –n or better.  If that probability is found unreasonable, a player should have his index reduced.  In examining the Handicap Reduction Table,[4] it appears the probability of –n or better has been used.  For example, for a player with 20 tournaments and an average net differential of -6, there is no handicap reduction.  That is, the probability of scoring two net differentials of -6 or better is 1 in 570 and within the range of reasonability (where 1 in 3385 would not be). [5]





[1] The USGA Handicap System, 2012-2015, USGA, Far Hills, NJ, 2012, p.117.
[2] Knuth, loc. cit.
[3] The confusion about what these numbers mean is shown in a USGA Publication (How Well Should You Play?):

“For example, the odds of our example player with a course handicap of 14 beating it by eight strokes (-8 net) once is 1,138 to one.  Put another way, the average player posts 21 scores a year.  That means to score this well, assuming the Handicap Index is correct, would take 54 years of golf to do once.”

This is incorrect.  The player is expected to accomplish the feat once in 54 years, but that is different than saying the feat will be accomplished at the end of 54 years.  The player may well do it tomorrow.  There is a 17 percent chance he will accomplish this feat in the first ten years.  There is a 37 percent chance he will not accomplish the feat even in 54 years.

[4] USGA Handicap System, p. 81.
[5] Appendix E rates performance by negative net differentials (score differential - handicap index).  The Handicap Reduction Table uses positive net differentials (handicap – average of two differentials) in measuring performance. For consistency the USGA should use one type of differential.  It is also not clear why the Handicap Reduction Table goes up to an average net differential of -14 when such scores are highly unlikely according to Appendix E. 








[1] It is USGA policy not to release its research to any outside agency.
[2]  The USGA recommends a 90 percent allowance for four-ball stroke play. This allowance appears to come from a small study conducted by Francis Scheid, a one time member of the USGA Handicap Research Team, in 1971 (See F. Scheid, “You’re Not Getting Enough Strokes,” Golf Digest, Trumbull, Connecticut, June 1971, p. 52). In this study, Scheid took 50 scorecards from members of his home club and simulated matches among them. Using such a small sample drawn from non-tournament conditions makes any result suspect.  
[3] In four-ball stroke play, for example, men play to 90 percent of their handicap, and women play to 95 percent.  The USGA has never presented any theory on why the absence of a Y-Chromosome should alter a player’s handicap allowance.
[4] Note: Appendix E has been abbreviated here.  It actually shows probabilities up to a -10 net differential.
[5] Knuth, D. L., et. al., Outlier Identification Procedure for Reduction of Handicaps, Science and Golf II, E & F Spon, Chapman and Hall, London 1994, p.230.  This definition is affirmed in an article written by John Paul Newport in the Wall Street Journal (July 2, 2011).  Dean Knuth, former Senior Director of Handicapping at the USGA, is quoted saying the odds of a midrange player shooting eight strokes better than his course handicap are 1 in 1,138. 
[6] Since the first draft of this paper, the USGA has dropped “odds” and now uses “probabilities” in Appendix E.
[7] Goldberg, Samuel, Probability: An Introduction, Prentice Hall, Englewood Cliffs, N.J, 1960, p. 70.  Let E be any event and P(E) be the probability of that event.  We say that the odds for E are a to b  if and only if:
                P(E)=a/(a+b).  If odds for E are a to b, then the odds against E are b to a.

[8] Since the first draft of this paper, the probability for this 13 to 21 index range has been dropped from 1 in 6 to 1 in 5.  No explanation was given for the change. 
[9] This mistake is not an isolated example. For at least 5 years, Appendix E has shown the following incorrect equation:
                                                                    74 -71.2 = 2.8 x 113/126 = 2.5 Handicap Differential
The equation should be:
                                                    (74-71.2) X 113/Slope Rating = 2.8 x 113/126 = 2.5 Handicap Differential
 (Note: This error was corrected for the 2012-2015 edition of the Handicap System.  Unfortunately, current edition contains new errors).