Wednesday, February 19, 2014

John Paul Newport: Not the Sharpest Wedge in the Bag II

John Paul Newport is the golf columnist for the Wall Street Journal.  His lack of analytics can often lead him astray as it did in his column of July 2-3, 2011 (“Fighting Back Against Sandbaggers”).  First, he took some tables off the Internet, and assumed they were valid—this is never a good idea or good journalism.  Second, some knave misled him on the interpretation of the tables and the vagaries of the Slope System.  This resulted in multiple errors. Third, Newport pronounced the Tournament Point System (TPS) the Salk vaccine for sandbagging on the basis of two phone calls.  I doubt the efficacy of the TPS, and strongly believe sandbaggers will be eating the few remaining cockroaches when our planet comes to its end.  More detailed comments are made below.   Excerpts from Newport’s column are in italics.   

Among the most fascinating pages of Knuth’s website…is one titled “Odds of Shooting an Exceptional Tournament Score.”    A player in the 13-21 handicap range, for example, will shoot better than his course handicap only one round in every six.

The table Newport cites on Knuth’s website is also Appendix E of the USGA Handicap System.  Part of that table is shown as Table 1 below.  When the probabilities shown in the Table were first developed there was no Slope System and the probabilities were related to handicaps.  When the Slope System was introduced handicap ranges were switched to index ranges.  Such a conversion is not theoretically correct, but intellectual consistency was never the USGA’s strong suit.   Newport should have used an index range of 13 to 21.9 to be in accordance with the USGA

Table 1
Appendix E - Exceptional Tournament Score Probability Table[1]

Net Differential
Handicap Index Ranges
0-5
6-12
13-21
22-30
>30
0
5
5
6
5
5
-1
10
10
10
8
7
-2
23
22
21
13
10
-3
57
51
43
23
15
The values in the table are the odds of shooting a net differential EQUAL OR BETTER than the number in the left column.

Looking at Table 1, an inconsistency should be obvious.   If a player has an index between 13 and 21, he is 20 percent less likely to score a net differential of zero or better than a player with either a lower or higher index.  This is the only case where the probability of any level of performance (e.g., -2,-3…-10 net differential) does not increase or stay the same as index is increased.  Why is this index range cursed with such poor performance?  I suspect it has to do with poor research technique.  Any difference in probability among groups for scoring a net differential of 0 or better may not be statistically significant.  This hypothesis cannot be confirmed, however, since the USGA does not release its studies for peer review.    When I mentioned this anomaly to the USGA, the probability for the 13.-21.9 index player was changed from 6:1 to 5:1 in the 2012-2015 edition of The USGA Handicap System.  The USGA never admitted a mistake nor cited new research for the change.

The odds of a midrange player shooting eight strokes better than his course handicap are 1 in 1,138.  The odds of him doing that twice in 20 rounds (much less than in a two- or four-round tournament are 1 in 14,912).

There are several errors in this quote.  Table 1 show the probability of shooting a net differential of  -8 or better, not just -8.  Newport also errs in using handicap instead of index.  An example will make this clear.  Assume a player is a 13.0 index.  He plays a course with a course rating of 72.0 and a Slope Rating of 150.  His course handicap is 17.  Assume he shoots eight strokes better than his handicap (i.e., his net score is 64 and his gross score is 81).  The player’s handicap and net differentials are:
Handicap differential = (81-72) x 113/150 = 6.8
Net differential = 6.8 -13.0 = - 6.2 rounded to -6.0
So in this case,  the probability of a player shooting 8 strokes or better than his handicap is 323:1 and not 1,381:1.  The probability of the player doing that twice in 20 rounds is 1 in 3,385, and not 1 in 14,912. (Note: These latter probabilities are taken from a Table shown in Knuth’s website that also has technical problems that are not discussed here.)   In his defense, Newport was just using “handicaps” as instructed by Knuth.  The fact that Newport did not notice the inconsistency between “handicap” and “index” demonstrates Newport’s shortcomings as journalist.

But clubs or associations that run numerous handicapped or net tournaments are using another more effective Knuth invention called the Tournament Point System.

Newport judged the Tournament Point System (TPS) effective based on little evidence.  It appears the TPS has been around for at least 7 years and possibly much longer.  If it was that effective, it should have swept the nation.  It appears Newport only found two clubs using the Point System, one of which may have been Knuth’s home club in California.  The other club that extolled the TPS was Lost Creek Country Club in Austin Texas.  A call to the Lost Creek pro shop in 2014, however, revealed the TPS it is no longer in use.   
A club in Oregon implemented a version of the TPS nine years ago.  It was not a fair test, but it did reveal critical flaws in the system:
Not Effective at Many Clubs – Many clubs do not have a large number of major tournaments necessary for effective implementation of the TPS.  Often, the Member-Guest is the only tournament with significant prize money.  Under the TPS, a player could win the Member-Guest every year and not be penalized.  If minor events are put under the TPS, the strategic player could chose to sit out these tournaments rather than risk a penalty for an up-coming major tournament.  The artful sandbagger could also go unpunished under the TPS.  If the TPS cuts two strokes, he merely submits scores that raise his handicap by two strokes.
Double Whammy – Assume a player with a 15 index had two net tournament differentials of -8.  Under section 10-3 of the USGA handicap system, a player’s index could be reduced to 8.2.  On a course with a 133 slope rating his handicap would come down from 18 to 10.  Under the TPS his handicap would be further reduced by another three strokes making him a 7 handicap.  This is draconian.  Now the tournament committee may think this unfair and not apply the TPS reduction.  But if the objective of the TPS was to remove discretion, it would have failed.
Too Complicated and Time Consuming – The TPS is not as simple as suggested in the Newport’s column.  When the Oregon club used the TPS for weekly and flighted events, just about everyone was getting a penalty.  The latest version of the TPS has now been changed to alter the point distribution and penalties as a function of flight size and number of tournaments played.  And who is going to keep track of all of this?  Pros certainly don’t want to do it.  Volunteers were relied upon at the Oregon club.  This led to haphazard record keeping.  Some tournaments were recorded, others were not.  Members showed their dissatisfaction by declining to participate in tournaments.  The TPS was quietly killed at the end of the season.

In summary, Newport’s main fault appears to be he believes anything that contains a modicum of mathematics or statistics.  This should be a disqualifying fault for being a Wall Street Journal columnist.  Unfortunately, it is not.





[1] Note: Appendix E has been abbreviated here.  It actually shows odds up to a -10 net differential

7 comments:

  1. You are probably aware that Australia has adopted the USGA Slope systerm. They are unique in that only competition rounds are counted, handicaps are updated after each submission (done via computer entry by the club on the day of the competition) and that certain “problems” were noted when the gradual switch was instituted.

    Most clubs have 2-3 of these “comps” per week.

    There was little doubt the old system favored low markers as a good score could result in an immediate drop of 2 or more strokes, but then the hcp could only go out by 0.1 for each bad round.

    The first integration (before slope) involved changing to best 10/20 and 0.96 excellence factor. What soon became evident was that “high markers” were winning at an unacceptable rate. The USGA was consulted and Golf Australia (GA) was told the system was not designed to handicap large field events. This of course begs the question as to how the “tournament” scores can be used effectively in policing USGA sandbaggers.

    GA, with the huge data base of scores their system produces, hired statisticians to examine and recommend changes. This resulted in a change to best 8/20 and 0.93 excellence factor. The furor subsided as results evened.

    Perhaps someone with your credentials would be able to gain access to their data. I'm sure you would find it very interesting.

    Other differences in the system:

    All results re converted to Stableford (which is by far the most common game), thus ESC is double bogey for all handicaps.

    A daily course rating formula has been instituted to account for vagaries of winter/summer, high wind, etc.



    ReplyDelete
    Replies
    1. I believe the point the USGA was trying to make was that no handicap system yields prefect equity when there is a large range in handicaps. That is why "flighting" a tournament is recommended. The USGA's Sec 10-3 which deals with tournament scores would still be in effect in flighted tournaments.
      I am not sure how a good score can drop player two strokes. Let's take the USGA system with ten differentials. Assume a player has ten low differentials of 10.0. This would make him a 9.6. To drop two strokes (i.e., become a 7.6), a player would have to post a differential of -10.8. This is unlikely.
      I am not a big fan of the Daily Scratch Rating(DSR). The DSR gives the illusion of accuracy without the reality. The Course Rating is a mythical number to begin with. To attempt fractional refinement in a base number that is not known with precision seem like a waste of computer time. On the plus side, I don't see where it does any harm.
      I was hoping GA would change the definition of the Slope Rating to the Old Slope Rating divided by 113. The Slope Rating would then be the percentage increase (or decrease) to be applied to your GA handicap to get your course handicap. For example, if the new Slope Rating was 120, you would get 120 percent of your GA handicap. This would have given the Slope Rating a more intuitive meaning. Alas, it was not to be.
      Thanks for your comment.

      Delete
    2. I meant the old "old" system prior to USGA integration. All it took was one good round! There was a formula for how much your hcp dropped based on how far below it you shot, then only 0.1 for each round above the hcp. A drop of 2 was routine for high handicaps having a good round.

      Lots of data there.

      Thanks

      Delete
  2. Thank you both for the info and dialogue. I run a golf league in Austin, Texas (www.AustinSkins.com) and we have made several changes to the USGA handicapping system over several years to come up with what we feel works extremely well. Interestingly, the ESC we use is exactly what GA has adopted: double bogey max for all golfers. We initially made this change because we were handicapping for a skins game, so we instinctively understood that blow up holes should not inflate the player's handicap and give him an unfair advantage in a skins game over players who do not have blow up holes and thus have lower handicaps.

    We also use best 5 scores of last 10 instead of the USGA's 10 out of 20. The reason we do this is two-fold: (1) we don't want people to feel like the same players win every week, so making each good score worth 20% of the entire handicap will lower the handicap faster for a good score and make it harder to win again the subsequent week. and (2) we have players who play only once per month and we felt that a handicap that included score that were almost 2 years old does not reflect the player's current ability. And actually, since then we have adopted a policy of only using scores that are 12 months old or newer for the same reason.

    Additionally, we only use 85% of the course handicap, which many higher handicappers really don't like at first, but the evidence shows that they can and do win. We started using 85% of the course handicap because we do not flight our tournaments and we discovered that the odds of one of the six to ten 10-14 handicappers in a given week shooting a net 67 are far lower than the odds of a specific 10-14 handicapper do it. So low handicappers were complaining that they could not win if one of the high handicappers was going to shoot 67 every week. We tried the 85% handicaps and its worked great. We pay our low net prize money like a Nassau: front/back/total, and its very common for the winners of the three to be from totally different handicap ranges.

    Our handicaps seem to be effective. We limit that max handicap to 18, and to be honest, players above about 15 have a hard time winning. But we frequently have winners in all handicap ranges, from about +4 to about 15. I've never seen such a wide range in any one flight, but we do it and it works pretty well.

    ReplyDelete
  3. It appears you have a very effective handicap system. I believe in a market test. Does the size of the group and the distribution of handicaps remain fairly constant? From your brief note, it looks like the answer to both questions is "yes."
    I like your ESC. It speeds up play and makes it more difficult to manipulate one's handicap. It is a little different from what is imposed by GA. Most play in Australia is Stableford where a score of more than one over net par is awarded 0 points. Therefore, the "effective" ESC (i.e., 0 points) varies by how many strokes a player gets at a hole.
    I admire your courage in entering a skins game against a +4--unless you are the +4.

    ReplyDelete
  4. The size of our group has been consistent and growing slowly, but steadily. The distribution of handicaps has moved toward the low end, but we realize that there are limits to our system and we try to see the data for what it is. Extremely high handicappers do not stand a chance in our system. Above about a 16 handicap, its not fair. We don't flight our tournaments, so we have +4 handicappers competing against 15 handicappers, and both win. The low handicappers are very consistent and they do win more frequently as a result of their consistency, but when a 10-12 handicappers shoots a net even par or net -2, they will usually win. Similarly, the +4 handicapper that shoots 70, net 74 does not win. It may take courage for a 7 handicap to go up against a +4, but it also takes courage for a +4 to participate in a game in which they could shoot 68 and not win because a 6 handicap shoots 76. The game is extremely competitive, and its hard to win for both high and low handicappers, which is the point. Everyone pretty much agrees that the format makes you play better and improve your game. Not everyone likes it. We definitely have people who try it and leave. Extremely low handicappers who are accustomed to winning every week because they are the best player in their group, often decide that they like it better in their old group where they win every week. High handicappers who are used to winning when they shoot a mediocre score and rely on strokes, also don't like it when they shoot a few shots over their handicap and don't win, like other flighted events. In our game, if you play to your handicap, you are almost guaranteed to win, but because our handicaps are lower and we only use 85% of handicaps, its much harder to do that. And many people just can't tolerate only winning when they play their absolute best. Its been very interesting. Scratch golfers hate losing to a 10 handicapper who wins when he shoots 80, but they have a ton of respect for a 10 handicapper who shoots 74 and wins. And the 10 handicapper understands that he is improving his game. Its really hard to win in our game, but the system makes everyone a better player.

    ReplyDelete
    Replies
    1. My comment on courage referred to the skins portion of your competition. The low handicap player has an advantage in most forms of skins (see "Why You Win (or Lose) at Skins" posted 6.25.2013). I am also not ready to hand-out a medal for courage to the +4 for playing in the net competition. The +4 plays as a +3. The 7-handicap plays a a 6. The +4 basically gains two strokes under your format. This bias, coupled with the built-in bias in the Handicap System that favors the low handicap player, gives the +4 an edge in the net competition. Courage is when you enter knowing the odds are stacked against you. This does not appear to be the case here.
      I do admire the equanimity of the scratch player who has a ton of respect for the player who shoots 74 with a 10-handicap. A more typical response would include questions about the validity of the handicap interspersed with a variety of expletives.
      It sounds like your group is becoming more homogeneous. That is to be expected. Most low handicap players do not like playing with high handicap players--they see the latter as proof Darwin didn't get it right. Equity concerns are also minimized as the Handicap System works best over a narrow range of abilities. In any event, I wish you continued success. .

      Delete