Tuesday, March 6, 2012

Is Tiger Woods Loss Averse: Probably Not

 Introduction and Summary - A recent article in the American Economic Review (hereinafter referred to as Pope) attempted to prove Tiger Woods and other golf professionals were loss averse. [1]  The authors claimed their research was an important contribution to economic literature by documenting loss aversion theory (LAT) in a competitive field setting.  Their assertion was based on finding PGA Tour golf professionals made slightly more of their par putts than their birdie putts.  This research was heralded in a N.Y. Times article by Alan Schwarz.[2]  David Brooks, also of the N.Y. Times, cited the finding as an example of the work of Daniel Kahneman whose work Brooks argued “will instigate a cultural shift that is already producing astounding results.”[3]  With the imprimatur of the N.Y. Times and the American Economic Review, one would expect the theory in Pope to be valid and the research behind it thorough.  This paper argues that is not the case. 
 To begin the examination, an alternative theory to explain the difference between success rates of various shot values (par, birdie, and bogey) is posited.  The alternative theory assumes the more information a golfer has about the speed and curvature of the green, the better his chances of making a putt.  Golf professionals have more information when attempting par putts than birdie putts and that explains the difference in success rates.  For the purposes of this paper, this theory is termed Rational Information Theory (RIT).

Both LAT and RIT predict a player will be more proficient on his par putts than his birdie putts.  They differ, however, on their success rates of their bogey putts relative to par putts.  LAT predicts a player will be better on his par putts than his bogey putts.  RIT predicts the player will have a higher success rate on his bogey putts than his par putts.  The data in Pope confirms players are better on their bogey putts than on their par putts.  This directly contradicts LAT.  Typically, when a theory fails to predict an experimental outcome, the theory is either revised or at least a corollary added to explain the unexpected outcome.   With a certain amount of academic hubris, however, Pope ignores the theory’s failing and concentrates only on the difference between par and birdie putts as proof of the paper’s thesis.

The various analyses conducted in Pope to affirm LAT are examined.  All of the models used in Pope do not include a variable measuring the information gained by a player from shots around the green.  It is possible the loss aversion effect measured in Pope could in fact be the contribution of this omitted variable.

Pope also claims risk aversion is shown by the player hitting birdie putts more softly than par and bogey putts.  This, it is implied, explains why players have a better success rate at par putts than birdie putts.  The data shows, however, players hit bogey putts more softly than par putts (and birdie putts) and come up short of the hole more often on bogey putts than par putts. If timidity (i.e., loss aversion) hurts a player on birdie putts, it should also harm his success rate on bogey putts.    Just the reverse is true, however, as players are better at bogey putts than par putts.   The findings on bogey putts refute LAT, but are simply dismissed in Pope as a statistical anomaly.   

The American Economic Review supposedly requires articles to describe how final data sets were developed.  Pope, however, fails to adequately define the control variables making replication of the final data set impossible.  The problems with several of the control variables are presented in the appendix.   The appendix also contains a listing of errors found in Pope.  Many of these errors are minor and do not affect the central thesis of the paper.  The number of errors, however, gives an indication of the care with which Pope was written and reviewed.

Perhaps the lesson here is data from the PGA Tour should be used only for the purposes it was intended.  This data is excellent at identifying good putters, long drivers, and a myriad of other golf statistics.  To use the data to identify quirks in the human psyche is not likely to be productive. 

Theories and Predictions – In order to assess the validity of two competing theories, we first describe each theory and the prediction it makes on the relative success rates of putts of different shot values (i.e., eagle, birdie, par).

Loss Aversion Theory (LAT) – This theory assumes a golfer will choose an optimal level of effort by setting the marginal cost of effort equal to the marginal benefit of effort when putting for birdie or eagle.  However, when putting for par, bogey or double bogey, the golfer chooses a higher effort since he derives a higher marginal benefit from making those putts in the loss domain than eagle or birdie putts which are in the gain domain.  There is also diminishing sensitivity in both domains—i.e., a player will try less hard on a bogey putt than a par putt.   LAT would then predict:

Controlling for putt characteristics, the probability of making a birdie putt is greater than the probability of making an eagle putt.  In addition, the probability of making a par putt is greater than the probability of making a bogey putt…[4] 

Rational Information Theory (RIT) – This theory assumes a golfer tries equally hard on each putt.  A player’s success, however, is a function of the information he has about the green.  Given the nature of the game, a player will usually have more information when he attempts a putt for par than when he attempts a similarly situated putt for birdie.  For example, a putt for par has usually been preceded by a previous putt or an approach shot from near the green.  On most occasions, a player cannot collect the same information for a birdie putt.  Therefore, RIT would predict:

A player would have greater accuracy on a par putt than a birdie putt.  This prediction is consistent with LAT.  RIT would predict a player is more accurate in putting for bogey than when putting for par since he typically will have more information when putting for bogey.  This prediction is in direct contradiction of LAT.   

Findings - The major difference between the two theories is in the relative accuracy of par and bogey putts.  Under LAT, the probability of making a par putt is greater than the probability of making a bogey putt.  Under RIT, the player would have a greater probability of making a bogey putt than a par putt.  Interestingly, Pope would appear to validate RIT and not the premise of the paper.  Table 2 of Pope shows the following estimates of the probability of making a putt compared with the probability of making a par putt.[5]

Table 2 (abridged) --The Effect of Different Shot Values on Putt Success


Putt Type
Probability of Making Putt Compared With Par Putt
Putt for Eagle
-0.024 (0.002)
Putt for Birdie
-0.019 (0.001)
Putt for Bogey
0.009 (0.001)



 The table also shows the estimated probability of making an eagle putt is less than the probability of making a birdie putt, and the probability of making a birdie putt is less than the probability of making a par putt.  This is what both theories would predict.

Table 2 also shows, however, the probability of making a bogey putt is greater than the probability of making a par putt.  This is inconsistent with LAT, but consistent with RIT.  This finding is confirmed in the ordinary least squares analysis presented in Table 3 shown in abridged form below.

Table 3 (abridged) – The Effect of Different Shot Values on Putt Success –Robustness Checks

Putt
Type
Model Values (OLS Estimation)
1
2
3
4
5
6
7
9
Eagle
-.040
(.002)
-.039
(.002)
-.030
(.002)
-.042
(.002)
-.039
(.002)
-.036
(.002)
-.036
(.003)
-.064
(.003)
Birdie
-.036
(.001)
-.036
(.001)
-.026
(.001)
-.029
(.001)
-.028
(.001)
-.028
(.001)
-.028
(.001)
-.030
(.001)
Bogey
.004
(.001)
.005
(.001)
.001
(.001)
.003
(.001)
.002
(.001)
.002
(.001)
.002
(.001)
.006
(.001)



 In each of the eight models presented, the probability of making a bogey putt is greater than the probability of making a par putt.  The table also shows the diminishing value of information as RIT would predict.  By the time a player hits his par putt, he has already gained a great deal of information from his previous shots or the previous shots of his fellow competitors.  The marginal information gained when attempting a bogey putt is likely to be small.  This is reflected in the very small difference in success rates between par and bogey putts (.006 in Model 8).

Pope attempts to buttress his argument with a round-by-round analysis shown in an abridged Table 5 below:

Table 5 (abridged) – The Effect of Different Shot Values on Putt Success – By Round

Value
4-Round Players Only
Round 1
Round 2
Round 3
Round 4
Eagle
-.065
-.057
-.028
-.031
Birdie
-.046
-.035
-.024
-.021
Bogey
.003
.006
.004
.003



Pope argues:[6]

“…the finding that the round of play moderates our effect is interesting because it is consistent with our reference point story.  In the first round, the reference point of par is likely to be very salient for golfers.  By the fourth round, however, other reference points such as the scores of other golfers are likely to be more salient.  These competing reference points are likely to diminish the influence of par on performance.”

 Pope seems to admit his theory is temporal in nature.  At some bewitching time, players might switch to a new reference point.  Since the difference in par and birdie success is still evident in round 4, however, at least some of the players are still clinging to their “par” reference point.    

A better and more coherent explanation of the declining difference in the birdie/par success rate is the declining value of additional information as the tournament proceeds.  A player learns from day to day about the peculiarities of the green.  By the fourth round, a player has a good idea of the speed and breaks of the green—certainly better than in the first round.  Shots from around the green in Round 4 may not provide additional new information to the player as they would in Round 1.  Under RIT, the difference in success rate by shot value should tend to zero as more rounds are played, and this is reflected in Table 5.

Table 5 is also inconsistent with LAT as players are better at bogey putts than par putts.  Pope does not explain why this inconsistency is found in Tables 2, 3, and 5.  Pope promises to revisit this curious result in the robustness section,[7] but does not do so.

In Defense of LAT - Pope contends “learning” has been accounted for.  The counter claim is it has not been properly controlled for.  Essentially, there is an omitted variable representing “learning” which is responsible for the findings in the paper.  Pope defends LAT by arguing learning has been controlled for, and loss aversion is also shown in an analysis of how far putts of various shot values are hit.  The inadequacy of each argument is described below:

1.   The Prior Putts Argument – Pope assumes a player learns from prior putts when dummy variables for the number of putts already attempted on the green by the player and the player’s fellow competitors are included in the model.  Pope writes:[8]

“The results of this specification suggest that learning is important.  As we report in column 3 (Model 3 as shown here) in Table 3, golfers are significantly more likely to make second and third putts on the green than they are to make otherwise similar first putts on the green.[9]  By including controls for prior putts on the green, the point estimates for birdie and eagle putts are reduced by 20 to 30 percent.”

Pope clearly admits learning is important.  The problem is his model does not account for short approach shots around the green.   If any player in the group putts from the fringe, for example, that shot is not accounted for in the model though it may yield information to the players.  This is not a negligible error.  If players have a 70 percent chance of being on the green in regulation, then there is a 65 percent chance at least one or more players will have missed the green.   Information gathered from their (most probably) short approach shots is not accounted for in the model.

2. Eliminating the Short Approach Shot Argument - Pope does admit a player can learn from short approach shots.  He attempts to correct for the deficiency in the model by:[10]

“…restricting our sample to par or birdie putt attempts with very long approach shots (more than 50 or more than 100 yards) where learning about the green is unlikely. ..Although the point estimates are smaller,…we continue to find highly significant differences between par, birdie, and eagles success when we include these controls.”

Pope does not document which controls were used in this analysis.  Let’s assume the complete set of controls was included.  By restricting the sample to putts preceded by long approach shots, Pope has only eliminated information a player gains from his own approach shot, and not that from his fellow competitors.  Therefore, the findings are interesting, but not compelling.

3. The Matching Model Argument - Pope uses a matching model to compare birdie and par putts taken from the same spot on a particular hole.[11]  Across these analyses, Pope found players made their par putts between 1.5 percent and 3.1 percent more often than they made their birdie putts.  This is not unexpected and would be predicted by both LAT and RIT (i.e., a player typically will have more information when he attempts a par putt).  In the matched samples, birdie putts were made approximately 85 percent of the time.  In the overall sample, birdie putts are only made 28 percent of the time.  The matched sample then must be made up of fairly short putts.  The par putt is likely to be a second putt while the birdie putt is a first putt.  The additional learning gained by the player putting for par could explain the differences found in Pope.

4. The How Hard You Hit the Putt Argument – Pope argues golfers are more risk averse when putting for birdie or eagle than when they hit par, bogey, and double bogey putts.  Pope defines risk averse putts as putts that are hit short.  Pope also expects birdie and eagle putts to be hit less hard than par, bogey and double bogey putts.  The results of the regression analysis are shown in an abridged Table 6 below:

Table 6 – Abridged

The Effect of Different Shot Values on Risk Aversion (All Missed Putts)

Shot Value
Putt Length
Left Short
Eagle
-0.800(.032)
0.013(.003)
Birdies
-0.190(.080)
0.003(.001)
Bogey
-0.365(.190)
0.007(.003)
Double Bogey
-0.053(.290)
0.008(.004)



The table shows a birdie putt, on average, will travel 0.19 inches less far than a par putt all other things being equal.  To give an idea of the magnitude of this measure of risk aversion, a ball will travel 5.27 inches in one revolution.  The effect of loss aversion is then approximately 4/100th of one revolution of a golf ball. [12]   

 The table, by the same reasoning, also shows a player is more risk averse on bogey putts than on birdie putts.  A bogey putt will travel 0.175 inches less far than a similarly situated birdie putt.  A bogey putt, according to the table, will be short 0.4 percent more often than a birdie putt.  This is contradictory to the theory of the paper, but no explanation of this curious result is given except to argue it is a statistical anomaly.

 Pope does conclude:[13]

“…birdie putts are more likely to be laid up short of the hole, but they are also significantly more likely to miss the hole to the left or right.”

This would be expected under RIT.  A Player has more information on the break of the putt on a par putt which tends to lower the size of the left/right error.   

Appendix

Errors and Omissions

Below, issues in Pope are shown in italics followed by a correction or explanation of the problem.

p. 132 – On scorecards, golfers draw a circle for holes they shot under par and a square for holes they shot over par.

Golfers, as a rule, do not make such artistic marks on their scorecards.  Television will often employ a variety of marks and colors to designate scores over or under par.  And to be technically correct, a player does not keep his own scorecard. 

p. 132 - After the second round golfers with a score that places them in the bottom third are eliminated.

For most tournaments, the cut is the low 70 and ties.  Therefore, approximately half the field plays the final two rounds. 

p. 137 -This table provides summary statistics for putts taken on the PGA Tour between 2004 and 2009.

It is not clear how Pope handled data from tournaments that differ from the norm.  The International Tournament was played with a modified Stableford format that assigned large gains to birdie and eagles.  Since this tournament does not fit the assumptions of the paper, it should have been deleted from the sample.    In the analysis that looked at putting success over four rounds, how was data from the fifth round of the Hope Classic treated?    Pope could have mentioned data were not available from the most important tournaments of the PGA season (the Masters, the U.S. Open, the British Open, and the PGA).   Pope assumes putting is the same across all tournaments regardless of their importance.  It is unfortunate data are not available to test this assumption.

p. 137 – In this regression, and across all of our regressions, we include a seventh-order polynomial for the distance to the hole.  Goodness-of-fit tests suggest that a seventh order polynomial is necessary and sufficient to control for this important variable.

Goodness-of-fit is not an acceptable reason for adopting the functional form of a variable.  There should be some theoretical reason why a seventh order polynomial should be chosen.  Moreover, the particular seventh order polynomial is never delineated, nor is the estimated coefficient for this variable ever identified in any of the regressions.

p. 138 Table 2 – The coefficient for double bogey putts is consistent with diminishing sensitivity, but the coefficient for bogey putts is not.

The “coefficients” of a logit regression do not represent probabilities, but the ratio of logged odds between the dummy variable and the reference group.[14] 

p. 139 – Differences in Player Ability – Some players may be good drivers (hitting long shots from the tee to the green) but bad putters, and others may be bad drivers but good putters.  If this were true, player differences could account for our finding that birdie putts are less accurate than par putts.  To address this question we include player fixed effects, and we report results from this regression in column 2.

What is a “player fixed effect”?  In theory, Pope must determine for each of the 2.5 million putts in the sample if was hit by a good driver or a good putter.  Pope gives no indication how players were put into various putting and driving classes. And to be correct, good drivers do not hit long shots to the green.  Driving proficiency is rated by length, accuracy, and length and accuracy in combination.  Pope probably meant players who have a relatively high percent of greens -in-regulation.   



p. 140 – We restrict our sample to birdie and par putts with very long approach shots…Although the point estimates are smaller…we continue to find highly significant differences between par, birdie and eagle success…

If the sample is restricted to birdie and par putts, how can there be a finding on “eagle” success?

p. 140 - …we include separate dummy variables for the number of putts already attempted on the green by the player and his partner.

The term is not “partner” but fellow competitor.  In  the early rounds, a player has two fellow competitors and not one as the singular form of “partner” implies.

p. 142 –Position in the Tournament – It is possible that golfers may be more likely to attempt birdie putts when they are far behind and are exerting less effort.

This sentence makes no sense unless Pope is attempting to use the teachings of a Zen master to explain performance.  There is no other reason why a player who is well behind would get more birdie chances—i.e., start hitting greens-in-regulation.

Pope does create dummy variables indicating the player’s total score if the putt is made.  These variables, however, are not necessarily good indicators of a player’s position in the tournament.  Being 5-under par on the first day is much different than being 5-under par on the fourth day of a tournament.  Being 5-under par at the U.S. Open is far different that being 5-under at the Phoenix Open.  Pope concludes the main result is not affected by these additional controls.  Later (p.146), Pope argues in the fourth round competing reference points are likely to diminish the influence of par on performance.   That is, players near the lead are likely to use the scores of other golfers as reference points.  Pope does not explain why position in the tournament is not important determinant of birdie success on p. 142, but very important on p.146.

p. 145 Table 5 – Comparison of full sample with 4-round players only.

The putts in round 3 and round 4 of the full sample are the exact same putts in round 3 and 4 of the sample restricted to players who played 4 rounds (e.g., the number of observations is 477,732 for round 3 of both samples).   Given the same sample, the estimates from the regression analysis should also be the same.  The coefficients and standard errors for the double bogey dummy variable, however, differ in the round 3 regressions.

p. 148 - This figure provides a graphical illustration of where each missed putt ---taken from over 220 (sic) inches away…

Everywhere else, the distance is cited as 270 inches.

p. 153 – Panel  A and B

A player in Panel A has a birdie coefficient of .032.  This player is not shown in Panel B.

p. 154 - …the table (7) provides the additional earnings each player would have earned had he increased (sic) his score by one stroke per tournament relative to the rest of the golfers.

The paper should have read “decrease” his score by one stroke. This error was pointed out when the paper was in draft form.[15]

p. 154 – Table 7 – Additional earnings for Tiger Woods are reported to be $945,532.

In a draft of this paper sent to the New York Times, Pope made large errors in this Table.  Pope explained to Alan Schwarz of the N. Y. Times that the estimates in Table 7 were made by a research assistant, and would be corrected in the published paper. [16]  An analysis of Tiger Woods’ earnings in 2007,[17] however reveals errors still exist.   Table 7 estimates Tiger Woods would have increased earning s $945,532 he improved by one stroke in each tournament.  A more careful analysis presented in the table below shows, Woods would have increased earnings of $1,139,841.







Tiger Woods 2007



Tournament


Winnings
Additional  Winnings if 1
Stroke Better


Comments
Buick Invitational
$936,000.00
$0
Won Tournament
WGC-Match Play
$130,000.00
$0
Match Play Tournament
Arnold Palmer Invitational
$51,058.33
$20,441.67
Move from a tie for 22nd to a 5-way tie for 18th.  Places 18 through 22nd are paid 6.5% of the total purse of $5.5M. Tiger would have earned ($5.5 *.065)/5 =$71,500.
WGC-CA
$1,350,000.00
$0
Won Tournament
Masters
$541,333.00
$268,667.00
Move from a 3-way tie for 2nd, to second alone.  Second alone would be paid $810,000. (Based on 2008 distribution results.)
Wachovia
$1,134,000.00
$0
Won Tournament
The Players
$38,700.00
$54,900.00
Move from a tie for 37th to a 10-way tie for 28th.  Places 28 through 37 are paid 6.1% of the total purse of $9M.  Tiger would have won ($9M*.061)/10
Memorial
$93,000.00
$23,000.00
Move from a tie for 15th to a 3-way tie for 13th.  Places 13 through 15 are paid 5.8% of the total purse of $6M.  Tiger would have won ($6M*.058)/3=$116,000.
U.S. Open
$611,336.00
$375,664.00
Move from a tie for 2nd to a tie for 1st.  Places 1 through 2 are paid 28.2% of the total purse of $7M.  Tiger would have the “expected” winnings of  ($7M*.282)/2=$987,000.
AT&T
$208,500.00
$64,500.00
Move from a tie for 6th to a 4-way tied for 3rd.  Places 3 through 6 are paid 18.2% of the total purse of $6M.  Tiger would have won *$6M*.182)/4=$273,000
British Open*
$120,458.00
$113,335.00
Move from a tie for 12th to a 5-way tie for 8th.  Places 8 through 12 are paid 13.5% of the total purse of $8.659M.  Tiger would have won ($8.569*.135)/5=$233,793. 
WGC –Bridgestone
$1,350,000.00
$0
Won Tournament
PGA
$1,260,000.00
$0
Won Tournament
Deutsche Bank
$522,666.67
$219,333.33
Move from a tie for 2nd to solo second.  Second place would have won 10.6% of the $7M total purse or $742,000.
BMW
$1,260,000.00
$0
Won Tournament
Tour Championship
$1,260,000.00
$0
Won Tournament
Total
$10,867,052.00
$1,139,841.00


*It was assumed that the British Open purse was distributed by the same rules as the PGA Tour follows.





[1] Pope, Devin G. and Schweitzer, Maurice E., Is Tiger Woods Loss Averse? {Persistent Bias in the Face of Experience, Competition, and High Stakes, American Economic Review (February 2011), pp. 129-157.
[2] Schwarz, Alan, “Most Pro Golfers Try to Avoid the Bogey Man,” N.Y. Times, reprinted in the Bend Bulletin, Bend Oregon, June 17, 2009.
[3] Brooks, David, “We Have Much Less Control Over Ourselves Than We Thought,” New York Times, October 20, 2011.
[4]Pope, op. cit., p.135.
[5] Putts for double bogey are not considered here.  RIT assumes a player gets more information the more actual strokes he takes.  If a player is putting for double bogey, however, he could have incurred a penalty stroke.  Therefore, a player may not have more information putting for double bogey than for par.  For example, a player hits a ball out of bounds, drives again, misses the green, and then chips on.  He now is putting for double bogey, but has the same information a player would have putting for par without the out-of-bounds penalty.
[6] Pope, op. cit., p. 146.
[7] Pope, op. cit., p. 138.
[8] Pope, op. cit., p. 140.
[9] Actually, Table 3 does not show a player is more likely to make second and third putts on the green than similar first putts.  Pope does not report the estimated coefficients of the dummy variables representing putting order.
[10] Pope, op. cit., p. 140.
[11] Pope, op. cit., p. 142.
[12] It is not clear whether the Shotlink System has sufficient accuracy to detect such minute differences.  Shotlink is operated by 250 volunteers at each tournament so “operator error” is not an insignificant problem.  There are times when the laser cannot shoot a ball, and the ball is only placed in a one-yard square quadrant.  Finally, if the laser operator cannot get a shot on the ball, he is asked to shoot the ground where he believes the ball was at rest. 
[13] Pope, op. cit. p. 149.
[14] Pampel, F. C., Logistic Regression: A Primer, Sage University Papers Series on Quantitative Applications in the Social Sciences, 07-132, Thousand Oaks, CA, p. 27.
[15] Letter to Devin Pope from the author, June 30, 2009.
[16] Email from Alan Schwarz of the N.Y. Times to the author, July 7, 2009.
[17] In the draft paper, earnings from 2008 were used.  In that year, a one-stroke improvement by Tiger Woods did not lead to a large increase in earnings (only a 4 percent increase in earnings).  It is assumed Pope switched to 2007 for dramatic effect.


Thursday, December 29, 2011

Is Tiger Woods Loss Averse?: A Debate

In June of 2009, Alan Schwarz of the N.Y. Times reported on a draft study that used putting statistics to conclude Tiger Woods was loss averse. I thought events of the previous November had pretty much proved he was not loss averse.  In any event, I read the draft study and did not find it convincing.  I wrote the author about the problems I saw, and so began a long and continuing debate on who was right.  Let’s start with the letter:   

June 30, 2009

Professor Devin Pope
Wharton School
544 Huntsman Hall
3730 Walnut Street
Philadelphia, PA 19104-6340

Dear Professor Pope,

I read an article (June 17, 2009) by Alan Schwarz of the N.Y. Times summarizing your research.[1]   Schwarz  wrote your research had two main conclusions: 1) Golf professionals  had a slightly better chance of making a par putt than a birdie putt, all other things being equal, and 2) Loss aversion costs a top player $1.2 million in prize money per year.    
In reviewing a draft of your research, it is possible your first conclusion is due to an error in variables rather than “loss aversion” as claimed in the paper.  Your second conclusion is based on an analysis rife with errors and should either be redone or omitted from the paper.  Let me address each concern:
Error in Variables – Your research indicates  all 188 professional golfers in your study exhibit bias by hitting birdie putts less accurately than they hit otherwise similar par putts.  I found it strange that not one player was better at birdie putts.  Even though golf professionals come from all races, nationalities, and socio-economic backgrounds, they all putt like a girlie-man.
One possible explanation for this curious result is a missing variable.  As stated in your article, par putts may be more accurate than birdie putts because golfers putting for par may have learned important information.  You argue it is possible players learn important information from watching their partners (technically their fellow competitors) putt on the green. To control for this effect, you include separate dummy variables for the number of putts already attempted on the green by the player and his playing companions.[2]  The research showed such information had a significant effect on whether a putt was made.  I do not believe, however, your variables measure all of the information the player has.
A PGA Tour player misses the green about 35 percent of the time.  On most of those occasions, the player picks up valuable information when he chips or putts from off the green.   The player typically only has this information on par putts, and this may account for your conclusion that par putts are more accurate than birdie putts.  I do not believe your variables measuring the putting history of a hole account for the information gained through short approach shots.  To refute my claim, you could redo your analysis using only putts where a player hit the green in regulation.  (Note: This would probably decimate your sample, since there would not be very many long par putts.)
The value of information gained from chipping or putting from off the green should decrease as the tournament progresses.  By round 4 the players and caddies should have a better idea of the breaks and speeds of each green than they did in round 1.  Therefore, the difference in the probability of making birdie and par putts should decrease as the tournament progresses.   Such a decrease is reported in Table 5 of the paper.  (A corollary would be that the absolute probability of making each type of putt should increase as the tournament progresses.   I do not believe your paper examines that issue.)
Errors in the Economic Impact Analysis – The claim that loss aversion costs a top player an average of $1.2 million per year is based on the data in Table 8 of the paper.  A cursory examination of this table shows it is rife with errors and calls any conclusion into question.  An abbreviated form of this table is reproduced below.  I will use Tiger Woods as an example of the errors.
Abridged Table 8-Understanding the Costs of Missing Birdie Putts




Golfer


Tournaments Played

Scoring Average (72 Holes)

Tournament Earnings (2008)
Additional Earnings if Scored 1 Stroke Higher(sic)
% Earnings Increase if Scored 1 Stroke Higher(sic)
Tiger Woods
6
67.7
$5,775,000
$410,000
6.6%


1.       Tournaments Played- Tiger did play in 6 events, but only 5 stroke-play events which were used in the study.
2.       Scoring Average (72 holes) – The heading should be the scoring average for 18 holes, not 72 holes.  Tiger Woods’ scoring average in 2008 was 68.9,[3] not 67.7 as reported in the table.
3.       Tournament Winnings – Winnings from the Accenture Match Play Tournament were mistakenly included in the total.  This tournament was not part of the study.   Mr. Woods’ total in the five stroke-play events was $4,425,000.
4.       Additional Earnings if Scored 1 Stroke Higher- The heading should read one stroke “lower.”  Assuming a one stroke lower score was meant, the results are still in error.  As shown in the table below, Mr. Woods won three events (Buick, Arnold Palmer, and the U.S. Open).  Scoring one less stroke would not have increased his earning in these events.  Mr. Woods placed solo second at the Masters.  Since he trailed the winner by three strokes, however, one less stroke would not have increased his earnings at this event.  One less stroke at the WGC-CA event would have pushed Mr. Woods from solo fifth place to a four-way tie for second.  He would have won an additional $183,750, not the $410,000 shown in Table 8.
Table-Tiger Woods’ 2008 Money Winnings


Tournament

Placement

Winnings
Placement With 1 Less Stroke
Winnings With 1 Less Stroke
Buick Invitational
1
$936,000
1
$936,000
Arnold Palmer Invitational
1
$1,044,000
1
$1,044,000
World Golf Championship
5
$285,000
T2
$468,750
Masters
2
$810,000
2
$810,000
U.S. Open
1
$1,350,000
1
$1,350,000
Totals

$4,425,000

$4,608,750


5.       % Earnings Increase if Scored 1 Stroke Higher- This heading should also read “1 stroke lower.” Table 8 reports a 6.6% increase in prize winnings for Mr. Woods.  Using the corrected numbers for Tiger Woods ($183,750/$4,425,000), his increase in earnings is 4.2%.  This is an error of 36%.  (Using the numbers shown in Table 8 ($410,000/$5,775,000), the percentage increase should be 7.1% not the 6.6% reported. The calculated percentages for the other players also appear to be in error.) 
I have a couple of minor editorial points.  You write the bottom third of the players are eliminated after two rounds.  Actually, it is closer to the bottom half.  Most tournaments have approximately 150 entrants.  The low 70 and ties make it to the weekend.  I assume your sample included the International Tournament which used modified Stableford scoring (bogey=-1, par = 0, birdie = 2, eagle =5).  This tournament should have been excluded from your sample.  It would be interesting to examine, however, if the probabilities remain the same when the value of a birdie or eagle are greatly increased relative to par.
You certainly have produced a thought provoking piece.  I hope these comments are of some assistance as you work through the peer review process.
Sincerely,

Laurence Dougharty
Cc: Alan Schwarz



Mr. Schwarz wrote the following response after conferring with Dr. Devin Pope, one of the authors.  Dr. Pope’s comments are in blue and identified by his initials “dp.” I have interjected my comments in italics and identified them by the initials “ld.”

I corresponded with the author of the paper and he (Dr. Pope) said the following:
dp -His second concern (about how we calculate the 1.2 million number) has a few valid points and a few not-so-valid points.  This was a simple back-of-the-envelope calculation that an RA put together for us in order to try and give people a feel for how much 1 stroke per tournament is worth.  It appears as if there are a few errors in the table.  Although not all the things he pointed out are errors. For example, he thinks we are doing this just for the tournaments in our data whereas we are actually doing it for all of the tournaments whether in our data or not.  We assume that our effect occurs in all of the tournaments and we then calculate using the entire sample.
ld – I did not think the tournament winnings calculation was restricted to tournaments where Shotlink data was available.  That would have excluded the Masters and U.S. Open.  The Accenture Match Play Tournament should have been excluded because it is impossible to measure the impact of one stroke on the final tournament standings.

dp -In other words, his second concern will probably cause us to have another RA go through and update the Table 8 for errors and then we will clean up the language so people aren't confused.  But no matter how you do it, 1 stroke = large number.
ld – An RA is doing research and his work is not checked? It makes me wonder if an RA was responsible for preparing the data for the regression analyses.

dp -His first concern regarding learning from chip shots is something we could potentially rule out that currently is not discussed in the paper.  It is extremely unlikely that this is causing our effect given that it does not explain the round-to-round differences and it also doesn't explain why people would leave it up short of the hole for birdie which is a major part of the paper.  That being said, we can hopefully rule out this concern with the data itself rather than theorizing.

Hope you find that helpful. And thanks for reading.

Alan Schwarz
The New York Times

I thought Mr. Schwarz was still convinced the research was sound, and I was only noting trivial errors.  I wrote the following e-mail to buttrress my case.  He passed it along to Dr. Pope who made comments (in blue and marked as “dp”). My counter arguments are italicized and identified as “ld.”
Mr. Schwarz,
Thank you for your courteous response.  I am still puzzled why you would base an article on an unpublished paper with questionable findings.  You may have thought the Wharton professor was the next Bill James of golf.  I do not believe that to be the case.  Moreover, a strong case can be made from the study’s own results that the “effect” (higher probability of making a par putt than a birdie putt) does not exist.      
In my original review of the study, I proposed that information gathered from a short approach shot could explain the small difference in probability for making par and birdie putts.  In his reply to you, the author wrote “it is extremely unlikely that this (my hypothesis) is causing our effect…”  This must be another “back of the envelope” probability estimate since the author has not performed any empirical test of my hypothesis.  Here are some reasons why the hypothesis could be true: 
Common sense No. 1 – Assume Player A hits an approach shot from 180 yards that stops 6 feet from the flagstick.  Assume an identical Player B putts from off the green to the same point 6 feet past the flagstick.  Who has the better chance of making the putt?  (I am assuming the players are in different groups, so there is no learning from one another.)  I suspect every Tour player you ask would choose Player B.  Missing a green should have a measurable effect on the probability of a player making his first putt. 
Common Sense No. 2 – The author reported that all 188 golf professionals in the study suffered from loss aversion.   (Was the entire gene pool of risk takers decimated by saber tooth tigers?)   It seems unlikely that not one player would be better at making birdie putts than par putts.  A more logical explanation is an important variable is missing from the regression model.  My hypothesis is a variable measuring “greens in regulation (GIR)” would eliminate the disparity between the probability of making birdie and par putts.  
dp -I can give lots of examples of biases that all people exhibit.
ld -It would have been nice if Dr. Pope had named a universal bias.  Not to jump ahead, but in the published version of the article, Dr. Pope did indeed find players who were not loss averse. 
Empirical Evidence No. 1 –If my hypothesis is true, players who have a high GIR percentage   should  have a smaller difference in the probability of making a par or birdie putt.  The author found the difference went up with a players rank (Figure 8b)—i.e., the 50th ranked player would be expected to have a larger difference than the 1st ranked player.  I assume a player’s GIR ranking is positively correlated with his ranking.  If so, the higher the GIR ranking, the smaller the difference in the probability of making par and birdie putts.  This is exactly what my hypothesis would predict.
                  dp -For other reasons, I have calculated the GIR percentage in our golf dataset.  I went ahead and quickly ran the correlation between GIR Perc and our par-birdie differential using the 188 golfers in our sample.  The correlation was almost zero (t-stat: .52; p = .65). 
ld – Dr. Pope does not present his work so it is hard to critique.  Did he correlate GIR and the par-birdie differential by tournament, by year, or use averages over the study period?  He did plot the par-birdie differential against a player’s 2007 world ranking.  Since rankings change during the study period, Figure 8b should be viewed with skepticism.
 Empirical Evidence No 2 – In your article, you cited Paul Goydos and Geoff Olgilvy as two players who made par putts much more often than birdie putts.  They also ranked low in GIR in 2007-- Goydos was 192nd and Ogilvy was 158th.  This would lend more credence to the hypothesis.
 dp -See above result.  Also, note in the article that Tiger Woods was about average in par-birdie differential even though he is by far the best at GIR Perc.
Theoretical Evidence – The author argues it is highly unlikely my hypothesis is true since it does not predict the decline in the “effect” over the tournament.  In fact, it does predict such a decline.  By the fourth round, the player and his caddie have probably seen 8 to 9 putts on each green.  The value of information gleaned from a short approach shot diminishes –i.e., you would pay more for information on a green you had not seen before than one you play regularly.  As the value of “approach knowledge” diminishes, my hypothesis would predict the decline in the effect.
dp -This is possible, but the evidence suggests that this is not what is driving the results. 
ld – What evidence?
The author argues my hypothesis does not explain why “players leave it up short if the hole for birdie which is a major part of the paper. “  He is correct.  But do players leave it short?  The author’s methodology is questionable—confusing maybe a better word.  He writes “birdie putts are hit .29 inches less hard that par putts.   This is confusing since “inches” is not a unit of force.  A reasonable assumption is he found that on putts of equal length, missed par putts went .29 inches further than missed birdie putts.  This is only 5 percent of one revolution of a golf ball.  If my assumption is correct, the reported difference in distance is physically insignificant even if not statistically insignificant. (Note: The paper does not explicitly state the equation being estimated, define the independent variables, nor specify how those variables are measured.  Given the peculiar estimating model, I do not have much confidence in the results.)
 dp -Yes, inches is not a unit of force…  the meaning of the sentence, however, is obvious.  The paper does provide the equation being estimated and defines the variables.  The specification used in Table 6 is the same as Column (4) of Table 3.  Furthermore, .29 inches is statistically significant (the standard error is reported in the table for everyone to see).
ld – If there was an equation, Dr. Pope could have referenced it.  An “obvious” meaning of the sentence is missed par putts travel .29 inches further than missed birdie putts. Shotlink, however, only promises an accuracy of 1 inch when measuring ball position.  The accuracy diminishes if the player taps in before the operator can sight the position of the ball.  In that case, the laser operator is instructed to shoot the ground where the ball was.  Given the accuracy of Shotlink a difference of .29 inches may be statistically significant but still insignificant. 
Table 6 of the study presents estimates of the probability that the ball will be hit short of the hole.  Again, the units attached to the findings are not clear.  I will assume the author claims a birdie putts is 0.5 percent more likely to come up short than a par putt.  If this is the finding, I find the percentage difference negligible.
dp -.5% is correct and this is significant with 99.99% confidence (again, see the standard error in the table).
ld -Let’s assume this is true.  Out of a 1000 putts of each kind, a player will leave three more birdie putts short.  That may be statistically significant, but I do not see such a small difference as important in explaining human behavior.  
An important finding of the study is buried in footnote 12.  The author writes “for putts less than 200 inches (16.7 feet), we find no differences.  Notably, the risk aversion account cannot explain the entire par-birdie accuracy gap.”  I assume this means birdie and par putts within this length are hit with the same force and come up short with equal probability.  Then what can explain the difference in the probability of making par and birdie putts shown in Figure 2?  Possibly GIR?  If the author’s theory cannot explain the “effect” in a region where most putts occur, the theory should be rejected.
dp -As any golfer should expect, no one is going to leave a 1 foot putt short of the hole.  The point of this footnote and the whole point of Columns 3 and 4 of Table 6 are to show that where you really start to see the difference between leaving a par and a birdie putt short are in the longer putts.  This is what one would expect!  For longer putts, Table 6 shows that birdie putts are 2.3% more likely on average to be left short of the hole (extremely statistically significant).
ld – I am not talking of 1-foot putts.  The draft is saying there is no effect on putts shorter than 16.7 feet.  The vast majority of par putts are from within this distance.  The author should have stratified his findings over narrower segments (e.g., 5-10 feet, 10-15 feet).  The sample size of each type of putt should also have been reported (e.g., How many par putts of over 270 inches were there?).   
Finally, the author argues for inclusion of the Accenture Match Play in Table 8 since “our effect occurs in all of the tournaments.”   The effect, however, in match play would not be measured in relationship to par, but a player’s standing with his opponent—a completely different loss function than stroke play.  But if the author wants to include all tournaments, he should also throw in winnings from the European Tour.
dp -Admittedly, this table could be done in a variety of ways.  Our numbers would be even larger if we included winnings from the European Tour.  It is just an attempt to give people who don’t understand what 1 stroke per tournament means a way of understanding the magnitude. 
ld – I don’t believe the author understands the difference between match and stroke play.
I do not mean to be too harsh on the study.  It is after all only in the draft stage.  My problem is with the N.Y. Times running with the results without fact checking.  I suspect the allure of having Tiger Wood’s name in the study’s title made it difficult to pass up.  This is not Jason Blair type stuff, however.  The only damage would be you placing one more extraneous thought in Jim Furyk’s head while he putts.  Furyk should be able to overcome this, but I have taken him off my fantasy golf team just in case.

Laurence Dougharty

In 2011, the study was published in the American Economic Review.  This study will be critiqued in the next post.





[1] Pope, Devin and Maurice Schweitzer, “Is Tiger Woods Loss Averse?  Persistent Bias in the Face of Experience, Competition, and High Stakes,” June 2009, Unpublished.
[2]  I did not realize the PGA Tour recorded the order of putts.  It must have been very time consuming to label each putt by two dimensions (the number of preceding putts by Player A, and the number of preceding putts by all other players. (Note: After writing this letter I learned Shotlink records the time of each shot.  I assume it is up to the researcher to develop a program that would sort the shots by each group on each green by time.)
[3] See PGATOUR.COM, Tiger Woods, Career Details, 2008.

Monday, November 21, 2011

Golf Digest: Press Adjunct of the USGA


     One reason for the paucity of research on golf handicapping is the USGA's virtual monopoly of the issue.  This monopoly guarantees there will be no forum for competing ideas.  It is like a world without combatants Google, Apple, and Microsoft, but only Microsoft. The press could play a role in questioning this monopoly, but it has not.  The reasons are probably twofold. First, golf writers are not trained in the subject matter, and find it easiest to parrot the USGA line.  Second,  golf publications have a symbiotic relationship with the USGA so there is little to be gained by challenging its host organism.  Let's go back in time to document one incident in the fawning relationship between Golf Digest and the USGA.
         Nothing demonstrates the insular mentality that resides USGA headquarters than a letter written to Golf Digest by Dean Knuth, Director of Handicapping. The letter demonstrates both the arrogance and ignorance by which the USGA rules the game.  It started when a reader wrote to Golf Digest (December 1996) challenging Knuth’s analysis:

In the October Digest (“The Gatescate Scandal”, Dean Knuth is quoted a saying the odds are 1 in 200 that you beat your handicap by three strokes, 1 in 570 that you beat it by five strokes, and 1 in 82,000 that you beat it by 10 strokes.  I suggest Dean check the batteries on his calculator.  If the odds are 1 in 200 that you beat your handicap by three strokes then Greg Norman would shoot a 66 once in every 200 rounds.  It would also mean that your garden variety, 18-handicapper would be a model of consistency, shooting within two strokes of his handicap more than 90 percent of the time.
            As far as 10 under goes, I’ll bet nearly everyone with a handicap of more than 15 had done it a least once and in a lot fewer rounds than 82,000.  I don’t have any idea what Bill Gates handicap is, but it’s pretty clear ol’ Dean is the wrong guy to calculate it.  He is statistically challenged.

            Golf Digest let Knuth reply directly below the reader’s letter--a privilege very few if any are afforded:

First off, Greg Norman had a USGA handicap index in 1995, his level of play equated to a plus 7.5.  That is to say, 7.5 strokes better than a scratch golfer.  The average USGA course rating is approximately 71, so Greg’s better-half scoring average would be 63.5.
            On tour, the courses are set up to a course rating of 76 on the more difficult stops.  There his best average would be 68.5.  To beat his handicap index by three strokes, he would have to shoot 60.5 on the average course (that we play), or 65.5 on the strong tour course.  (Note: The average tour player is a plus 3.5).
            With respect to the garden-variety 18-handicapper, he or she averages three strokes over his or her course handicap and plays to it 25 percent of the time.  Beating your handicap by three strokes or more – twice in tournaments – becomes such a rare event that Section10-3 of the USGA handicap system automatically reduces the player’s USGA index.  Less than 1 percent of golfers are reduced under that procedure, so its an uncommon event except by the sandbaggers of the links.  However, it is true that the size of the handicap index does affect the probability of making a low net score.
            I have been called a number of thing, but never before “statistically challenged.”  I scored a perfect 800 on the math section of my SAT test in high school, graduated from the Naval Academy and got a masters in system technology.  USGA statistics are based on a database  of million of scores and were worked out by our handicap research team of statisticians, mathematicians, and professors.

            Knuth argues that he is not “statistically challenged” because he scored 800 on his SAT and had a masters in system technology.  Arguing “credentials” is never an effective or reasonable method to quiet one’s critics.  This especially true when those credentials consist of a 30-year-old test score and a degree that is the academic equivalent of making the cut at the B.C. Open.
            Knuth commits so many errors in his short letter, he makes the reader’s case for him.  Knuth wrote that beating your handicap by three strokes or more –twice in tournaments – automatically reduces a player’s USGA index.  Knuth’s assertion is not true.  A player who has only two tournament scores must average at least four strokes below his index before his index is reduced.  It is disconcerting that the USGA Director of Handicapping demonstrates a lack of understanding of the system he governs (See section 10.3 of The USGA Handicap System). 
            Knuth mistakenly wrote beat your “handicap” when he should have written beat your “index.”  The USGA procedure reduces a player’s index for performance that beats his index, not his handicap.  A player who beats his handicap twice by four strokes may or may not get a penalty reduction depending on the Slope Rating of the course.  This can be easily demonstrated.  The player’s score is:

            Score = Course Rating + (Handicap - 4)

The player’s index differential for this round is:

Index Differential = (Score - Course Rating) (113/Slope Rating)

Substituting,
                                         = (Handicap - 4) • (113/Slope Rating)
                                         = (Index • Slope Rating/113 - 4) • (113/Slope Rating)
                                         = Index - 4• (113/Slope Rating)

The penalty for exceptional tournament performance is determined in part by the difference between the a player’s index and the index differential:

            Index - Index Differential = 4 • (113/Slope Rating)

If the Slope Rating were 155, the player would only beat his index by 2.9 strokes, and no penalty would be assessed under Section 10.3.  If the player beat his handicap twice by four strokes on course with a low Slope Rating (e.g., Slope Rating = 80), the player would be penalized. 
            Knuth mysteriously omits the effect of the Slope System in estimating Greg Norman’s scores.  It is mysterious since Knuth is the major proponent of the System.  Knuth argues that if Norman had an index of +7.5, his best average would be 68.5 on tour course with a rating of 76.  Knuth’s assertion would be true if the Slope Rating of the Course was 113.  More than likely, however, the Slope Rating would be at least 145.  In that case Norman’s best average would be 66.4 (i.e., Norman's unrounded handicap would be 9.6.  This illustrates a troubling paradox of the Slope System: the higher the Slope Rating, the lower the score Norman would be expected to shoot.
            Golf Digest was made aware of these errors, but never ran any correction.  Knuth also did not respond.  Knuth knew he did not have command of the facts or the theory.  If his opponent could not be cowered by his impressive (but not verified) high school test scores, Knuth knew he would lose the argument on the merits.
            Golf Digest as a major publication could be a countervailing force to the USGA.  Like the press in other settings, it could serve as a watchdog.  Golf Digest has consistently rejected that role believing its institutional health is better served by being compliant rather than critical.