The Francis Scheid Affair

            Nothing illustrates the bankrupt nature of the USGA handicap program better that what I term the “Francis Sheid Affair.”  It exposes the dearth of research behind handicap policy, and the unwillingness of the USGA to address criticism of the handicap section.

            The affair started innocently enough when I read a paper by Francis Sheid of the USGA, presented at the Second World Scientific Congress of Golf (1994).  The Congress guidelines required papers to be original.  The organizers of the Congress properly believed that if papers presented were merely rehashes of old work, the Congress would suffer in prestige. 

Since Sheid was a member of the USGA Handicap Research Team, I was expecting an article on the cutting edge of research on handicap issues.  Scheid’s article was entitled “The Search for The Perfect Handicap.”[1]  In this article, Sheid evaluated various handicap systems.  He concluded that “the current official handicap systems do about has well as any.”  It was an interesting piece, but clearly not of overwhelming significance.

            I was struck, however that I had seen the title somewhere before.  I went back to the proceedings of the First Scientific Congress of Golf(1990).  In a paper presented at this conference, Scheid had included the following reference:

 

"Scheid, F. J. (1978) The search for the perfect handicap.  Proceedings of the Winter Simulation Conference."

         The reference is not well documented.  There is, for example, no indication which organization published the paper.  This had the effect, intended or not, of deterring anyone from finding this paper.  However, with the help of some of the best research librarians in the United States the article was uncovered.[2]

            For all intents and purposes Scheid’s 1978 and 1994 articles were identical.  This meant 1) Sheid violated the conference guidelines calling for original research , 2) Sheid, a lead member of the Handicap Advisory Team, had not done anything in the period 1978-1994 he thought worthy of publishing.

            Scheid’s 1978 paper was sent to Martin Farrally, the Congress chairman, along with an analysis of the similarity between the two papers.  I thought he would be outraged that the USGA had pulled such a stunt on his conference.  Unfortunately, he was not.  Instead, Farrally replied:

“Since I have no competence in golf statistics I am not able to comment on whether Scheid’s paper was a regurgitation of 16 year old piece of work.  As the Congress director, I have to trust the judgement (sic) of the theme co-ordinators.”[3]

            Farrally did not say that he would investigate the claims.  Here merely stated that if it got by his theme coordinators, that was good enough for him.  He did not question his theme coordinator if he/she was aware the 1994 paper was a re-write of an earlier work.  Since some of the coordinators were paid consultants to the USGA, it is not likely they judged USGA generated papers too harshly.  Moreover, the entire Congress depended upon support from the USGA.  Apparently Farrally, felt that if the USGA wanted to slap the Congress in the face that was all right, because they had paid for that privilege.

            Even though Farrally refused an independent investigation of the affair, he did send my complaint on to Scheid.  Sheid wrote to Farrally that he was innocent.[4]  Farrally did not send a copy of this letter to me but did send a copy to David Fay, Executive Director of the USGA.  Fay mistakenly believed that Sheid presented a credible explanation and forwarded Scheid’s letter to me. 

Sheid’s defense was incredulous, and Fay should have spotted it.  In Fay’s defense, he knows little about statistics.  To his discredit, he never made any effort to settle the dispute, and became the leader in the effort to protect Scheid from the punishment he deserved.

            The weakness of Scheid’s defense can easily be seen.  His defense is shown below in italics.  A rebuttal of each of his points follows.

The original research (i.e., the 1978 paper) was based on the limited data available to me at the time.  I completely repeated all experiments using an entirely new and much larger data base.

 A comparison of the size of the samples used in each paper does not indicate a larger data base in Scheid-1994:

· Scheid-1978 – “At each of the courses for which data was in hand, fifty pairs of golfers (at eighteen courses) were chosen…At least…80,000 matches were played at each club to produce these results.

· Scheid-1994 – “at each of several clubs fifty pairs of golfer were select…About 80,000 matches were simulated a each club.

Since in common usage “several” is less than eighteen, it appears the sample size in 1994 was actually smaller than in 1978. Unfortunately, Scheid did not even meet the minimal scientific standard of documenting his sample size in his 1994 paper, so his claim could be conclusively disproved.

Scheid, however, proves his own guilt in another paper written for the 1994 Congress.[5] Sheid references two papers entitled The Search for the Perfect Handicap (1979), and the Search for the Perfect Handicap Part II, 1980. If he really did the research in the 90’s as he claims, why wouldn’t he reference the later work? The obvious answer is that there was no later work.

The results were reassuringly close to the original ones though not identical. Scheid tries to make the argument that since the results were different, there must be two different research studies.  His argument does not stand up to even a cursory inspection.

            Tables 1 and 2 present a comparison of the error tables contained in Scheid’s two papers.  Tables1 and 2 only report handicap types that are presented in both Scheid-1978 and Scheid-1994.  Not one handicap type reported in both of Scheid’s papers has a different error.  Where a comparison can be made, all the results are identical!

            The probability that the median error for fourteen different handicap types would remain exactly the same in two different samples is infinitesimal (the probability of this result is approximately .0002).

 Table 1

Median Errors, Match Play

 

Error	Handicap Type Scheid-1978	Handicap Type Sheid-1994
.4	Mean, (2-19), (3-18), (4-17), (8,13)	Mean, (2-19), (3-18), (4-17), (8,13)
.5	(1-15), MEDIAN,(6-15), (B15), (8)	(1-15), MEDIAN, (6,15), B15,(8)
.7	B12	B12
.8	USGA	USGA
1.0	(2), (1-5)	(2), (1-5)
>1.0	PT12	PT12

 

Table 2

Median Errors to Equalize Percentile 1 Scores

Error	Handicap Type Sheid-1978	Handicap Type Scheid-1994
1.2	(1), (1,5), Norm*	(1), (1-5),NORM
1.4	(2), (2-19), USGA, (1-15)	(2), (2-19), USGA, (1-15)
>1.4	PT12, MEDIAN,	PT12, MEDIAN

·         NORM is reported as 1.3 in Scheid-1978.  The error category “1.3” is not reported in Scheid-1994.  It is assumed that the error in NORM was rounded down to at “1.2” error.

 

After all, reproducibility is important in science and I was delighted with the confirmation that the larger experiment brought.

 

To get exactly the same results 16 years apart is indeed a spectacular finding.  But if it was so important, why did not Scheid report it in 1994?  The only reasonable conclusion for omitting such a spectacular finding is that Scheid did not want to draw attention to his previous paper.

 The earlier paper…was offered about twenty years ago.  I was anxious that the scientific golf community hear of its major result.  The Congress seemed the ideal place to do this.

 If Scheid was so anxious, it has to be asked why he did not submit his paper to the 1990 Congress rather than wait until 1994?  It is even more curious that his 1990 Congress submission (“On the Normality and Independence of Golf Scores”) contained a reference to Scheid-1978, but Scheid-1994 did not.  In 1990 paper he did not allude to his “major “ finding.  He simple listed the paper in the reference section even though it had nothing to do with the body of work.  Simply put, the arguments Scheid makes in his defense are not consistent with the facts in the case.  

The paper also included a new result, that even a two-dimensional system based on the normal model would not bring a significant improvement in accuracy… 

This is not a new result but appears in Scheid-1978.  In Scheid-1978, he writes “The normal model also does as well as any…(But) official USGA and British handicaps take a close second place.”  Scheid-1994 draws the same conclusion

Scheid does add an analysis of team play in Scheid-1994.  This is a minor addition, however, and has no effect on the overall conclusion of the research.

            The USGA had steadfastly refused to admit that Scheid violated the rules of the Congress.  David Fay Executive Director of the USGA, wrote: “we will not censure Francis Scheid or offer an apology.”[6]  Never, however, has the USGA offered an affirmative defense of Scheid’s conduct.  This has been consistent with the USGA policy of dealing with criticism – attack the messenger and ignore the message.

            The concern here is not that Scheid submitted old research, but that his sixteen-year old work was the best the USGA had to offer.  The USGA should be the recognized authority on handicapping as it is on equipment testing.  The USGA has not shown the breadth and depth of research on handicapping issues to deserve such status.

            During the past ten years, major changes in the handicap system have been implemented – the Slope System, a revised and re-revised equitable stroke control, and the reduction in index for exceptional tournament performance.  Yet the USGA has not published any empirical research demonstrating these revisions have had a positive effect on the equity of competition. 

            The poor performance of the USGA is due in part to its monopoly status.  Members of the handicap research team appear to be selected because they had some commercial tie to the USGA or its executives.  There is no record of the USGA seeking to hire the best talent in the country to study handicapping problems.  Instead the USGA has relied on the “old crony” system and the results show it.  As documented throughout blog, the USGA has made numerous mistakes and questionable judgment calls on handicap policy.  Unfortunately, there are no forces on the horizon that could change the USGA.  The USGA Executive Committee will continue to rule by its inherent motto, “We’re God, and you’re not.”

 

---

[1] Sheid, F.J., “The search for the perfect handicap,” Science and Golf II: Proceedings of the World Scientific Congress of Golf,  Edited by A.J. Cochran and M.R. Farrally, E & FN Spn, London, 1994.

[2] Scheid, Francis, the Search for the perfect handicap, 1978 Winter Simulation conference, IEEE, 1978

 [3] Letter from Martin Farrally, Director of the World Scientific Congress of Golf to the author, December 13, 1994

[4] Letter from Francis Scheid to Martin Farrally, April 13, 1995.

[5] D.L. Knuth, F.J. Scheid, and F.P. Engel, “:Outlier identification procedure for reduction of handicaps,” Scheid can Golf II: Proceedings of the 1994 World Scientific Congress of Golf, E & FN Spon, London, 1994

[6] Letter from David Fay, Executive Director of the USGA to the author, December 7, 1998.