Regression toward the mean

From Wikipedia, the free encyclopedia

In statistics, regression toward the mean^[1]^[2], also called reversion to the mean, is defined as follows:

Let X₁, X₂ be random variables with identical marginal distributions with mean μ. The joint distribution of X₁ and X₂ is said to exhibit regression toward the mean if for every number c > μ, we have μ ≤ E[X₂ | X₁ = c] < c, with the reverse inequalities holding for c < μ.^[3]

Informally, we define regression toward the mean as follows. Consider a population of widgets. Each widget has two numbers, X₁ and X₂ (say, its left span (X₁ ) and right span (X₂)). Suppose that the probability distributions of X₁ and X₂ in the population are identical, and that the means of X₁ and X₂ are both μ. We now take a random widget from the population, and denote its X₁ value by c. (Note that c may be greater than, equal to, or smaller than μ.) We have no access to the value of this widget's X₂ yet. Let d denote the expected value of X₂ of this particular widget. (i.e. Let d denote the average value of X₂ of all widgets in the population with X₁=c.) If the following condition is true:

Whatever the value c is, d lies between μ and c (i.e. d is closer to μ than c is),

then we say that X₁ and X₂ show regression toward the mean.

The following is an example of regression toward the mean. A class of students takes two editions of the same test on two successive days. (Referring to the definition above, the widget is a student. X₁ and X₂ are respectively the first and second test scores of a student.) It has frequently been observed that the worst performers on the first day will tend to improve their scores on the second day, and the best performers on the first day will tend to do worse on the second day. The phenomenon occurs because student scores are determined in part by underlying ability and in part by chance. For the first test, some will be lucky, and score more than their ability, and some will be unlucky and score less than their ability. Some of the lucky students on the first test will be lucky again on the second test, but more of them will have (for them) average or below average scores. Therefore a student who was lucky on the first test is more likely to have a worse score on the second test than a better score. Similarly, students who score less than the mean on the first test will tend to see their scores increase on the second test.

[edit] History

The concept of regression comes from genetics and was popularized by Sir Francis Galton during the late 19th century with the publication of Regression Towards Mediocrity in Hereditary Stature. Galton observed that extreme characteristics (e.g., height) in parents are not passed on completely to their offspring. Rather, the characteristic in the offspring regress towards a mediocre point (a point which has since been identified as the mean). By measuring the heights of hundreds of people, he was able to quantify regression to the mean, and estimate the size of the effect. Galton wrote that, "the average regression of the offspring is a constant fraction of their respective mid-parental deviations." This means that the difference between a child and its parents for some characteristic is proportional to its parents deviation from typical people in the population. So if its parents are each two inches taller than the averages for men and women, on average it will be shorter than its parents by some factor (which, today, we would call one minus the regression coefficient) times two inches. For height, Galton estimated this correlation coefficient to be about 2/3: the height of an individual will measure around a mid-point that is 2/3rds of the parents' deviation.

Although Galton popularized the concept of regression, he fundamentally misunderstood the phenomenon; thus, his understanding of regression differs from that of modern statisticians.^{[citation needed]} Galton was correct in his observation that the characteristics of an individual are not determined completely by their parents; there must be another source. However, he explains this by arguing that, "A child inherits partly from his parents, partly from his ancestors. Speaking generally, the further his genealogy goes back, the more numerous and varied will his ancestry become, until they cease to differ from any equally numerous sample taken at haphazard from the race at large."^[4] In other words, Galton believed that regression toward the mean was simply an inheritance of characteristics from ancestors that are not expressed in the parents: he did not understand regression to the mean as a statistical phenomenon. In contrast to this view, it is now known that regression toward the mean is a mathematical inevitability: if there is any random variation between the height of an individual and parents – if the correlation is not exactly equal to 1 – then the predictions must regress to the mean regardless of the underlying mechanisms of inheritance, race or culture. Thus, Galton was attributing random variation in height to the ancestry of the individual.

[edit] Importance

The most important reason to care about regression toward the mean is in the design of experiments.

Take a hypothetical example of 1,000 males of a similar age who were examined and scored on the risk of experiencing a heart attack. Statistics could be used to measure the success of an intervention on the 50 who were rated at the greatest risk. The intervention could be a change in diet, exercise, or a drug treatment. Even if the interventions are worthless, the test group would be expected to show an improvement on their next physical exam, because of regression toward the mean. The best way to combat this effect is to divide the group randomly into a treatment group that receives the treatment, and a control group that does not. The treatment would then be judged effective only if the treatment group improves more than the control group.

Alternately, a group of disadvantaged children could be tested to identify the ones with most college potential. The top 1% could be identified and supplied them with special enrichment courses, tutoring, counseling and computers. Even if the program is effective, their average scores may well be less when the test is repeated a year later. However, in these circumstances it may be considered unfair to have a control group of disadvantaged children whose special needs are ignored. A mathematical calculation for shrinkage can adjust for this effect, although it will not be as reliable as the control group method (see also Stein's example.)

The effect can also be exploited for general inference and estimation. The hottest place in the country today is more likely to be cooler tomorrow than hotter. The best performing mutual fund over the last three years is more likely to see performance decline than improve over the next three years. The most successful Hollywood actor of this year is likely to have less gross than more gross for his or her next movie. The baseball player with the greatest batting average by the All-Star break is more likely to have a lower average than a higher average over the second half of the season.

[edit] Misunderstandings

The concept of regression toward the mean can be misused very easily.

In the student test example above, it was assumed implicitly that what was being measured did not change between the two measurements. But suppose it was a pass/fail course and you had to score above 70 on both tests to pass. Then the students who scored under 70 the first time would have no incentive to do well, and might score worse on average the second time. The students just over 70, on the other hand, would have a strong incentive to study overnight and concentrate while taking the test. In that case you might see movement away from 70, scores below it getting lower and scores above it getting higher. It is possible for changes between the measurement times to augment, offset or reverse the statistical tendency to regress toward the mean. Do not confuse causal regression toward the mean (or away from it) with the statistical phenomenon.

Do not think of statistical regression toward the mean as a causal phenomenon. If you are the student with the worst score on the first day's exam, there is no invisible hand to lift up your score on the second day, without effort from you. If you know you scored in line with your ability, you are equally likely to score better or worse on the second test. On average the worst scorers improve, but that's only true because the worst scorers are more likely to have been unlucky than lucky. You know how lucky or unlucky you were, so regression toward the mean is irrelevant from your point of view.

Although individual measurements regress toward the mean, the second sample of measurements will be no closer to the mean than the first. Consider the students again. Suppose their tendency is to regress 10% of the way toward the mean of 80, so a student who scored 100 the first day is expected to score 98 the second day, and a student who scored 70 the first day is expected to score 71 the second day. Those expectations are closer to the mean, on average, than the first day scores. But the second day scores will vary around their expectations, some will be higher and some will be lower. This will make the second set of measurements farther from the mean, on average, than their expectations. The effect is the exact reverse of regression toward the mean, and exactly offsets it. So for every individual, we expect the second score to be closer to the mean than the first score, but for all individuals, we expect the average distance from the mean to be the same on both sets of measurements.

Related to the point above, regression toward the mean works equally well in both directions. We expect the student with the highest test score on the second day to have done worse on the first day. And if we compare the best student on the first day to the best student on the second day, regardless of whether it is the same individual or not, there is no tendency to regress toward the mean. We expect the best scores on both days to be equally far from the mean.

[edit] Regression fallacies

Main article: regression fallacy

Many phenomena tend to be attributed to the wrong causes when regression to the mean is not taken into account.

An extreme example is Horace Secrist's 1933 book The Triumph of Mediocrity in Business, in which the statistics professor collected mountains of data to prove that the profit rates of competitive businesses tend toward the average over time. In fact, there is no such effect; the variability of profit rates is almost constant over time. Secrist had only described the common regression toward the mean. One exasperated reviewer, Harold Hotelling, likened the book to "proving the multiplication table by arranging elephants in rows and columns, and then doing the same for numerous other kinds of animals".^[5]

The calculation and interpretation of "improvement scores" on standardized educational tests in Massachusetts probably provides another example of the regression fallacy. In 1999, schools were given improvement goals. For each school, the Department of Education tabulated the difference in the average score achieved by students in 1999 and in 2000. It was quickly noted that most of the worst-performing schools had met their goals, which the Department of Education took as confirmation of the soundness of their policies. However, it was also noted that many of the supposedly best schools in the Commonwealth, such as Brookline High School (with 18 National Merit Scholarship finalists) were declared to have failed. As in many cases involving statistics and public policy, the issue is debated, but "improvement scores" were not announced in subsequent years and the findings appear to be a case of regression to the mean.

The psychologist Daniel Kahneman, winner of the 2002 Nobel prize in economics, pointed out that regression to the mean might explain why praises often do not work, but rebukes often do.^[6]

“

I had the most satisfying Eureka experience of my career while attempting to teach flight instructors that praise is more effective than punishment for promoting skill-learning. When I had finished my enthusiastic speech, one of the most seasoned instructors in the audience raised his hand and made his own short speech, which began by conceding that positive reinforcement might be good for the birds, but went on to deny that it was optimal for flight cadets. He said, "On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don't tell us that reinforcement works and punishment does not, because the opposite is the case." This was a joyous moment, in which I understood an important truth about the world: because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them. I immediately arranged a demonstration in which each participant tossed two coins at a target behind his back, without any feedback. We measured the distances from the target and could see that those who had done best the first time had mostly deteriorated on their second try, and vice versa. But I knew that this demonstration would not undo the effects of lifelong exposure to a perverse contingency.

”

UK law enforcement policies have encouraged the visible siting of static or mobile speed cameras at accident blackspots. This policy was justified by a perception that there is a corresponding reduction in serious road traffic accidents after a camera is set up. However, statisticians have pointed out that, although there is a net benefit in lives saved, failure to take into account the effects of regression to the mean results in the beneficial effects' being overstated. It is thus claimed that some of the money currently spent on traffic cameras could be more productively directed elsewhere.^[7]

Statistical analysts have long recognized the effect of regression to the mean in sports; they even have a special name for it: the "Sophomore Slump." For example, Carmelo Anthony of the NBA's Denver Nuggets had an outstanding rookie season in 2004. It was so outstanding, in fact, that he couldn't possibly be expected to repeat it: in 2005, Anthony's numbers had dropped from his rookie season. The reasons for the "sophomore slump" abound, as sports are all about adjustment and counter-adjustment, but luck-based excellence as a rookie is as good a reason as any.

Regression to the mean in sports performance may be the reason for the "Sports Illustrated Cover Jinx" and the "Madden Curse." John Hollinger has an alternate name for the law of regression to the mean: the "fluke rule," while Bill James calls it the "Plexiglass Principle."

Because popular lore has focused on "regression toward the mean" as an account of declining performance of athletes from one season to the next, it has usually overlooked the fact that such regression can also account for improved performance. For example, if one looks at the batting average of Major League Baseball players in one season, those whose batting average was above the league mean tend to regress downward toward the mean the following year, while those whose batting average was below the mean tend to progress upward toward the mean the following year.^[8]

[edit] A mathematical note

The following is an interesting mathematical fact:

Let X, Y be random variables with any joint distribution (discrete or continuous). Assume means exists and that X and Y have identical marginal distributions. Then for all c in the range of the distribution, so that $P(X \ge c) \ne 0$ , we have $E[ Y| X \ge c ] \le E[ X| X \ge c ]$ , with the reverse inequality holding for all $X \le c \,\!$ .

[edit] Proof

First we look at some probabilities. By elementary laws:

$P[X \ge c] = P[X \ge c \land Y \ge c] + P[X \ge c \land Y < c]$ and

$P[Y \ge c] = P[X \ge c \land Y \ge c] + P[X < c \land Y \ge c]$

But the marginal distributions are equal, which implies

$P[X \ge c] = P[Y \ge c]$

So taking these three equalities together we get

$P[X \ge c \land Y < c] = P[X < c \land Y \ge c]$

Going on the conditional probabilities we infer that

$P[Y < c | X \ge c ] = \frac {P[X \ge c \land Y < c]}{P[X \ge c]} = \frac {P[X < c \land Y \ge c]}{P[Y \ge c]} = P[X < c | Y \ge c ]$

Looking now at expected values we have

$E[Y | X \ge c] =$

$P[Y \ge c | X \ge c]\, \times\, E[Y | X \ge c\, \land\, Y \ge c] \;+\; P[ Y < c | X \ge c]\, \times\, E[Y | X \ge c\, \land\, Y < c]$

But of course

$E[Y | X \ge c\, \land\, Y < c] < c$ , so

$E[Y | X \ge c] \le$

$P[Y \ge c | X \ge c]\, \times\, E[Y | X \ge c\, \land\, Y \ge c] \;+\; P[ Y < c | X \ge c]\, \times\, c$

Similarly we have

$E[Y | Y \ge c] =$

$P[X \ge c | Y \ge c]\, \times\, E[Y | X \ge c\, \land\, Y \ge c] \;+\; P[ X < c | Y \ge c]\, \times\, E[Y | X < c\, \land\, Y \ge c]$

and again of course

$E[Y | X < c\, \land\, Y \ge c] \ge c$ , so

$E[Y | Y \ge c] \ge$

$P[X \ge c | Y \ge c]\, \times\, E[Y | X \ge c\, \land\, Y \ge c] \;+\; P[ X < c | Y \ge c]\, \times\, c$

Putting these together we have

$E[Y | X \ge c] \le E[Y | Y \ge c]$

and, since the marginal distributions are equal, we also have

$E[X | X \ge c] = E[Y | Y \ge c]$ , so

$E[Y | X \ge c] \le E[X | X \ge c]$

which concludes the proof.

[edit] See also

Statistics portal

Internal validity

[edit] Notes

^ Howard Raiffa and Robert Schlaifer, Applied Statistical Decision Theory, Wiley-Interscience (2000) ISBN 978-0471383499
^ George Casella and Roger L. Berger, Statistical Inference, Duxbury Press (2001) ISBN 978-0534243128
^ Myra L. Samuels, Statiscal Reversion toward the Mean, The American Statistician, Vol 45, No 4 (November 1991), pp 344–346.
^ Galton, F. (1886). "Regression Toward Mediocrity in Hereditary Stature". Nature.
^ Hotelling, H. (1933). Review of The triumph of mediocrity in business by Secrist, H., Journal of the American Statistical Association, 28, 433–435.
^ Daniel Kahneman's autobiography
^ The Times, 16 December 2005 Speed camera benefits overrated
^ For an illustration see Nate Silver, "Randomness: Catch the Fever!", Baseball Prospectus, May 14, 2003.

[edit] References

J.M. Bland and D.G. Altman (June 1994). "Statistic Notes: Regression towards the mean". British Medical Journal 308: 1499. PMID 8019287. http://bmj.bmjjournals.com/cgi/content/full/308/6942/1499. Article, including a diagram of Galton's original data.

Francis Galton (1886). "Regression Towards Mediocrity in Hereditary Stature". Journal of the Anthropological Institute 15: 246–263. http://galton.org/essays/1880-1889/galton-1886-jaigi-regression-stature.pdf.

Stephen M. Stigler (1999). Statistics on the Table. Harvard University Press. See Chapter 9.

[edit] External links

Regression to the mean: what it is and how to deal with it, International Journal of Epidemiology.
Regression Toward the Mean and the Study of Change, Psychological Bulletin

A non-mathematical explanation of regression toward the mean.
A simulation of regression toward the mean.
Amanda Wachsmuth, Leland Wilkinson, Gerard E. Dallal. Galton's Bend: An Undiscovered Nonlinearity in Galton's Family Stature Regression Data and a Likely Explanation Based on Pearson and Lee's Stature Data (A modern look at Galton's analysis.)

Massachusetts standardized test scores, interpreted by a statistician as an example of regression: see discussion in sci.stat.edu and its continuation.

[Raiffa-0] Howard Raiffa and Robert Schlaifer, Applied Statistical Decision Theory, Wiley-Interscience (2000) ISBN 978-0471383499

[Casella-1] George Casella and Roger L. Berger, Statistical Inference, Duxbury Press (2001) ISBN 978-0534243128

[2] Myra L. Samuels, Statiscal Reversion toward the Mean, The American Statistician, Vol 45, No 4 (November 1991), pp 344–346.

[galton1886-3] Galton, F. (1886). "Regression Toward Mediocrity in Hereditary Stature". Nature.

[4] Hotelling, H. (1933). Review of The triumph of mediocrity in business by Secrist, H., Journal of the American Statistical Association, 28, 433–435.

[5] Daniel Kahneman's autobiography

[6] The Times, 16 December 2005 Speed camera benefits overrated

[7] For an illustration see Nate Silver, "Randomness: Catch the Fever!", Baseball Prospectus, May 14, 2003.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

Regression toward the mean

From Wikipedia, the free encyclopedia

Contents

[edit] History

[edit] Importance

[edit] Misunderstandings

[edit] Regression fallacies

[edit] A mathematical note

[edit] Proof

[edit] See also

[edit] Notes

[edit] References

[edit] External links

Views

Personal tools

Navigation

Search

Interaction

Toolbox

Languages