A natural experiment is a study in which individuals (or clusters of individuals) are exposed to the experimental and control conditions that are determined by nature or by other factors outside the control of the investigators. The process governing the exposures arguably resembles random assignment. Thus, natural experiments are observational studies and are not controlled in the traditional sense of a randomized experiment (an intervention study). Natural experiments are most useful when there has been a clearly defined exposure involving a well defined subpopulation (and the absence of exposure in a similar subpopulation) such that changes in outcomes may be plausibly attributed to the exposure. In this sense, the difference between a natural experiment and a non-experimental observational study is that the former includes a comparison of conditions that pave the way for causal inference, but the latter does not.
Natural experiments are employed as study designs when controlled experimentation is extremely difficult to implement or unethical, such as in several research areas addressed by epidemiology (like evaluating the health impact of varying degrees of exposure to ionizing radiation in people living near Hiroshima at the time of the atomic blast) and economics (like estimating the economic return on amount of schooling in US adults).
History
One of the best-known early natural experiments was the 1854 Broad Street cholera outbreak in London, England. On 31 August 1854, a major outbreak of cholera struck Soho. Over the next three days, 127 people near Broad Street died. By the end of the outbreak 616 people died. The physician John Snow identified the source of the outbreak as the nearest public water pump, using a map of deaths and illness that revealed a cluster of cases around the pump.
In this example, Snow discovered a strong association between the use of the water from the pump, and deaths and illnesses due to cholera. Snow found that the Southwark and Vauxhall Waterworks Company, which supplied water to districts with high attack rates, obtained the water from the Thames downstream from where raw sewage was discharged into the river. By contrast, districts that were supplied water by the Lambeth Waterworks Company, which obtained water upstream from the points of sewage discharge, had low attack rates. Given the near-haphazard patchwork development of the water supply in mid-nineteenth century London, Snow viewed the developments as "an experiment...on the grandest scale." Of course, the exposure to the polluted water was not under the control of any scientist. Therefore, this exposure has been recognized as being a natural experiment.
Recent examples
Family size
An aim of a study Angrist and Evans (1998) was to estimate the effect of family size on the labor market outcomes of the mother. For at least two reasons, the correlations between family size and various outcomes (e.g., earnings) do not inform us about how family size causally affects labor market outcomes. First, both labor market outcomes and family size may be affected by unobserved "third" variables (e.g., personal preferences). Second, labor market outcomes themselves may affect family size (called "reverse causality"). For example, a woman may defer having a child if she gets a raise at work. The authors observed that two-child families with either two boys or two girls are substantially more likely to have a third child than two-child families with one boy and one girl. The sex of the first two children, then, constitutes a kind of natural experiment: it is as if an experimenter had randomly assigned some families to have two children and others to have three. The authors were then able to credibly estimate the causal effect of having a third child on labor market outcomes. Angrist and Evans found that childbearing had a greater impact on poor and less educated women than on highly educated women although the earnings impact of having a third child tended to disappear by that child's 13th birthday. They also found that having a third child had little impact on husbands' earnings.
Game shows
Within economics, game shows are a frequently studied form of natural experiment. While game shows might seem to be artificial contexts, they can be considered natural experiments due to the fact that the context arises without interference of the scientist. Game shows have been used to study a wide range of different types of economic behavior, such as decision making under risk and cooperative behavior.
Smoking ban
In Helena, Montana a smoking ban was in effect in all public spaces, including bars and restaurants, during the six-month period from June 2002 to December 2002. Helena is geographically isolated and served by only one hospital. The investigators observed that the rate of heart attacks dropped by 40% while the smoking ban was in effect. Opponents of the law prevailed in getting the enforcement of the law suspended after six months, after which the rate of heart attacks went back up. This study was an example of a natural experiment, called a case-crossover experiment, where the exposure is removed for a time and then returned. The study also noted its own weaknesses which potentially suggest that the inability to control variables in natural experiments can impede investigators from drawing firm conclusions.'
Nuclear weapons testing
Nuclear weapons testing released large quantities of radioactive isotopes into the atmosphere, some of which could be incorporated into biological tissues. The release stopped after the Partial Nuclear Test Ban Treaty in 1963, which prohibited atmospheric nuclear tests. This resembled a large-scale pulse-chase experiment, but could not have been performed as a regular experiment in humans due to scientific ethics. Several types of observations were made possible (in people born before 1963), such as determination of the rate of replacement for cells in different human tissues.
Vietnam War draft
An important question in economics research is what determines earnings. Angrist (1990) evaluated the effects of military service on lifetime earnings. Using statistical methods developed in econometrics, Angrist capitalized on the approximate random assignment of the Vietnam War draft lottery, and used it as an instrumental variable associated with eligibility (or non-eligibility) for military service. Because many factors might predict whether someone serves in the military, the draft lottery frames a natural experiment whereby those drafted into the military can be compared against those not drafted because the two groups should not differ substantially prior to military service. Angrist found that the earnings of veterans were, on average, about 15 percent less than the earnings of non-veterans.
Industrial melanism
With the Industrial Revolution in the nineteenth century, many species of moth, including the well-studied peppered moth, responded to the atmospheric pollution of sulphur dioxide and soot around cities with industrial melanism, a dramatic increase in the frequency of dark forms over the formerly abundant pale, speckled forms. In the twentieth century, as regulation improved and pollution fell, providing the conditions for a large-scale natural experiment, the trend towards industrial melanism was reversed, and melanic forms quickly became scarce. The effect led the evolutionary biologists L. M. Cook and J. R. G. Turner to conclude that "natural selection is the only credible explanation for the overall decline".
See also
- Common garden experiment
References
- DiNardo, J. (2008). "Natural experiments and quasi-natural experiments". In Durlauf, Steven N.; Blume, Lawrence E (eds.). The New Palgrave Dictionary of Economics (Second ed.). Palgrave Macmillan. pp. 856–864. doi:10.1057/9780230226203.1162. ISBN 978-0-333-78676-5.
- Dunning, Thad (2012). Natural Experiments in the Social Sciences: A Design-Based Approach. Cambridge University Press.
- Friedman, G. D. (1980). Primer of Epidemiology (2nd ed.). New York: McGraw-Hill. ISBN 978-0-07-022434-6.
- Rosenzweig, M. R.; Wolpin, K. I. (2000). "Natural 'Natural Experiments' in Economics". Journal of Economic Literature. 38 (4): 827–874. doi:10.1257/jel.38.4.827.
- Snow, J. (1855). On the Mode of Communication of Cholera (2nd ed.). London: Churchill. Excerpted in MacMahon, B. & Pugh, T.F. (1970). Epidemiology. Boston: Little Brown.
- The 1854 cholera outbreak is the example of a natural experiment discussed often by David A. Freedman, e.g. in Statistical Models: Theory and Practice (Cambridge University Press) [1], chapter 1.3 (pp. 6–9).
- MacMahon, B., & Pugh, T.F. (1970). Epidemiology: Principles and Methods. Boston: Little, Brown.
- Snow's studies of the pattern of the disease were convincing enough to persuade the local council to disable the well pump by removing its handle. After the handle of the well-pump was replaced, the incidence of new cases dropped. In stopping the use of water from the well-pump, the authorities conducted, in effect, a second study, an uncontrolled intervention study, a study with an intervention group but no control group.
- Angrist, J.; Evans, W. (1998). "Children and Their Parents' Labor Supply: Evidence from Exogenous Variation in Family Size". American Economic Review. 88 (3): 450–477. JSTOR 116844.
- Post; Van den Assem; Baltussen; Thaler (2008). "Deal or No Deal? Decision Making under Risk in a Large-Payoff Game Show" (PDF). American Economic Review. 98 (1): 38–71. doi:10.1257/aer.98.1.38. JSTOR 29729963. S2CID 12816022. SSRN 636508.
- van den Assem; van Dolder; Thaler (2012). "Split or Steal? Cooperative Behavior When the Stakes Are Large". Management Science. 58 (1): 2. doi:10.1287/mnsc.1110.1413. hdl:1765/31292. S2CID 1371739. SSRN 1592456.
- Sargent, R. P.; Shepard, R. M.; Glantz, S. A. (2004). "Reduced incidence of admissions for myocardial infarction associated with public smoking ban: before and after study". British Medical Journal. 328 (7446): 977–980. doi:10.1136/bmj.38055.715683.55. PMC 404491. PMID 15066887.
- Angrist, Joshua D. (1990). "Lifetime Earnings and the Vietnam Draft Lottery: Evidence from Social Security Administrative Records". American Economic Review. 80 (3): 313–336. JSTOR 2006669.
- "Natural experiments in econometrics" – via www.youtube.com.
- Cook, L. M.; Turner, J. R. G. (2008). "Decline of melanism in two British moths: spatial, temporal and inter-specific variation". Heredity. 101 (6): 483–489. doi:10.1038/hdy.2008.105. PMID 18941471.
A natural experiment is a study in which individuals or clusters of individuals are exposed to the experimental and control conditions that are determined by nature or by other factors outside the control of the investigators The process governing the exposures arguably resembles random assignment Thus natural experiments are observational studies and are not controlled in the traditional sense of a randomized experiment an intervention study Natural experiments are most useful when there has been a clearly defined exposure involving a well defined subpopulation and the absence of exposure in a similar subpopulation such that changes in outcomes may be plausibly attributed to the exposure In this sense the difference between a natural experiment and a non experimental observational study is that the former includes a comparison of conditions that pave the way for causal inference but the latter does not Natural experiments are employed as study designs when controlled experimentation is extremely difficult to implement or unethical such as in several research areas addressed by epidemiology like evaluating the health impact of varying degrees of exposure to ionizing radiation in people living near Hiroshima at the time of the atomic blast and economics like estimating the economic return on amount of schooling in US adults HistoryJohn Snow s map showing the clustering of cholera cases in Soho during the London epidemic of 1854 One of the best known early natural experiments was the 1854 Broad Street cholera outbreak in London England On 31 August 1854 a major outbreak of cholera struck Soho Over the next three days 127 people near Broad Street died By the end of the outbreak 616 people died The physician John Snow identified the source of the outbreak as the nearest public water pump using a map of deaths and illness that revealed a cluster of cases around the pump In this example Snow discovered a strong association between the use of the water from the pump and deaths and illnesses due to cholera Snow found that the Southwark and Vauxhall Waterworks Company which supplied water to districts with high attack rates obtained the water from the Thames downstream from where raw sewage was discharged into the river By contrast districts that were supplied water by the Lambeth Waterworks Company which obtained water upstream from the points of sewage discharge had low attack rates Given the near haphazard patchwork development of the water supply in mid nineteenth century London Snow viewed the developments as an experiment on the grandest scale Of course the exposure to the polluted water was not under the control of any scientist Therefore this exposure has been recognized as being a natural experiment Recent examplesFamily size An aim of a study Angrist and Evans 1998 was to estimate the effect of family size on the labor market outcomes of the mother For at least two reasons the correlations between family size and various outcomes e g earnings do not inform us about how family size causally affects labor market outcomes First both labor market outcomes and family size may be affected by unobserved third variables e g personal preferences Second labor market outcomes themselves may affect family size called reverse causality For example a woman may defer having a child if she gets a raise at work The authors observed that two child families with either two boys or two girls are substantially more likely to have a third child than two child families with one boy and one girl The sex of the first two children then constitutes a kind of natural experiment it is as if an experimenter had randomly assigned some families to have two children and others to have three The authors were then able to credibly estimate the causal effect of having a third child on labor market outcomes Angrist and Evans found that childbearing had a greater impact on poor and less educated women than on highly educated women although the earnings impact of having a third child tended to disappear by that child s 13th birthday They also found that having a third child had little impact on husbands earnings Game shows Within economics game shows are a frequently studied form of natural experiment While game shows might seem to be artificial contexts they can be considered natural experiments due to the fact that the context arises without interference of the scientist Game shows have been used to study a wide range of different types of economic behavior such as decision making under risk and cooperative behavior Smoking ban In Helena Montana a smoking ban was in effect in all public spaces including bars and restaurants during the six month period from June 2002 to December 2002 Helena is geographically isolated and served by only one hospital The investigators observed that the rate of heart attacks dropped by 40 while the smoking ban was in effect Opponents of the law prevailed in getting the enforcement of the law suspended after six months after which the rate of heart attacks went back up This study was an example of a natural experiment called a case crossover experiment where the exposure is removed for a time and then returned The study also noted its own weaknesses which potentially suggest that the inability to control variables in natural experiments can impede investigators from drawing firm conclusions Nuclear weapons testing Nuclear weapons testing released large quantities of radioactive isotopes into the atmosphere some of which could be incorporated into biological tissues The release stopped after the Partial Nuclear Test Ban Treaty in 1963 which prohibited atmospheric nuclear tests This resembled a large scale pulse chase experiment but could not have been performed as a regular experiment in humans due to scientific ethics Several types of observations were made possible in people born before 1963 such as determination of the rate of replacement for cells in different human tissues Vietnam War draft An important question in economics research is what determines earnings Angrist 1990 evaluated the effects of military service on lifetime earnings Using statistical methods developed in econometrics Angrist capitalized on the approximate random assignment of the Vietnam War draft lottery and used it as an instrumental variable associated with eligibility or non eligibility for military service Because many factors might predict whether someone serves in the military the draft lottery frames a natural experiment whereby those drafted into the military can be compared against those not drafted because the two groups should not differ substantially prior to military service Angrist found that the earnings of veterans were on average about 15 percent less than the earnings of non veterans Industrial melanism With the Industrial Revolution in the nineteenth century many species of moth including the well studied peppered moth responded to the atmospheric pollution of sulphur dioxide and soot around cities with industrial melanism a dramatic increase in the frequency of dark forms over the formerly abundant pale speckled forms In the twentieth century as regulation improved and pollution fell providing the conditions for a large scale natural experiment the trend towards industrial melanism was reversed and melanic forms quickly became scarce The effect led the evolutionary biologists L M Cook and J R G Turner to conclude that natural selection is the only credible explanation for the overall decline See alsoCommon garden experimentReferencesDiNardo J 2008 Natural experiments and quasi natural experiments In Durlauf Steven N Blume Lawrence E eds The New Palgrave Dictionary of Economics Second ed Palgrave Macmillan pp 856 864 doi 10 1057 9780230226203 1162 ISBN 978 0 333 78676 5 Dunning Thad 2012 Natural Experiments in the Social Sciences A Design Based Approach Cambridge University Press Friedman G D 1980 Primer of Epidemiology 2nd ed New York McGraw Hill ISBN 978 0 07 022434 6 Rosenzweig M R Wolpin K I 2000 Natural Natural Experiments in Economics Journal of Economic Literature 38 4 827 874 doi 10 1257 jel 38 4 827 Snow J 1855 On the Mode of Communication of Cholera 2nd ed London Churchill Excerpted in MacMahon B amp Pugh T F 1970 Epidemiology Boston Little Brown The 1854 cholera outbreak is the example of a natural experiment discussed often by David A Freedman e g in Statistical Models Theory and Practice Cambridge University Press 1 chapter 1 3 pp 6 9 MacMahon B amp Pugh T F 1970 Epidemiology Principles and Methods Boston Little Brown Snow s studies of the pattern of the disease were convincing enough to persuade the local council to disable the well pump by removing its handle After the handle of the well pump was replaced the incidence of new cases dropped In stopping the use of water from the well pump the authorities conducted in effect a second study an uncontrolled intervention study a study with an intervention group but no control group Angrist J Evans W 1998 Children and Their Parents Labor Supply Evidence from Exogenous Variation in Family Size American Economic Review 88 3 450 477 JSTOR 116844 Post Van den Assem Baltussen Thaler 2008 Deal or No Deal Decision Making under Risk in a Large Payoff Game Show PDF American Economic Review 98 1 38 71 doi 10 1257 aer 98 1 38 JSTOR 29729963 S2CID 12816022 SSRN 636508 van den Assem van Dolder Thaler 2012 Split or Steal Cooperative Behavior When the Stakes Are Large Management Science 58 1 2 doi 10 1287 mnsc 1110 1413 hdl 1765 31292 S2CID 1371739 SSRN 1592456 Sargent R P Shepard R M Glantz S A 2004 Reduced incidence of admissions for myocardial infarction associated with public smoking ban before and after study British Medical Journal 328 7446 977 980 doi 10 1136 bmj 38055 715683 55 PMC 404491 PMID 15066887 Angrist Joshua D 1990 Lifetime Earnings and the Vietnam Draft Lottery Evidence from Social Security Administrative Records American Economic Review 80 3 313 336 JSTOR 2006669 Natural experiments in econometrics via www youtube com Cook L M Turner J R G 2008 Decline of melanism in two British moths spatial temporal and inter specific variation Heredity 101 6 483 489 doi 10 1038 hdy 2008 105 PMID 18941471
