In instruments we trust

A central question in economics is almost always, “how does X affect Y?”. We look not merely for relationships between variables (as X increases, Y falls), but seek causal explanations (the fall in Y is a result of the increase in X). Unfortunately for this kind of econometric analysis, the world is complex. We seldom can control the settings we observe; many variables can be changing simultaneously. Proving causality in economics typically requires techniques that can isolate the effect of the specific relationship of interest.

An example to illustrate why this might be so: it is common knowledge that, on average, those who study at university earn higher incomes than those who left school at the first possible opportunity. So, education causes higher wages for those with education? Perhaps. But it is also possible — likely even — that if all universities closed tomorrow, and everyone left school at the same age, that (again, on average) those who would have gone to university might still have better career prospects and earn a higher income than those who would never have studied further if given the chance. That is, there is some other factor at play beyond education itself, which influences 1) the likelihood of someone pursuing education and 2) later-life employment outcomes.

We don’t necessarily know what this external factor is — for example, it could be a product of differences in family background and upbringing, or it might be that each individual is born with his/her own innate level of ability. It is enough to know that there is some unobserved variable which confuses the relationship between education and employment. If we want to know the direct contribution of education on employment outcomes, we must be able to strip out the effect of any external factors.

Helpfully, the econometrician’s toolkit is not barren. But one must know the key properties of each tool: what are the strengths and weaknesses of each tool? Under what conditions does a given tool work best? How do different tools stack up in tackling a specific task? Is some combination of tools required to get the job done?

This post focuses on a specific tool: the instrumental variable. Broadly speaking, an instrumental variable generates some effect on the explanatory variable (in the example above, education) which can then be observed in the explanatory variable’s effect on the outcome (for example, wage). For this to work, the following conditions must be satisfied:

1. There must be a measurable effect of the instrument on the explanatory variable. If the instrument makes no real difference to the explanatory variable, then it’s not going to make a difference to the outcome either.
2. There must be no direct relationship between the instrument and the outcome. The only way the instrument affects the outcome is through its effect on the explanatory variable (Z changes X, X changes Y).
3. There must be no external factor to which both the instrument and the outcome are related. That is, it must not be the case that some unobserved factor jointly influences both the instrument and the outcome — otherwise we are back where we started.

The third restriction here is the most challenging: indeed, it is virtually impossible to verify that no external factor exists that might influence the results. One can certainly try to exclude all the imaginable possibilities, but even this does not do anything to address the possible external factors that one has not imagined. It is fundamentally a question of faith: do you believe in the validity of the instrument?

Distance to Wittenberg

It is perhaps fitting therefore, that one of my favourite examples of an instrumental variable being used relates to the effect of faith on economic outcomes. Specifically, measuring the long-term effect of the Protestant Reformation across the counties of Prussia. The Reformation began in the town of Wittenberg in 1517, when a German theologian and cleric — Martin Luther — posted his Ninety-Five Theses on the role of indulgences in the Catholic church. His arguments, and the events that followed, triggered a schism — the foundation of Protestantism, breaking away from the Catholic church.

A significant element of Luther’s view was that people should be able to read and understand the bible themselves. The dominant practice of the time was for the gospel to be communicated in Latin; Luther produced a version of the bible translated into German. But for this to make a substantive difference, it required that people could read: not necessarily a widespread skill among commoners of the era. The solution to this was universal access to schooling, where children would learn to read the word of God.

Becker and Woessmann (2009) hypothesise that this literacy boost in areas that converted to Protestantism delivered longer-term economic benefits that could be observed centuries later. However, there is — to use the jargon — an endogeneity problem here. If there is a relationship between Protestantism and economic development, it need not be the former causing the latter: rather, areas that had greater economic potential may have been more inclined to convert to Protestantism. Or that some unobserved external factor motivated both conversion and longer-term economic growth.

To overcome this, Becker and Woessmann apply an instrumental variable approach. Specifically, they use distance to Wittenberg as an instrument. All else being equal, the closer a town was to Wittenberg (the epicentre of the Reformation), the more likely it was that the town’s inhabitants converted to Protestantism. At the same time, Wittenberg was not itself a major economic centre — it is a matter of happenstance that the Reformation started there; beyond Luther’s work, Wittenberg played no great influential role that would have affected the economic development of other places. That is, distance to Wittenberg should have no effect on economic outcomes except through its contribution to Protestantism.

The analysis unfolds in three stages:

1. How much does distance to Wittenberg affect conversion to Protestantism in a given county?
2. How much does the county’s Protestant share, attributable to distance to Wittenberg (identified in stage 1), affect the literacy rate?
3. How much does the county’s literacy rate, attributable to the county’s Protestant share, which in turn is attributable to distance to Wittenberg (identified in stage 2), affect long-term economic performance? (In the results below, this economic measure is per capita income tax measured in 1877–78.)

Becker and Woessmann include additional controls for various population attributes — though these are not relevant to understanding how the instrument works. The first-stage results in the table show that for every 100-kilometre increase in the distance to Wittenberg, the share of Protestants falls 9.7 percentage points. That is, the further away we move from Wittenberg, the lower the share of Protestants. The second-stage results show that a one percentage point increase in the share of Protestants induces a 0.19 percentage point increase in the literacy rate. (This can also be interpreted as a difference in the literacy rate of 19 percentage when comparing a fully Protestant county to a fully Catholic county.) The third-stage results show that a one percentage point increase in the literacy rate results in higher per capita (nationally set) income tax receipts in 1877–78 — implying higher incomes.

3-stage least squares results

	1st stage: Share Protestants	2nd stage: Share literates	3rd stage: Per capita income tax
Distance to Wittenberg (km)	−0.097 (0.011) ∗∗∗
% Protestants		0.190 (0.028) ∗∗∗
% literate			3.242 (1.169) ∗∗∗
Observations	426	426	426
R-squared	0.442	0.699	0.374

A test of faith

To expand on the strategy here: the point of the second-stage question is to use the instrument (distance to Wittenberg) to give a controlled bump to the share of Protestantism, and measure the effect that this bump has on the literacy rate. (The third-stage question in effect repeats the exercise, using the result from stage 2 as an additional instrument against the economic outcome. For brevity, I focus only on the second-stage question here — but the same considerations apply in stage 3.) Assuming the instrument is valid, this bump effect reveals how Protestantism affects literacy — we use A to cause a change in B, and then look at how the specific effect of that change in B induces a change in C. Put another way, because Becker and Woessmann engineer a situation where they control the circumstances in which the share of Protestantism changes, they circumvent the endogeneity problem that would otherwise compromise measurement of the effect of Protestantism on literacy.

For this to be true, one must be satisfied that the instrument is valid. With reference to the three conditions noted above, the first of these — the relationship between the instrument and the explanatory variable — is clearly established by the first-stage regression. That is, there is a strong and statistically significant relationship between distance to Wittenberg and the share of Protestantism. The second condition — that there is no direct relationship between the instrument and the outcome — is the subject of discussion and analysis in the article. Becker and Woessmann test the relationship between a range of different economic development indicators and distance to Wittenberg. It is sufficient to say here, they do not find any relationship.

The third condition — the leap of faith — is harder to validate. Could there be some other factor that contributed both to the likelihood of conversion to Protestantism and influenced longer-term economic outcomes? For example, Dittmar (2011) conducts a similar analysis to Becker and Woessmann, but instead explores the contribution of the printing press — a technology which emerged in the 15^th century, originating in the German city of Mainz. (In that case, the instrument is distance to Mainz rather than Wittenberg.) The spread of this new technology might well have had consequences for the Reformation. Indeed, the printing press was instrumental in facilitating the distribution of religious material, including Luther’s German-translated bible. Rubin (2015) explores this possible effect: did being an early adopter of the printing press increase the likelihood of subsequently converting to Protestantism? His answer is “yes”, suggesting that one should question conclusions of any definitive relationship between the Reformation (or for that matter, the printing press) and economic outcomes.

So, Becker and Woessmann got it wrong? It is possible that their results overstate the effect of Protestantism, whether due to interaction with the printing press or some other factor. The authors themselves acknowledge that some other factor — for example, an innate “work ethic” — might well have induced both Protestantism and higher literacy. It would be fair to say that the Reformation didn’t happen in a vacuum: underlying social and cultural factors could well have enabled the spread of Luther’s teachings while also influencing longer-term economic outcomes. But the argument that the Reformation’s emphasis on literacy and schooling had longer-term benefits for Protestant communities is, at a minimum, intuitively appealing.

The key takeaway I intend for this post is less to do with Becker and Woessmann’s results and more to do with the challenges of using instruments. The econometric theory of instrumental variables is solid — if you can find a valid instrument, it’s a compelling way to overcome endogeneity problems. But that “if” is a big one. In reading any study that applies an instrumental variable approach, my starting point is one of scepticism. One cannot know if an instrument is valid. One can only be satisfied to the extent that one might believe it to be valid. Believers can be powerful advocates for their cause. But it takes just one potential weakness for sceptics to sow doubts among the faithful.