Why we should be careful about cross-group comparison of causal effect

I was sitting in a friend’s thesis defense the other day, and a statistics professor commented: You need to define clearly what is from data and what is from the science!

Here, the phrase “the science”, as I figured out, should be translated to “the theory” in social science. The dichotomy between “data” and “science” is pretty much close to the one between “measurement” and “theory”. A statistician may feel stunned when your interpretation of the data suddenly hits “the science”. A social scientist may feel less so, because it is all too easy to confound “theory” with “measurement”.

So does there exist an unbreakable distinction between “measurement” and “theory”? This is so grand a thesis that I do not venture to build through one blog. But let us take one most common example of a “causal effect”.

Consider this experiment (Experiment 1):
I have two kettles of water and they are identical in volume and temperature. The temperature for both of them is 50C. I heat Kettle A for 5 minutes and do nothing to Kettle B. The temperature of Kettle A rises to 70C while the other one remains unchanged.

This “experiment” leads us to believe the following statement of “causal effect” (Statement 1).
“The heating caused the temperature to rise by 20C.”

Next, consider a slightly different experiment (Experiment 2):
I have two kettles of water and they are identical in volume and temperature. The temperature for both of them is 70C. I heat Kettle A for 5 minutes and do nothing to Kettle B. The temperature of Kettle A rises to 90C while the other one remains unchanged.

Shall we arrive at the same conclusion as Statement 1? Yes it sounds right. The two experiments give identical causal effect of heating.

Note that it is implicitly assumed in Statement 1 that the causal effect is the absolute difference in a linear reduced-form.

Now let us change our definition of “causal effect” to “percentage change”. Let us make the following statement (Statement 2):
“In Experiment 1, the heating causes the temperature of water to rise by 40%; In Experiment 2, the heating causes the temperature of water to rise by 28.6%. The heating has a larger effect in Experiment 1 than Experiment 2.”

It is not difficult to note that the difference is due to the difference in baseline temperature. Defining causal effect in proportional change instead of absolute change not only changes the quantitative conclusion, but also the qualitative conclusion (moving from “the same effect” to “a larger effect in Experiment 1”).

This is not the end of the story. Now let us set a ceiling to the temperature. Suppose that the boiling point of plain water is 100C. It cannot go above 100C. Consider the following statement (Statement 3):

“In Experiment 1, the heating causes the difference between the temperature of water and its boiling point to decrease by 40%; In Experiment 2, the heating causes the difference between the temperature of water and its boiling point to decrease by 66.7%. The heating has a larger effect in Experiment 2 than Experiment 1.”

See- the qualitative conclusion changed again (moving from “a larger effect in Experiment 1” to “a larger effect in Experiment 2”). This change is due to my change of the definition of causal effect to “the distance between current stage and the ceiling”.

The readers may have noticed that I have, on purpose, set the baseline temperature of my two experiments to be different. In laboratory science, an experienced researcher might not be so stupid to design two experiments that have different baselines and then feel stunned at the “confusing” results. However, empirical researchers in social science never had the opportunity to set everything equal at baseline. I make up this example because I think this is a good instance that illustrates why comparing “causal effect” across groups in social studies should be executed with extreme caution. Here comes my love for philosophy: Causal effect, in a broad sense, is a “measurement” rather than a “theory” – or, as David Hume has alluded to, “theories” are constrained by the way our brain “measures” the empirical world.

Back to the practical world, Economists crazily invented “elasticity” to capture the percentage change, but my example shows that elasticity is just one of many possible measures that could give opposite conclusions. Sociologists are concerned much less with “effect size”, and even when they do, they sometimes do not articulate their measurement. Psychologists speaks up for “framing effect” a lot in human reasoning, but who says that scientists are not humans and will not be biased?


Oh Hume is the real inspiration!

Advertisements
Post a comment or leave a trackback: Trackback URL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: