In this new post, Duygu Dagli and I took a quick look at the analysis of experimental data. I really enjoyed writing this piece because Lord’s revelation is one of my favorites (pardon the pun).
Lord’s paradox is related to the better known Simpson’s paradox and it highlights the importance of constructing the right conceptual model before moving on to modeling the data. In the post, I speculated about one potential conceptual model and discussed its implications for modeling the data at hand.
Frankly, the example in the post had much to unpack. I just picked up on the part that relates to causal models and Lord’s paradox. I also ended up touching on an interesting discussion around the use of diff-in-diff vs. lagged regression models.
After running an experiment, how do you estimate the average treatment effect (ATE)? Which model do you choose to use? In this post, we use five different models with different assumptions to answer the same question. We find five different ATEs… Which one is the correct average treatment effect in this experiment? How do we decide?
In this post, Gorkem Turgut Ozer and I explore these questions (and more) by discussing the differences across models and potential implications. We ended up covering an interesting paradox I enjoyed learning about!
To me, a main takeaway is, business value from data is maximized when the right conceptual model meets the right method. For this to happen more often, data science and pricing leaders need the technical skills to be able to ask the right questions. They also need to build a trusting relationship with their teams to delegate and learn from them.