On Data Science, Causal Models, & AI

For years, Zara has been my go-to case to discuss data centricity in fashion retail. Zara is a staple example of how a focus on data and analytics combined with the right, complementary business processes can create wonders even in a market with high degrees of demand uncertainty due to the hedonic nature of consumption.

Shein seems to be emerging as a contender, moving further into data-driven (not only data-informed) fast fashion. Its operation is also called real-time fashion rather than fast fashion. Shein doesn’t own any physical stores (none at all) and ships all of its products directly from China.

Bloomberg reports that “Shein has developed proprietary technology that harvests customers’ search data from the app and shares it with suppliers, to help guide decisions about design, capacity and production. It generates recommendations for raw materials and where to buy them, and gives suppliers access to a deep database of designs for inspiration.”

Shein reduces the design to customer turnaround to 10 days, a record compared to already-fast Zara’s two- to three-week lead time. It’s not a niche operation either, given the reports of a $10 billion annual sales and a potential $30 billion valuation.

I’ve found the whole story interesting. It all sounds impressive but also dangerous. The article already mentions some of the “accidents” its algorithm-driven fashion caused along with sustainability concerns.

Data-driven decision making can lead to paralysis. Last week, the FDA and CDC committees couldn’t make a decision about the booster shots because (complete) data was not available. Well, making decisions in the absence of complete data is a process of imagination and deep thinking, one that puts hypothesis development at the center and humans continue to prevail over machines in the process.

To avoid such a paralysis, more focus can be put on developing and rethinking hypotheses and their likelihoods. In emergent problems, an in-depth discussion on hypotheses and likelihoods is probably more helpful than an obsession to access complete data. Otherwise, by defining complete data as a prerequisite, as it would be in data-driven decision making, we will continue to be paralyzed looking into the future.

If we turn to data-informed decision making, however, hypotheses would take more control (not gut feeling but properly developed hypotheses*). We could then make decisions to be improved as more data becomes available without being paralyzed in the present. Rather than seeking the truth, we would seek probable truths (as in Bayesian thinking).

While we may be able to remain strictly data-driven for some problems and decisions, we should be comfortable proceeding informed (not driven) by data for others.

* This post made me think of a book I enjoyed reading last Fall: Defense of the Scientific Hypothesis: From Reproducibility Crisis to Big Data

I avoid posting technical notes here. This is an exception because I have an agenda.

Log transformation is widely used in modeling data for several reasons: Making data “behave,” calculating elasticity etc.

When an outcome variable naturally has zeros, however, log transformation is tricky. Many data modelers (including seasoned researchers) instinctively add a positive constant to each value in the outcome variable. One popular idea is to add 1 to the variable and transform raw zeros to log-transformed zeros. Another idea is to add a very small constant, especially when the scale of the outcome variable is small.

Well, bad news is these are arbitrary choices and the resulting estimations may be biased. To me, if an analysis is correlational (as most are), a small bias may not be a big concern. If it is causal, and for example, an estimated elasticity will be used to take action (with an intention to change an outcome), that’s trouble waiting to happen. This is a problem of data centricity.

What is a solution (other than deserting to Poisson etc.)? A recent study by Christophe Bellégo and his coauthors offers a solution called iOLS (iterated OLS). To avoid bias, the iOLS algorithm adds an observation-specific value to the outcome variable. Voila! I haven’t tested it yet but I like the idea. Read their nicely written paper here: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3444996

My (not so hidden) agenda is regarding the implementation. The authors offer a Stata implementation (https://github.com/ldpape/iOLS). I would love to see it in R (or Python). Hence this is a call for action.

This latest article is not as coherent and a bit all over the place, but Amazon’s response is not helping either. Amazon says “Offering products inspired by the trends to which customers are responding is a common practice across the retail industry.” Amazon needs to nurture trust in its ecosystem but seems to be doing the opposite.

I don’t actually see any rampant issues except for access to product search data. Amazon is the dominant leader of the product search market (above Google and others). As a sign of good faith in building trust, Amazon could make (aggregated, anonymous) search data available and offer “Amazon Trends” like Google Trends. Needless to say, third-party sellers may be offered a more in-depth access.

To guess is to admit not knowing in the first place.

The problem with Dilbert’s coworkers and with most managerial teams is resisting to admit they don’t know. Even horoscopes and guesswork should start with the acknowledgment of a knowledge gap. Without such an acknowledgment, the time and effort needed to formulate and solve a problem is not justified. To guess is then to pretend knowing.

Guesswork supersedes learning from data because there is nothing to learn when it is all known. Successful data centric companies need a culture that encourages not knowing as much as knowing.

Absent of imagination, data workers perform at best on par with intelligent agents, finding associations but failing in causality. Identifying causal links requires thinking in counterfactuals, which, in turn, requires imagining what could have been.

What is absent must be imagined while what is present remains obvious, even to an algorithm. Data centric companies should invest at least as much in the thinking skills and imaginative ability as in the coding skills of their data workers for value creation.