Category: Uncategorized

I avoid posting technical notes here. This is an exception because I have an agenda.

Log transformation is widely used in modeling data for several reasons: Making data “behave,” calculating elasticity etc.

When an outcome variable naturally has zeros, however, log transformation is tricky. Many data modelers (including seasoned researchers) instinctively add a positive constant to each value in the outcome variable. One popular idea is to add 1 to the variable and transform raw zeros to log-transformed zeros. Another idea is to add a very small constant, especially when the scale of the outcome variable is small.

Well, bad news is these are arbitrary choices and the resulting estimations may be biased. To me, if an analysis is correlational (as most are), a small bias may not be a big concern. If it is causal, and for example, an estimated elasticity will be used to take action (with an intention to change an outcome), that’s trouble waiting to happen. This is a problem of data centricity.

What is a solution (other than deserting to Poisson etc.)? A recent study by Christophe Bellégo and his coauthors offers a solution called iOLS (iterated OLS). To avoid bias, the iOLS algorithm adds an observation-specific value to the outcome variable. Voila! I haven’t tested it yet but I like the idea. Read their nicely written paper here: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3444996

My (not so hidden) agenda is regarding the implementation. The authors offer a Stata implementation (https://github.com/ldpape/iOLS). I would love to see it in R (or Python). Hence this is a call for action.

This latest article is not as coherent and a bit all over the place, but Amazon’s response is not helping either. Amazon says “Offering products inspired by the trends to which customers are responding is a common practice across the retail industry.” Amazon needs to nurture trust in its ecosystem but seems to be doing the opposite.

I don’t actually see any rampant issues except for access to product search data. Amazon is the dominant leader of the product search market (above Google and others). As a sign of good faith in building trust, Amazon could make (aggregated, anonymous) search data available and offer “Amazon Trends” like Google Trends. Needless to say, third-party sellers may be offered a more in-depth access.

To guess is to admit not knowing in the first place.

The problem with Dilbert’s coworkers and with most managerial teams is resisting to admit they don’t know. Even horoscopes and guesswork should start with the acknowledgment of a knowledge gap. Without such an acknowledgment, the time and effort needed to formulate and solve a problem is not justified. To guess is then to pretend knowing.

Guesswork supersedes learning from data because there is nothing to learn when it is all known. Successful data centric companies need a culture that encourages not knowing as much as knowing.

Absent of imagination, data workers perform at best on par with intelligent agents, finding associations but failing in causality. Identifying causal links requires thinking in counterfactuals, which, in turn, requires imagining what could have been.

What is absent must be imagined while what is present remains obvious, even to an algorithm. Data centric companies should invest at least as much in the thinking skills and imaginative ability as in the coding skills of their data workers for value creation.

Data centricity requires a shift in mindset, no matter whether it is policy making or business strategy making. Without this shift, decision makers may swim in a pool of charts and tables but can’t see.

What struck me in this opinion piece is the depiction of how multisided (e.g., two-sided) platforms evolve, in an animated GIF by Ryan Kuo. Platform owners feel the need to say “Trust us” at some point, long after contractual relationships are established.

Platform owners gain power and lock in participants (e.g., sellers, buyers, app developers, users) by accumulating network effects and creating switching costs*. More power leads to governance decisions that are increasingly one-sided (e.g., decisions on application approval, product listings, content sharing, or commission/fees). Conflict of interest arises quickly. Trust deteriorates.

Lack of trust can make data centric companies vulnerable to disruption in the long term, even if network effects offer a protection in the short term. One sure way not to gain trust is having to say “Trust us.”

*Cross-side network effects: The more sellers on a platform, the more value for buyers. More buyers join and more sellers follow. As a seller builds a profile full of five star reviews, switching becomes costly. Lock-in can also arise from same-side network effects. In a social platform, value for a user increases with more users.

Traditionally, computers process data quite differently from how human brains do so. Computers are designed for precision while human brains rely on intuition. With artificial intelligence (#AI), or more specifically, deep learning and neural networks, one idea is to mimic the way human brains work. Does this mean that the hardware, or the body also needs to change? Are CPUs and GPUs not up to the task anymore?

Graphcore.ai claims so, and argues that CPUs and even GPUs are out, and IPUs are in. Graphcore’s #IPU stands for intelligence processing units and is prone to imprecision by design. It is a high-performance computing unit that processes data very imprecisely.

Consider a task like going to a restaurant. A human brain wouldn’t calculate the GPS coordinates but use associations; e.g., recall the restaurant’s name, its neighborhood, and neighboring shops. The difference resembles one between Boolean logic and fuzzy logic, and is true.

What is under the hood, or hardware, matters. One component of achieving data centricity is building an infrastructure that fits the objective, and successful data centric companies know they need to invest in it.

With an increasing availability of data and off-the-shelf analysis tools, interventions are thriving.

Interventions rarely create value. Rarity is expected simply because the probability of noise is often disproportionately higher. However, larger amounts of data exacerbate the problem of finding value in interventions while none exists. E.g., a frequentist test using a 0.01 p-value threshold would justify an intervention if the probability of an effect occurring by chance is less than 1%. This probability gets smaller with more data, not because the intervention gains value*. 1% should be a moving target, but it is often treated as a fixed one. It should be adjusted also for other reasons, such as running multiple tests.

More importantly, it should be adjusted for unintended consequences. While quantifying the consequences is difficult, we can incentivize analytics teams for finding out what not to do. Action is visible but inaction is not. Successful data centric companies should not mistake thoughtful inaction for idleness. On the contrary, they should encourage and reward it.

*Assuming the actual effect is not zero. Valid for most (if not all) problems outside natural sciences.