On Data, Causality, and Artificial Intelligence

A valuable insight on the proof-of-concept to production gap in computer vision that underlines again the importance of context:

“It turns out,” Ng said, “that when we collect data from Stanford Hospital, then we train and test on data from the same hospital, indeed, we can publish papers showing [the algorithms] are comparable to human radiologists in spotting certain conditions.”

But, he said, “It turns out [that when] you take that same model, that same AI system, to an older hospital down the street, with an older machine, and the technician uses a slightly different imaging protocol, that data drifts to cause the performance of AI system to degrade significantly. In contrast, any human radiologist can walk down the street to the older hospital and do just fine.”

Source

An obvious but often neglected fact is the overemphasized value of accuracy as a performance metric. In a two-class problem where 99% of the cases are of 0 (Not a spam email), achieving an accuracy of 99% is as easy as classifying all emails as safe. Sensitivity, specificity, and other metrics exist for a reason.

The story of Waymo, Google’s self-driving car, resembles the value of solving the remaining 1% of the problem where conventional machine learning gets stuck due to the limitations of training data. If 1% of the error turns into a make or break point, one needs to get creative. On a long tail that extends to infinity, walking faster or running does not probably help as much as a leap of imagination.

I must note that it’s not fair to expect an autonomous car to be “error-free” given we do not expect human drivers to perform error-free at the driver license exams and road tests. The two will just make different errors.

To me, this is interesting not because of the lack of transparency in methodology but the potential reason for the rankings to be wrong.

I want to believe that this is a mistake not fraud, but really? Applying the weights before normalizing the scores? And the Bloomberg Businessweek spokesperson says “the magazine’s methodology was vetted by multiple data scientists.”

I have created a quick scenario as a reminder to my former (and current) students (posted in the comments as LinkedIn doesn’t allow here). In the example, the scores are standardized across the five items (which are randomly generated and assigned weights). In the Businessweek rankings, standardization is supposed to be across institutions so that the weights proportionately affect each institution’s score on the corresponding item. Nevertheless, the source of the error is the same. If the weights are applied before normalizing the data, the scores are adjusted by the weights disproportionately. Ranking changes accordingly.

“Zillow Quits Home-Flipping Business, Cites Inability to Forecast Prices,” WSJ reports.* I try to avoid passing along news stories but it’s not everyday I receive a predictive analytics story as breaking news.

I wonder whether the reason is really “an inability to forecast the prices” or “relying too much on an ability to forecast the prices” for a “$20 billion a year” venture as it was debuted.

Zillow announced plans for this data-driven venture in 2018 by citing consumers who “expect magic to happen with a simple push of a button.” In a statement yesterday, Zillow seems to have realized magic is not happening: “But it would be naïve to predict that unpredictable events won’t happen in the future.”

Maybe it is never a good idea to develop a whole business model that grossly underestimates the changes in error (both reducible and irreducible) due to potential bifurcations in market forces.

Source

Alex Webb of Bloomberg offers a linguistic explanation for why technology ceased to be meaningful:

“English lacked an equivalent to the French technique and German Technik. The English word “technique” hadn’t caught up with the innovations of the Industrial Revolution, and it still applied solely to the way in which an artist or artisan performed a skill.”

He contrasts technique as in “artistic technique” in English with technique as in “Lufthansa Technik” in German and argues that technology emerged in the early 20th century for the lack of a better alternative.

Whether the reason is linguistic, sheer overhype, or semantic satiation, we may be better off dropping the “tech company” reference at this point unless it is elaborated further. For the companies that are more tech than your average tech, a good alternative may be “deep tech.”

Why old? Well, it would be unfair to compare #Apple with today’s #Microsoft, the owner of #GitHub, a sponsor of Open Source Initiative and proponent of innovation through collaboration and co-creation (!). The exclamation will have to stay for a while.

The fight between #Apple and #Hey (hey.com, a contender to #Gmail) is not a surprise but a reminder that Apple is increasingly in the business of value capture, not creation. The gist of the story is, Apple forces Hey to sell subscriptions on its iOS platform but Hey refuses because the cost of doing so is a 30% commission for every subscriber. You can find the details in Kara Swisher’s article: nyti.ms/3ebfyvL

Apple seems to be stuck with incremental one-sided ideas, another iPhone with a larger screen or “dark mode” on its iOS platform, and have forgotten the value of co-creation, which propelled the company at the first place. Apple should be encouraging not oppressing experiments like Hey. For that, it is time for Apple to analyze its data from a fresh perspective that is not short-sighted on quarterly revenue, and rethink its model to embrace diversity again. That is what a successful data centric company would do.

McDonald’s CEO is out, but his data-focused legacy is likely to stay. McDonald’s has seemingly invested in digital transformation, including the digitization of operational processes such as in-store ordering through kiosks and home delivery. On the face of it, a shift from ordering at a counter to ordering at a kiosk is straightforward and replaces fast-food workers with giant tablets. Obvious gains include efficiency from increased order processing capacity, and, as the ex-CEO argues, increased spending on kiosks*. There is more to that, though. Kiosks can collect richer customer data and potentially create more customer data. Using the kiosks, McDonald’s can learn which products and extras customers scroll through before making a selection, whether they add an extra patty only to decide later not to keep it in their final order, etc. McDonald’s can also add data products to these kiosks such as a recommender system to create and collect more behavioral data, and guide customers into spending more and better. The company is planning to digitize its drive-through ordering as well, by replacing orders put through fast-food workers with a voice recognition software and possibly conversational AI (WSJ coverage). In addition, McDonald’s plans to make its drive-through menus dynamic, changing based on the weather, traffic, time of day, and potentially personal data (e.g., a drive-through customer’s licence plate number can be a nice unique identifier).

Most organizations use data in one way or another. Some still collect the data using mechanical tally counters, but that, as well, is data. Simple counts of visits to different exhibits in a museum can provide the director and curator with valuable insights (or not, depending on how strong the thumb muscles of staff are). Descriptive analysis is probably the most common approach to data analysis across organizations. For example, year-over-year (YoY) comparisons, a favorite across industries, are essentially descriptive. A YoY comparison of revenue is not more informative than the data collected by tally counters in a museum: “More visitors entered Exhibit A this year compared to last year.” These descriptive findings are like figuring out today is hotter than yesterday. They can be useful, albeit to a limited extent, and can be misleading when the difference is falsely attributed to a factor, sometimes just because that is readily available even if it was not measured (e.g., the YoY revenue is down because the sales team is underperforming). Measuring relevant factors and including them in a statistical model is another step up on the ladder of data centricity, no matter how simple the model is.

Most of the finance industry compete on models and algorithms today. Even so, Renaissance Technologies, a pioneer of quantitative trading in hedge funds, manages $110 billion of assets with the help of a linear regression model:

“Nick Patterson, who spent a decade as a researcher at Renaissance, says, “One tool that Renaissance uses is linear regression, which a high school student could understand” (OK, a particularly smart high school student; it’s about finding the relationship between two variables). He adds: “It’s simple, but effective if you know how to avoid mistakes just waiting to be made.””
Source: Computer Models Won’t Beat the Stock Market Any Time Soon

Simplicity helps the interpretation, and is preferable if the desired outcome (e.g., high prediction accuracy) is still achieved. In most cases, however, value creation using data involves causal reasoning and inference, higher steps on the ladder. This is one of a series of posts on “data centricity,” a framework I have been developing. Comments and feedback are welcome in any form.

One advantage of living in the age of big data is a diminishing need to ask customers explicitly for feedback. A variety of methods for unobtrusive learning from customers have emerged thanks to digitalization (vs. digitization). For example, customers now write reviews every day about products and services without being asked to do so. The behavior of customers can be captured by tracking their website visits. Sensors are now so cheap that a retailer can put sensors all over the floor in its stores and track the physical movements of customers. Compared to the tools and technologies used in an Amazon Go store, such a data collection initiative can be considered a small step today. Motion sensors for store shelves, neural network-powered cameras, and wireless beacons can easily be added as complements. From a managerial perspective, the phenomenon is more than a shift from a push mindset to a pull mindset. Leveraging it fully requires careful planning and execution. This is probably why “data centric” companies are capturing more value from unobtrusive methods while most retailers still struggle to learn from the reviews on their own product pages. Capturing most of the value also requires a systematic effort, rather than ad hoc attempts, ideally starting from product development into the full product life cycle. For example, when launched in 2004, Yelp required asking friends for recommendations. Users could not write reviews without being explicitly asked for. Yelp switched to the current model four months after the launch, based on the data on how early users behaved at the site. This is a short intro to a series of posts on “data centricity,” a concept I have been developing. Comments and feedback are welcome in any form.