On AI, Data Science, & Causal Models

The business problem is to put a lifeguard station on a beach to save some lives (i.e., find the best location for the lifeguard station). This is not really a predictive modeling problem. But that’s the hammer our data scientists have and they have access to fancy libraries. There is also some historical data: swimmers rescued and drowned at other beaches. It all checks out. Resistance to 𝘱𝘪𝘱 𝘪𝘯𝘴𝘵𝘢𝘭𝘭 𝘱𝘳𝘰𝘱𝘩𝘦𝘵 is futile.

Transforming the problem into an objective function could have signaled that this is an optimization problem (a prescriptive modeling problem), but that step was skipped. In the picture shown, we may need a solution:

– minimizes distance => 𝗦𝗼𝗹𝘃𝗲𝗱 𝘂𝘀𝗶𝗻𝗴 𝗽𝗶𝗽 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗳𝗮𝗻𝗰𝘆_𝗹𝗶𝗯𝗿𝗮𝗿𝘆
while also…
– minimizing time => 𝗧𝗵𝗲 𝗱𝗼𝗺𝗮𝗶𝗻 𝗲𝘅𝗽𝗲𝗿𝘁 𝗲𝗻𝘁𝗲𝗿𝘀 𝘁𝗵𝗲 𝗿𝗼𝗼𝗺
– minimizing swimming => 𝗧𝗵𝗲 𝗹𝗮𝗯𝗼𝗿 𝘂𝗻𝗶𝗼𝗻 𝗶𝗻𝘁𝗲𝗿𝘃𝗲𝗻𝗲𝘀
– minimizing time to ice cream => 𝗧𝗵𝗲 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝘃𝗲 𝗹𝗲𝗮𝗱𝗲𝗿𝘀𝗵𝗶𝗽 𝘀𝘁𝗲𝗽𝘀 𝗶𝗻
– [not shown] minimizing walking on sand => 𝗧𝗵𝗲 𝗗𝗲𝗽𝗮𝗿𝘁𝗺𝗲𝗻𝘁 𝗼𝗳 𝗟𝗮𝗯𝗼𝗿 𝗿𝗲𝗾𝘂𝗶𝗿𝗲𝗺𝗲𝗻𝘁
and hopefully not…
– maximizing time => 𝗔 𝗷𝘂𝗻𝗶𝗼𝗿 𝗱𝗮𝘁𝗮 𝘀𝗰𝗶𝗲𝗻𝘁𝗶𝘀𝘁 𝘀𝗼𝗹𝘃𝗲𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺

So, the ideal solution requires more thinking about the problem. For example, maximizing the number of lives saved may actually require constraints on how to minimize time so that lifeguards don’t risk their lives during the rescue.

The law of the instrument works a little too well in predictive modeling (and more generally in machine learning). Objective functions are often lost in translation when they should be an explicit step in the modeling process. Best practice tends to favor performance metrics, even though achieving the highest performance on the wrong function is clearly useless (and sometimes detrimental).

More focus on objective functions and less obsession with “better performance” may be what we need. This would underline the importance of problem formulation and domain knowledge, and undermine the 𝘱𝘪𝘱 𝘪𝘯𝘴𝘵𝘢𝘭𝘭 𝘱𝘳𝘰𝘱𝘩𝘦𝘵 solution.

A combination of Warren Powell‘s writing and the accompanying xkcd comic inspired this post (courtesy of xkcd.com).

I have had a copy of Scott Cunningham‘s “Mixtape” since it came out. I’ve skimmed through it before, but last night, while putting together a few slides, I read an entire chapter and loved it. It has just enough detail to keep the reader from going to the cited work. It is also candid and fun. The print version is nicely sized and designed so you can blend in and look cool while others around you are reading the latest fiction. The book has already made its impact and this is probably a late call but I had to share.

Source

We seem to have a growing barrier to discussing AI: the use of AI as an umbrella term. My former students will say “Here we go again,” but if something means everything, it means nothing. If we take the time to define what we mean when we refer to AI, it will probably help the conversation.

Attached is a figure I’ve been using in my classes since 2017 to make this point (sorry, not the cat picture but the following figure of a timeline from AI to ML to Deep Learning). We might be better off referring to specific models and algorithms (or at least a group of models, such as LLMs, instead of AI).

Over the weekend, I attended a series of discussions on “AI” at the Academy of Management‘s annual conference. I had the opportunity to hear the perspectives of great scholars from a variety of backgrounds. Once again, I was puzzled as to what was meant by “AI” in most of the discussions.

Source

In Data Duets, Duygu Dagli and I offered our take on Ashesh Rambachan and Jonathan Roth‘s recently published but long overdue paper now titled “A more credible approach to parallel trends.”

Problem:
We want to test the causal effect of a promotion on sales, let’s say a coupon. The coupon was sent to newer customers. Did the coupon increase the sales? Or would the new customers have bought more anyway?

Solution:
We’ll never know the answer to the last question but we can answer the first question after making some assumptions. More on this in the post.

This is less elaborate than our earlier posts on synthetic controls and Lord’s paradox. We will probably keep it this way so that we can post more often.

Source

The fact that 73% of consumers trust content created by generative AI models is intriguing. And it’s not just people playing around with a chatbot for trivial conversations:*

– 67% believe they could benefit from receiving medical advice from a generative AI model
– 66% would seek advice from a generative AI model on relationships (work, friendships, romantic relationships) or life/career plans
– 64% are open to buying new products or services recommended by a generative AI model
– 53% trust generative AI-assisted financial planning

To put this into perspective, only 62% of people trust their doctor the most for medical advice.**

* 2023 survey by Capgemini of 10,000 consumers
** 2023 survey by OnePoll for Bayer of over 2000 adults

Source

H/T to Travis Gerke, I’ve just discovered the wonderful work of Benjamin Elbers. tidylog provides feedback for dplyr and tidyr operations in R. This is another simple and powerful idea that basically uses wrapper functions for the dplyr and tidyr functions with feedback added. This will help greatly with both teaching and “doing.”

Source

Pandas AI is an interesting and somewhat natural direction for embedding large language models into data science/analytics. This is less of a black box than automated exploratory data analysis tools, but still makes things easier.

We will likely see more ideas like Gabriele Venturi‘s here. For any serious project, though, we’ll still need skilled humans who can understand how the algorithm is responding to queries, and can check and confirm that it’s responding as expected.

Source

In this new post, Duygu Dagli and I took a quick look at the analysis of experimental data. I really enjoyed writing this piece because Lord’s revelation is one of my favorites (pardon the pun).

Lord’s paradox is related to the better known Simpson’s paradox and it highlights the importance of constructing the right conceptual model before moving on to modeling the data. In the post, I speculated about one potential conceptual model and discussed its implications for modeling the data at hand.

Frankly, the example in the post had much to unpack. I just picked up on the part that relates to causal models and Lord’s paradox. I also ended up touching on an interesting discussion around the use of diff-in-diff vs. lagged regression models.

After running an experiment, how do you estimate the average treatment effect (ATE)? Which model do you choose to use? In this post, we use five different models with different assumptions to answer the same question. We find five different ATEs… Which one is the correct average treatment effect in this experiment? How do we decide?

In this post, Gorkem Turgut Ozer and I explore these questions (and more) by discussing the differences across models and potential implications. We ended up covering an interesting paradox I enjoyed learning about!

To me, a main takeaway is, business value from data is maximized when the right conceptual model meets the right method. For this to happen more often, data science and pricing leaders need the technical skills to be able to ask the right questions. They also need to build a trusting relationship with their teams to delegate and learn from them.

Source

Across the board, ChatGPT is passing exams by answering a mix of short-answer, essay, and multiple choice questions:

– U.S. medical licensing exam (says the attached study)
– Wharton School MBA exam on Operations Management
– University of Minnesota Law School exams in Constitutional Law, Employee Benefits, Taxation, and Torts

If ChatGPT is able to pass these exams, it is not because ChatGPT is revolutionary (while it is surely impressive) but because they are just bad exams. These exams must be lacking enough components that require some form of creative thinking and use of imagination.

Source

What is demonstrated here is a successful translation from human language to code. OpenAI has another project for this purpose: Codex. Microsoft’s GitHub Copilot serves as a specialized version (both are descendents of GPT-3). DeepMind’s AlphaCode and the open source PolyCoder also target English to code translation.

What is missing (and provided by Marco) is the articulation of a solution that stems from a conceptual model, which in turn, is informed by causal links. For example: diversification reduces asset-specific risk.

Unless ChatGPT reasonably limits the weights of each individual stock based only on the objective at the beginning (minimize SD) without being explicitly instructed, we’d better curb our enthusiasm here.

Source

Just tried out ChatGPT, the new large language model trained by OpenAI, and I was blown away by its capabilities! It can generate human-like text responses to any prompt, making it a powerful tool for conversation simulation, language translation, and more.

I also had a chance to play around with the code, and it’s surprisingly simple to use. Here’s a quick example of how to generate a response from ChatGPT using the Python API:
…

—

Not bad but a bit too excited (blown away, really?). Also the shameless self-promotion using my voice without any disclosure. We’re not off to a good, trusting relationship.

Pierre-Simon Laplace in Théorie Analytique Des Probabilités

We need more decision / data scientists to ask “What is the probability of sun rising tomorrow?” yet we don’t seem to put a strong emphasis on probability theory for several reasons.

Probability is not thoroughly covered (ideally as a standalone course) in most data science / business analytics programs. In predictive analytics, common packages/libraries for ensemble methods focus on classifications, almost hiding the probability calculations (which are distorted in some anyway). Most frequentist reporting are limited to point estimates and errors, again hiding the underlying probabilistic assumptions. Et Cetera.

In search of a short reading to share with my students, I’ve come across a recent book (updated in May, 2022) by Dirk P. Kroese, Zdravko Botev, Thomas Taimre, Radislav (Slava) Vaisman Ph.D. The book is open access and the appendices serve as a nice refresher. Appendix C is a little primer on probability for example.

Source

In this latest collaboration with Duygu Dagli, we shed some light on it and soon other topics. We plan to bridge conceptual and applied takes on statistical modeling and machine learning methods and the business problems they may help solve. We’ll likely touch on causal inference, predictive analytics, and how firms can organize to be more data centric. Check it out and stay tuned for the upcoming pieces on experiments and the analysis of experimental data.

Source

Jeremy Siegel almost loses it here for good reason. The Federal Reserve Board called it transitory inflation when prices were skyrocketing last Fall. When the same Fed (except a couple members) now argues that inflation is not yet slowing at a reasonable pace despite the price contractions recent data shows, questions arise. Fed gives the impression that it is now overcorrecting what it overcorrected earlier by loosening the monetary policy too much and for too long.

In case after case, our data modeling and inference practices are tested against lags in data. Lack of a high predictive power, we resort to overcorrections. Overcorrecting is doing more than enough (vs. not doing enough) and sounds better than coming short. But then, the pendulum swings back a little harder.

What do we learn from such swings? Well, one rather obvious takeaway is to put more emphasis on correctly understanding and modeling lags in time series. Another one is to be content with coming short occasionally, especially when the cost of overcorrecting is much higher than the cost of coming short.

Source

The commercial refrigerator/shelf looking device in the picture is an ikejime machine. Ikejime is considered the fastest and most humane method of killing fish. The method also leads to the best taste of fish because fish are killed instantly before their bodies go into distress and produce lactic acid and ammonia into their muscles. Ikejime involves the insertion of a spike quickly and directly into the hindbrain, causing immediate brain death.

That is, if fishermen know where the hindbrain exactly is for each species, and can insert the spike quickly and precisely within minutes of catching a fish, and have time to do so repeatedly. Well, that’s what robots are for.

Shinkei Systems’ machine is a combination of hardware and an edge detection algorithm, the engine behind object recognition in convolutional neural networks. Challenges abound. The machine operates on a fishing boat that tilts around even at zero speed. Apparently, “even in the same species, even with the same contour, the brain can be in a different location” as well.

Working with fishermen in Maine, New Hampshire and Cape Cod, Shinkei Systems seems to have been accomplishing the task on fresh-caught fish at a rate of one every 10-15 seconds. Moving forward, accuracy should increase and time to complete the task should decrease, leading to further opportunities.

Source

“We wanted to make sure people with proximity to the data, their country, the language they speak, had a hand in choosing what language came into the model’s training,” says Jernite.

#BLOOM, the first truly open-science, open-access, and multilingual (46 languages) large language model with 176B parameters (slightly larger than GPT-3) will soon be released as a complete pretrained model. Behind the project is #BigScience, a wide-scale collaboration of over 1,000 researchers.

The project is quite impressive overall, both for the extent of collaboration and outcome. It’s also an engineering delight to watch. The model has been trained using 384 A100 GPUs (with 80 GB of memory each) since March 11, 2022.

BigScience provides updates on training every day (having hit its initial target earlier than planned, the model is currently being trained for “a few more days”). See the links in the comments to follow the updates and download the model. The full model will be released on #HuggingFace (also a partner of the project).

This is a significant step forward for at least two reasons: the way the training data was collected and the core values behind the initiative. BigScience seems to have prioritized data quality by hand crafting the training data. In a world of models that favor kitchen sink approaches (because they can!), this is a progress. More obviously, BLOOM paves the way for a true democratization by removing the strings that have been attached to the use of such models by #OpenScience, #Google, and #Facebook (apply for API access, accredited researcher only etc.).

Source

While attending the Symposium on Data Science & Statistics to present our study in Improving Algorithms for Big Data session two weeks ago, I learned about useful new methods (and met the great people behind them).

One of my favorites is Sparsity Ranked Lasso (SRL) by Ryan Peterson. The paper mainly focuses on lasso but the idea is also extended to other regularization approaches such as elastic net.

Takeaway: Use SRL over ordinary lasso especially if your model has interaction terms and polynomials. On average, SRL’s predictive performance is better than lasso in the 112 datasets from the Penn Machine Learning Benchmark database. Ryan goes on to show also that SRL overperforms a Random Forest (RF) in a case study both in accuracy and efficiency. Even if SRL performs on par with a RF, why not use SRL as it is both interpretable and explainable!

The part I loved about SRL is the simple yet important challenge, which the authors call “covariate equipoise”: the prior belief that all covariates are equally likely to enter into a model. Basically, a model’s simplicity is usually defined by its parsimony: the number of parameters. This is no matter whether a parameter is an interaction (or a polynomial form) of the other terms in the model. This is problematic for obvious reasons and SRL solves it by treating covariate groups differently based on their type.

And yes, there is a package for that: sparseR. Link to the R package and nicely written paper are in the comments.

R package – Paper

A seemingly persistent effect of the pandemic on #Uber is a 50% decrease in the mobility’s share of revenue (a decrease from a 80% share to less than a 40% share of rides in total revenue). Based on revenue, Uber is now more a delivery company than a mobility company.

This is data centricity extrapolated: a shift from carrying people to carrying objects while solving pretty much the same data-driven optimization problem. The article is from last year but the effect persists as of Q1 2022: Only 37% of the revenue is from carrying people.

Source

Now that object detection is almost a solved problem, work on the next frontier, text-to-image generation, began to thrive. #Google Research’s most recent work on generative models, #Imagen, uses text embeddings from a large language model called #T5 (similar to #GPT3 and #OPT175B) to encode text for image synthesis.

Interestingly, the study finds that increasing the size of the language model improves performance more than increasing the size of the image diffusion model. Imagen achieves exceptional similarity between real and synthetic images (measured by the distance metric FID, Imagen achieves a score of 7.27 on the COCO dataset). Human raters confirm the performance of the model.

The paper is nicely written with a much-needed ethics discussion at the end, and full of colorful images. Apparently, Imagen does not perform as well when generating images that portray humans.

Synthetic data generation and image restoration are two common use cases of #GANs. I will post a link to one such study on medical images in the comments. #Arts and #crafts is obvious. I can also think of use cases for #fashion and potentially #personalization of products in #retail. What are some other business use cases?

Source

Interesting read and perspective on modeling the learning in hippocampus and potentially applying the model structure to the design and development of algorithms. Clone-structured cognitive graph (CSCG) uses Markov chains and dynamic Markov compression. So CSCGs form a probabilistic sequence model.

Source