On Data Science, Causal Models, & AI

For a new project, I’ve been exploring options to develop a backend to query multiple large language models and just came across this great solution.

It’s an open source project called LiteLLM and it provides a unified interface to call 100+ LLMs using the same input and output format, including OpenAI, Anthropic, models on Hugging Face, Azure etc.

There is cost tracking and rate limits. To make things easier, there is even a user interface. What I found most useful is the ease of comparison and benchmarking between LLMs. Kudos to the developer team.

I can see so many business use cases for integrations like this: rapid prototyping and experimentation, performance benchmarking and optimization, cost control…

Source

Beyond the analogy of LLMs being a lossy compression of the Web, the point about the creative process is spot on in this article. The more we relegate the creative process to the tools of efficiency, the more we risk the output being mediocre.

Will letting a large language model handle the boilerplate allow writers to focus their attention on the really creative parts?

Obviously, no one can speak for all writers, but let me make the argument that starting with a blurry copy of unoriginal work isn’t a good way to create original work. If you’re a writer, you will write a lot of unoriginal work before you write something original. And the time and effort expended on that unoriginal work isn’t wasted; on the contrary, I would suggest that it is precisely what enables you to eventually create something original. The hours spent choosing the right word and rearranging sentences to better follow one another are what teach you how meaning is conveyed by prose.

Sometimes it’s only in the process of writing that you discover your original ideas.

Source

Bing Copilot may have a point here, but the language could have been less patronizing. All kidding aside, another LLM will surely take the job.

Source

Google seems to have just revealed its latest text-to-video diffusion model, Google Lumiere, just as the debate over fake images and videos heats up, with the following note:

𝘚𝘰𝘤𝘪𝘦𝘵𝘢𝘭 𝘐𝘮𝘱𝘢𝘤𝘵
𝘖𝘶𝘳 𝘱𝘳𝘪𝘮𝘢𝘳𝘺 𝘨𝘰𝘢𝘭 𝘪𝘯 𝘵𝘩𝘪𝘴 𝘸𝘰𝘳𝘬 𝘪𝘴 𝘵𝘰 𝘦𝘯𝘢𝘣𝘭𝘦 𝘯𝘰𝘷𝘪𝘤𝘦 𝘶𝘴𝘦𝘳𝘴 𝘵𝘰 𝘨𝘦𝘯𝘦𝘳𝘢𝘵𝘦 𝘷𝘪𝘴𝘶𝘢𝘭 𝘤𝘰𝘯𝘵𝘦𝘯𝘵 𝘪𝘯 𝘢𝘯 𝘤𝘳𝘦𝘢𝘵𝘪𝘷𝘦 𝘢𝘯𝘥 𝘧𝘭𝘦𝘹𝘪𝘣𝘭𝘦 𝘸𝘢𝘺. 𝘏𝘰𝘸𝘦𝘷𝘦𝘳, 𝘵𝘩𝘦𝘳𝘦 𝘪𝘴 𝘢 𝘳𝘪𝘴𝘬 𝘰𝘧 𝘮𝘪𝘴𝘶𝘴𝘦 𝘧𝘰𝘳 𝘤𝘳𝘦𝘢𝘵𝘪𝘯𝘨 𝘧𝘢𝘬𝘦 𝘰𝘳 𝘩𝘢𝘳𝘮𝘧𝘶𝘭 𝘤𝘰𝘯𝘵𝘦𝘯𝘵 𝘸𝘪𝘵𝘩 𝘰𝘶𝘳 𝘵𝘦𝘤𝘩𝘯𝘰𝘭𝘰𝘨𝘺, 𝘢𝘯𝘥 𝘸𝘦 𝘣𝘦𝘭𝘪𝘦𝘷𝘦 𝘵𝘩𝘢𝘵 𝘪𝘵 𝘪𝘴 𝘤𝘳𝘶𝘤𝘪𝘢𝘭 𝘵𝘰 𝘥𝘦𝘷𝘦𝘭𝘰𝘱 𝘢𝘯𝘥 𝘢𝘱𝘱𝘭𝘺 𝘵𝘰𝘰𝘭𝘴 𝘧𝘰𝘳 𝘥𝘦𝘵𝘦𝘤𝘵𝘪𝘯𝘨 𝘣𝘪𝘢𝘴𝘦𝘴 𝘢𝘯𝘥 𝘮𝘢𝘭𝘪𝘤𝘪𝘰𝘶𝘴 𝘶𝘴𝘦 𝘤𝘢𝘴𝘦𝘴 𝘪𝘯 𝘰𝘳𝘥𝘦𝘳 𝘵𝘰 𝘦𝘯𝘴𝘶𝘳𝘦 𝘢 𝘴𝘢𝘧𝘦 𝘢𝘯𝘥 𝘧𝘢𝘪𝘳 𝘶𝘴𝘦.

This is the only paragraph in the paper on safe and fair use. The model output certainly looks impressive, but, without a concrete discussion of ideas and guardrails for safe and fair use, this reads like nothing more than a checkbox to avoid bad publicity from the likely consequences.

Source

In a sample of 6.4 billion sentences in 90 languages from the Web, this study finds that 57.1% is low-quality machine translation. In addition, it is the low quality content produced in English (to generate ad revenue) that is translated en masse into other languages (again, to generate ad revenue).

The study discusses the negative implications for the training of large language models (garbage in, garbage out), but the increasingly poor quality of public web content is concerning nevertheless.

Source

In this Q&A about Walmart’s custom-trained, proprietary “My Assistant” language model, I saw an excerpt from another article in which Walmart’s Head of People Product uses Excel as an analogy for generative models.

“𝘈𝘤𝘤𝘰𝘳𝘥𝘪𝘯𝘨 𝘵𝘰 𝘗𝘦𝘵𝘦𝘳𝘴𝘰𝘯, 𝘢𝘯𝘺 𝘨𝘦𝘯𝘦𝘳𝘢𝘵𝘪𝘷𝘦 𝘈𝘐 𝘳𝘰𝘭𝘭𝘰𝘶𝘵 𝘪𝘴 𝘨𝘰𝘪𝘯𝘨 𝘵𝘰 𝘦𝘯𝘤𝘰𝘶𝘯𝘵𝘦𝘳 𝘢 𝘤𝘩𝘢𝘯𝘨𝘦 𝘤𝘶𝘳𝘷𝘦 𝘯𝘰𝘵 𝘶𝘯𝘭𝘪𝘬𝘦 𝘸𝘩𝘢𝘵 𝘔𝘪𝘤𝘳𝘰𝘴𝘰𝘧𝘵 𝘌𝘹𝘤𝘦𝘭 𝘦𝘹𝘱𝘦𝘳𝘪𝘦𝘯𝘤𝘦𝘥 𝘪𝘯 𝘵𝘩𝘦 1980𝘴 𝘣𝘦𝘧𝘰𝘳𝘦 𝘣𝘦𝘪𝘯𝘨 𝘢𝘤𝘤𝘦𝘱𝘵𝘦𝘥 𝘢𝘴 𝘤𝘰𝘳𝘱𝘰𝘳𝘢𝘵𝘦 𝘨𝘰𝘴𝘱𝘦𝘭. 𝘚𝘪𝘮𝘪𝘭𝘢𝘳 𝘵𝘰 𝘩𝘰𝘸 𝘦𝘢𝘳𝘭𝘺 𝘶𝘴𝘦𝘳𝘴 𝘰𝘧 𝘔𝘪𝘤𝘳𝘰𝘴𝘰𝘧𝘵 𝘌𝘹𝘤𝘦𝘭 𝘩𝘢𝘥 𝘵𝘰 𝘣𝘦 𝘵𝘳𝘢𝘪𝘯𝘦𝘥 𝘵𝘰 𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥 𝘩𝘰𝘸 𝘵𝘰 𝘩𝘢𝘳𝘯𝘦𝘴𝘴 𝘵𝘩𝘦 𝘱𝘰𝘸𝘦𝘳 𝘰𝘧 𝘢 𝘗𝘪𝘷𝘰𝘵𝘛𝘢𝘣𝘭𝘦 𝘢𝘯𝘥 𝘝𝘓𝘖𝘖𝘒𝘜𝘗 𝘧𝘰𝘳𝘮𝘶𝘭𝘢𝘴, 𝘨𝘦𝘯𝘦𝘳𝘢𝘵𝘪𝘷𝘦 𝘈𝘐 𝘶𝘴𝘦𝘳𝘴 𝘩𝘢𝘷𝘦 𝘵𝘰 𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥 𝘱𝘳𝘰𝘮𝘱𝘵𝘪𝘯𝘨 𝘢𝘯𝘥 𝘩𝘪𝘨𝘩-𝘪𝘮𝘱𝘢𝘤𝘵 𝘶𝘴𝘦 𝘤𝘢𝘴𝘦𝘴 𝘵𝘰 𝘵𝘳𝘶𝘭𝘺 𝘩𝘢𝘳𝘯𝘦𝘴𝘴 𝘪𝘵𝘴 𝘱𝘰𝘸𝘦𝘳.”

2024 will be the year that more companies adopt generative models as an aid to their employees. But it is interesting to use an analogy to deterministic functions like PivotTable and VLOOKUP to drive the adoption of a black box model with probabilistic outputs. Let’s see how that plays out for Walmart.

Source

The use of AI algorithms for drug discovery is one of the most promising areas for its societal value. Historically, most deep learning approaches in this area have used black box models, providing little insight into discoveries.

A recent study published in Nature uses explainable graph neural networks to address the urgent need for new antibiotics due to the ongoing antibiotic resistance crisis.

The study begins with the testing and labeling of 39,312 compounds,

– which become training data for four ensembles of graph neural networks,
– which make predictions for a total of 12,076,365 compounds in the test set (hits vs. non-hits based on antibiotic activity and cytotoxicity),
– of which 3,646 compounds are selected based on the probability that they will act as antibiotics without being toxic to humans,
– which are then reduced to 283 compounds by a series of empirical steps,
– and to 4 compounds by experimental testing,
– and of the 4, two “drug-like” compounds are tested in mice,

and one of the two is found to be effective against MRSA infections in this controlled experiment, thus closing the causal loop.

This is a great application of combining explainable predictive models with causal identification, and demonstrates that machine learning models used in high-stakes areas can be explainable without compromising performance.

Source

The dose-response work of Callaway, Goodman-Bacon, and Pedro Sant’Anna seems to be coming along nicely. If you haven’t had enough of the parallel trends assumption, get ready for the “strong” parallel trends assumption!

“𝘐𝘯 𝘵𝘩𝘪𝘴 𝘱𝘢𝘱𝘦𝘳, 𝘸𝘦 𝘥𝘪𝘴𝘤𝘶𝘴𝘴 𝘢𝘯 𝘢𝘭𝘵𝘦𝘳𝘯𝘢𝘵𝘪𝘷𝘦 𝘣𝘶𝘵 𝘵𝘺𝘱𝘪𝘤𝘢𝘭𝘭𝘺 𝘴𝘵𝘳𝘰𝘯𝘨𝘦𝘳 𝘢𝘴𝘴𝘶𝘮𝘱𝘵𝘪𝘰𝘯, 𝘸𝘩𝘪𝘤𝘩 𝘸𝘦 𝘤𝘢𝘭𝘭 𝘴𝘵𝘳𝘰𝘯𝘨 𝘱𝘢𝘳𝘢𝘭𝘭𝘦𝘭 𝘵𝘳𝘦𝘯𝘥𝘴. 𝘚𝘵𝘳𝘰𝘯𝘨 𝘱𝘢𝘳𝘢𝘭𝘭𝘦𝘭 𝘵𝘳𝘦𝘯𝘥𝘴 𝘰𝘧𝘵𝘦𝘯 𝘳𝘦𝘴𝘵𝘳𝘪𝘤𝘵𝘴 𝘵𝘳𝘦𝘢𝘵𝘮𝘦𝘯𝘵 𝘦𝘧𝘧𝘦𝘤𝘵 𝘩𝘦𝘵𝘦𝘳𝘰𝘨𝘦𝘯𝘦𝘪𝘵𝘺 𝘢𝘯𝘥 𝘫𝘶𝘴𝘵𝘪𝘧𝘪𝘦𝘴 𝘤𝘰𝘮𝘱𝘢𝘳𝘪𝘯𝘨 𝘥𝘰𝘴𝘦 𝘨𝘳𝘰𝘶𝘱𝘴. 𝘐𝘯𝘵𝘶𝘪𝘵𝘪𝘷𝘦𝘭𝘺, 𝘵𝘰 𝘣𝘦 𝘢 𝘨𝘰𝘰𝘥 𝘤𝘰𝘶𝘯𝘵𝘦𝘳𝘧𝘢𝘤𝘵𝘶𝘢𝘭, 𝘭𝘰𝘸𝘦𝘳-𝘥𝘰𝘴𝘦 𝘶𝘯𝘪𝘵𝘴 𝘮𝘶𝘴𝘵 𝘳𝘦𝘧𝘭𝘦𝘤𝘵 𝘩𝘰𝘸 𝘩𝘪𝘨𝘩𝘦𝘳-𝘥𝘰𝘴𝘦 𝘶𝘯𝘪𝘵𝘴’ 𝘰𝘶𝘵𝘤𝘰𝘮𝘦𝘴 𝘸𝘰𝘶𝘭𝘥 𝘩𝘢𝘷𝘦 𝘤𝘩𝘢𝘯𝘨𝘦𝘥 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 𝘵𝘳𝘦𝘢𝘵𝘮𝘦𝘯𝘵 𝘢𝘯𝘥 𝘢𝘵 𝘵𝘩𝘦 𝘭𝘰𝘸𝘦𝘳 𝘭𝘦𝘷𝘦𝘭 𝘰𝘧 𝘵𝘩𝘦 𝘵𝘳𝘦𝘢𝘵𝘮𝘦𝘯𝘵. 𝘞𝘦 𝘴𝘩𝘰𝘸 𝘵𝘩𝘢𝘵 𝘸𝘩𝘦𝘯 𝘰𝘯𝘦 𝘰𝘯𝘭𝘺 𝘪𝘮𝘱𝘰𝘴𝘦𝘴 𝘵𝘩𝘦 “𝘴𝘵𝘢𝘯𝘥𝘢𝘳𝘥” 𝘱𝘢𝘳𝘢𝘭𝘭𝘦𝘭 𝘵𝘳𝘦𝘯𝘥𝘴 𝘢𝘴𝘴𝘶𝘮𝘱𝘵𝘪𝘰𝘯, 𝘤𝘰𝘮𝘱𝘢𝘳𝘪𝘴𝘰𝘯𝘴 𝘢𝘤𝘳𝘰𝘴𝘴 𝘵𝘳𝘦𝘢𝘵𝘮𝘦𝘯𝘵 𝘥𝘰𝘴𝘢𝘨𝘦𝘴 𝘢𝘳𝘦 “𝘤𝘰𝘯𝘵𝘢𝘮𝘪𝘯𝘢𝘵𝘦𝘥” 𝘸𝘪𝘵𝘩 𝘴𝘦𝘭𝘦𝘤𝘵𝘪𝘰𝘯 𝘣𝘪𝘢𝘴 𝘳𝘦𝘭𝘢𝘵𝘦𝘥 𝘵𝘰 𝘵𝘳𝘦𝘢𝘵𝘮𝘦𝘯𝘵 𝘦𝘧𝘧𝘦𝘤𝘵 𝘩𝘦𝘵𝘦𝘳𝘰𝘨𝘦𝘯𝘦𝘪𝘵𝘺. 𝘛𝘩𝘶𝘴, 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 𝘢𝘥𝘥𝘪𝘵𝘪𝘰𝘯𝘢𝘭 𝘴𝘵𝘳𝘶𝘤𝘵𝘶𝘳𝘦, 𝘤𝘰𝘮𝘱𝘢𝘳𝘪𝘴𝘰𝘯 𝘢𝘤𝘳𝘰𝘴𝘴 𝘥𝘰𝘴𝘢𝘨𝘦𝘴 𝘮𝘢𝘺 𝘯𝘰𝘵 𝘪𝘥𝘦𝘯𝘵𝘪𝘧𝘺 𝘤𝘢𝘶𝘴𝘢𝘭 𝘦𝘧𝘧𝘦𝘤𝘵𝘴. 𝘛𝘩𝘦 𝘱𝘭𝘢𝘶𝘴𝘪𝘣𝘪𝘭𝘪𝘵𝘺 𝘰𝘧 𝘴𝘵𝘳𝘰𝘯𝘨 𝘱𝘢𝘳𝘢𝘭𝘭𝘦𝘭 𝘵𝘳𝘦𝘯𝘥𝘴 𝘥𝘦𝘱𝘦𝘯𝘥𝘴 𝘰𝘯 𝘵𝘩𝘦 𝘦𝘮𝘱𝘪𝘳𝘪𝘤𝘢𝘭 𝘤𝘰𝘯𝘵𝘦𝘹𝘵 𝘰𝘧 𝘵𝘩𝘦 𝘢𝘯𝘢𝘭𝘺𝘴𝘪𝘴, 𝘢𝘯𝘥 𝘸𝘦 𝘥𝘪𝘴𝘤𝘶𝘴𝘴 𝘴𝘰𝘮𝘦 𝘧𝘢𝘭𝘴𝘪𝘧𝘪𝘤𝘢𝘵𝘪𝘰𝘯 𝘴𝘵𝘳𝘢𝘵𝘦𝘨𝘪𝘦𝘴 𝘵𝘩𝘢𝘵 𝘤𝘢𝘯 𝘣𝘦 𝘶𝘴𝘦𝘥 𝘵𝘰 𝘢𝘴𝘴𝘦𝘴𝘴 𝘪𝘵.”

Source

2024 will be the year of lawsuits against generative AI companies. We’ve already had the GitHub Copilot case over assisted coding (https://lnkd.in/eH8Ap-eJ) and the Anthropic case over AI lyrics (https://lnkd.in/eY6UF9Cn). Now the Times joins the fray (https://lnkd.in/e8wHHMzx), and more are likely to follow.

So far, Gen AI companies have defended themselves by arguing fair use and transformative use – that their models create something substantially new and serve a different purpose than the original (thus doesn’t substitute the original, as in Google Books). But recent Supreme Court decisions such as Warhol v. Goldsmith made clear that transformative use claims face high bars.

What might come next?
– New business models for content licensing
– Restrictions on public access to some internal models
– Calls for updated copyright laws and content use regulations
– Technical solutions like attribution, data provenance, and content tagging
– What else?

In our latest short post at Data Duets, Duygu Dagli and I share a recent paper (and an R package) that offers a better solution.

How often do you log transform your data? How do you deal with zeros? Do you just add 1 or some other constant before taking the log?

In our latest short post, Gorkem Turgut (G.T.) Ozer and I share a recent paper (and an R package) that offers a better solution!

Source