For data scientists working on tabular data, which is still most of the data science world, Google’s TabFM is a nice solution using transformers (thankfully not LLMs).
TabFM promises to break the tedious cycle of endless hyperparameter tuning and feature engineering in ensemble methods like XGBoost. Instead of a manual training loop, TabFM treats tabular prediction as in-context learning: it ingests the entire dataset as a single prompt and predicts outcomes in a single forward pass. Clearly, the context window becomes the primary bottleneck as the dataset grows.
My day-to-day is mostly causal inference, with some projects using predictive models. Also interesting to me here is how the team solved the training problem: they pretrained the model on hundreds of millions of synthetic tables generated using Structural Causal Models.
This aligns neatly with our posts at Data Duets, specifically around the value of using transformers to generate datasets and our broader Augmented Data Science framework.