For data scientists working on tabular data, which is still most of the data science world, Google’s TabFM is a nice solution using transformers (thankfully not LLMs).<\/p>\n
TabFM promises to break the tedious cycle of endless hyperparameter tuning and feature engineering in ensemble methods like XGBoost. Instead of a manual training loop, TabFM treats tabular prediction as in-context learning: it ingests the entire dataset as a single prompt and predicts outcomes in a single forward pass. Clearly, the context window becomes the primary bottleneck as the dataset grows.<\/p>\n
My day-to-day is mostly causal inference, with some projects using predictive models. Also interesting to me here is how the team solved the training problem: they pretrained the model on hundreds of millions of synthetic tables generated using Structural Causal Models.<\/p>\n
This aligns neatly with our posts at Data Duets<\/a>, specifically around the value of using transformers to generate datasets<\/a> and our broader Augmented Data Science framework<\/a>.<\/p>\n