While attending the Symposium on Data Science & Statistics to present our study in Improving Algorithms for Big Data session two weeks ago, I learned about useful new methods (and met the great people behind them).
One of my favorites is Sparsity Ranked Lasso (SRL) by Ryan Peterson. The paper mainly focuses on lasso but the idea is also extended to other regularization approaches such as elastic net.
Takeaway: Use SRL over ordinary lasso especially if your model has interaction terms and polynomials. On average, SRL’s predictive performance is better than lasso in the 112 datasets from the Penn Machine Learning Benchmark database. Ryan goes on to show also that SRL overperforms a Random Forest (RF) in a case study both in accuracy and efficiency. Even if SRL performs on par with a RF, why not use SRL as it is both interpretable and explainable!
The part I loved about SRL is the simple yet important challenge, which the authors call “covariate equipoise”: the prior belief that all covariates are equally likely to enter into a model. Basically, a model’s simplicity is usually defined by its parsimony: the number of parameters. This is no matter whether a parameter is an interaction (or a polynomial form) of the other terms in the model. This is problematic for obvious reasons and SRL solves it by treating covariate groups differently based on their type.
And yes, there is a package for that: sparseR. Link to the R package and nicely written paper are in the comments.