Generalized Linear Models (GLMs) are one of the most widely used inferential modeling techniques. Their simplicity makes them easy to interpret, so when communicating causal inference to stakeholders they’re a very effective tool.
Elastic net regularization, a widely used regularization method, is a logical pairing with GLMs — it removes unimportant and highly correlated features, which can hurt both accuracy and inference. These two methods are a useful part of any data science toolkit.
Prior to March of 2021, the combination of GLMs and elastic net regularization was fairly complex. However, researchers at Stanford released a paper that leverages cyclic coordinate descent to allow efficient computation of a model’s coefficients for any link function, not just the simple ones. They also implemented this model in the famous R package called glmnet.
In this post we’ll discuss the fundamentals of GLMs, elastic net regularization, and understand how the two can work together. Let’s dive right in…
Technical TLDR
In R, the glmnet package now supports all link function families. If your link is canonical, pass a string to the family parameter. If not, pass a family function. Note that you can check the documentation for a list of the supported canonical functions.
On the backend, the package has two major changes…
- Substitutes the OLS solver for maximum likelihood. Note that this is only done for non-canonical link functions — canonical links are still solved using OLS.
- Use Iteratively Reweighted Least Squares (IRLS) to solve. IRLS (also called Fisher Scoring) is needed because the maximum likelihood is not guaranteed to be concave.
Ok, I think I got it. But how does this new method actually work?
Let’s slow down a bit and start with how GLM’s actually work.
What’s a Generalized Linear Model (GLM)?
GLM’s are a fairly simple concept that has been around since the 1970s. With the rise of big data, they have become invaluable tools in developing inferential models i.e. models that give us statistically valid conclusions about our data.
Why you might ask?
Well GLM’s are just linear regressions with an extra trick: a link function. The purpose of a link function is to transform our dependent variable so that the linear model can fit better.
Let’s take a look at a simple example.
Read more on towardsdatascience.com
Michael Berk – I’m a Data Scientist writing 52 posts that bring “academic” research to DS industry.
