Regularization is a set of techniques that helps learning models to converge and it allows to get a good model generalization capacity. This kind of result is known as “right fit“:

We can see that the function follows the points thoroughly, but without getting a trend too rigorous.

When we train a model, there are two types of fitting problems:

  • Underfitting: trained model isn’t able to represent the input features:
  • Overfitting: our model has learned to recognize the training examples, in a very good way, but it isn’t able to categorize new incoming samples:

Even though in different ways, underfitting and overfitting are not able to give us a generalization of the model. They show two kinds of problems:

  • Bias: a measure of approximation error over all the samples in the training set. That is, to put it simply, how far our model is from samples;
  • Variance: variance of approximation model over all the samples;

Underfitting presents high bias and low variance, whereas overfitting low bias and high variance. As a consequence, we need to deal mainly with these two kinds of wrong fitting. We can state that the problems related to generalization come out from the choice of parameters. In the high bias case, we usually don’t have enough significant parameters, or maybe, the training set is too small. High variance, on the contrary, means that we have too many parameters that, maybe, are not all required. Generally, there is the need to penalize or give more importance to some parameters and less to others. Unfortunately, we don’t know which of them to work on. To address the problem, we use techniques called regularizers. They are applied to the function that gives us the estimation related to the known target values, which is the Cost function:

cost + regularization = L(x, y, \theta) + \lambda * R(\theta)

L2 Regularization

Also known as Ridge Regression andTikhonov Regularization, it uses the half of the squared Euclidean norm of all parameters:

L(x, y, \theta) + \frac{\lambda}{2} * \sum \theta^2

It produces a non-sparse output and penalizes all parameters. It means that all parameters are retained, none of them goes to zero, but all values are small.

L1 Regularization

It is also called LASSO. It penalizes the sum of the absolute value of weights:

 L(x, y, \theta) + \lambda * \sum \left | \theta \right |

It produces a sparse output performing a feature selection. For example, if our model uses 1000 parameters, maybe not all parameters are useful and L1 zeroes a part of them. In the end, we could get only 100 values. It doesn’t limit weights growth.


It is a different way to regularize a network [1]Improving neural networks by preventing co-adaptation of feature detectors. It doesn’t apply to loss function, but it is based on the idea that some neurons can be saturated during the training process and they shouldn’t be activated to prevent that their contribution can harm the training process. “Saturation” is expressed as co-adaptation, which means a neuron is only useful as a function of another specific neuron. This problem is, clearly, a limitation to model generalization. The question is “What neurons should be deactivated? And how can they be chosen?” The answer is that we don’t know which neuron can cause the problem, so we can arbitrarily choose a set of them. This means that we can make a random choice.

N.B. dropout should be applied only in training phase, whereas while testing it has to be removed, because we need the full generalization power of our model.

Usage in tensorflow

L1/L2 regularization

In tensorflow, it is very simple to use regularization. Regardless of which kind of regularizer we want to use (L1 or L2), we define it during the variable creation process. I like to define a function that handles the variable creation for me:

“create_variable” has a weight_decay parameter that defines whether the variable should be affected by regularization or not. Why do we choose to avoid regularization on any variables? The function is used for weights and bias creation. Bias shouldn’t be influenced by regularization as reported in [2] The only parameters that should be influenced by it are the weights. Based on the fact that tensorflow builds a computation graph, we are able to apply the loss on the variable and store it into a collection, which is a key/value storage. In this way, regularization will be applied to the updated weight value each time it is needed. I have explained this concept because at first glance it may appear that we apply regularization to weights when they are created for the first time and only to the initial value, but that’s not the case!

The second element that needs to be modified to allow regularization to take place is the model cost computation. It is applied after the cross-entropy function, in this way we compute the “pure” loss for the model and then apply the regularization term:

It is a very simple way to regularize our model! We only need to get values in the “weight_decay” collection and sum them with the cross entropy mean value. The magic has happened!

In the end, we only need to write code to create a variable and choose a regularization type.

I used L2 as default, but we can pass whatever regularization type we want. Unfortunately, L1 isn’t a tensorflow function, so we have to create it by ourselves and use it instead of L2:

In theory, there is some optimization that can be done to write less code, but I think that this is the clearest way.


Dropout is applied to neurons and not to parameters, so in the case of a Multi-Layer Perceptron, we can use this kind of regularization on one or more layers. We define a hidden layer by a variable dropout, with a default value of 1:

In this way, we apply dropout only to specific layers, while to the others the probability that a neuron should be kept is 1. As previously mentioned, we have to apply it in training phase only, so we should be able to change the dropout value for a specific layer, this can be obtained by using a placeholder which will be later passed to the layer:

Now we are able to choose the value of dropout in the session run:

Full code is available on GoDeep GitHub repository.

Star Follow @vincenzosantopietro Watch



References   [ + ]

1. Improving neural networks by preventing co-adaptation of feature detectors