36 views

### How is the L2 regularization derived? [duplicate]

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$\operatorname{argmax}\sum \log(P(x_i\mid\Theta ))$$ and ...
7k views

### Why is the regularization term *added* to the cost function (instead of multiplied etc.)?

Whenever regularization is used, it is often added onto the cost function such as in the following cost function. $$J(\theta)=\frac 1 2(y-\theta X^T)(y-\theta X^T)^T+\alpha\|\theta\|_2^2$$ This ...
12k views

### Why is Laplace prior producing sparse solutions?

I was looking through the literature on regularization, and often see paragraphs that links L2 regulatization with Gaussian prior, and L1 with Laplace centered on zero. I know how these priors look ...
5k views

### Why use mean of posterior distribution instead of probability?

I'm reading the Think Bayes (pdf link) by Allen B. Downey, and on this example I don't understand well the purpose of Mean in the chapter 3.2 The locomotive problem....
2k views

### How can I prove that the median is a nonlinear function?

My question flows out of the top answer to this question, from which I learned that a "linear function" is any function $f$ with properties of additivity and homogeneity of degree 1:  f(x + y) = f(...
4k views

### How does the L2 regularization penalize the high-value weights

I am reading about regularization in machine learning model. I want to understand how mathematically the L2 term penalizes the high-value weights to avoid overfitting? Any explanation?
3k views

### Using the median for calculating Variance

I have a 1-D random variable which is extremely skewed. In order to normalize this distribution, I want to use the median rather than the mean. my question is this: can I calculate the variance of the ...
2k views

### What are distribution assumptions in Ridge and Lasso regression models?

What are the assumptions for the distribution of the features for regression models like Lasso regression or Ridge regression? Why is it better to have features with Gaussian distributions?
529 views

### Question about conventions for L1 and L2 regularization

We can regularize a linear model with L1 or L2 regularization. But we usually write L2 with a square: $\|x\|_2^2$ and L1 with $\|x\|_1$. It seems a little bit strange and inconsistent for me, ...
2k views

### Practical applications of the Laplace and Cauchy distributions

I want to know if there are any examples of real-life applications of the Laplace and Cauchy density functions. How do they differ in their applications? This related post, however, does not answer ...
1k views

### What are the drawbacks of using least squares loss for regression?

It seems to be like the most popular loss function for regression, for everything from OLS (it's in the name!) to sophisticated regularized regressions. Why is it so popular and what are the ...
1k views

### L1 and L2 penalty vs L1 and L2 norms

I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models. From what I understand: L1: Laplace Prior L2: Gaussian Prior are two of the ...
2k views

### What does it mean an histogram vector normalization with L1/L2 norms?

I was reading these slides about Bag of Features (BoF). At slide 23 you can read: each image is represented by a vector, typically 1000-4000 dimension, normalization with L1/L2 norm What does ...