Linked Questions

0
votes
0answers
36 views

How is the L2 regularization derived? [duplicate]

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$\operatorname{argmax}\sum \log(P(x_i\mid\Theta ))$$ and ...
51
votes
7answers
7k views

Why is the regularization term *added* to the cost function (instead of multiplied etc.)?

Whenever regularization is used, it is often added onto the cost function such as in the following cost function. $$ J(\theta)=\frac 1 2(y-\theta X^T)(y-\theta X^T)^T+\alpha\|\theta\|_2^2 $$ This ...
22
votes
2answers
12k views

Why is Laplace prior producing sparse solutions?

I was looking through the literature on regularization, and often see paragraphs that links L2 regulatization with Gaussian prior, and L1 with Laplace centered on zero. I know how these priors look ...
5
votes
2answers
5k views

Why use mean of posterior distribution instead of probability?

I'm reading the Think Bayes (pdf link) by Allen B. Downey, and on this example I don't understand well the purpose of Mean in the chapter 3.2 The locomotive problem....
4
votes
3answers
2k views

How can I prove that the median is a nonlinear function?

My question flows out of the top answer to this question, from which I learned that a "linear function" is any function $f$ with properties of additivity and homogeneity of degree 1: $$ f(x + y) = f(...
2
votes
2answers
4k views

How does the L2 regularization penalize the high-value weights

I am reading about regularization in machine learning model. I want to understand how mathematically the L2 term penalizes the high-value weights to avoid overfitting? Any explanation?
9
votes
1answer
3k views

Using the median for calculating Variance

I have a 1-D random variable which is extremely skewed. In order to normalize this distribution, I want to use the median rather than the mean. my question is this: can I calculate the variance of the ...
3
votes
2answers
2k views

What are distribution assumptions in Ridge and Lasso regression models?

What are the assumptions for the distribution of the features for regression models like Lasso regression or Ridge regression? Why is it better to have features with Gaussian distributions?
3
votes
3answers
529 views

Question about conventions for L1 and L2 regularization

We can regularize a linear model with L1 or L2 regularization. But we usually write L2 with a square: $\|x\|_2^2$ and L1 with $\|x\|_1$. It seems a little bit strange and inconsistent for me, ...
4
votes
2answers
2k views

Practical applications of the Laplace and Cauchy distributions

I want to know if there are any examples of real-life applications of the Laplace and Cauchy density functions. How do they differ in their applications? This related post, however, does not answer ...
4
votes
2answers
1k views

What are the drawbacks of using least squares loss for regression?

It seems to be like the most popular loss function for regression, for everything from OLS (it's in the name!) to sophisticated regularized regressions. Why is it so popular and what are the ...
2
votes
1answer
1k views

L1 and L2 penalty vs L1 and L2 norms

I understand the usages of L1 and L2 norms however I am unsure of usage of L1 and L2 penalty when building models. From what I understand: L1: Laplace Prior L2: Gaussian Prior are two of the ...
2
votes
2answers
2k views

What does it mean an histogram vector normalization with L1/L2 norms?

I was reading these slides about Bag of Features (BoF). At slide 23 you can read: each image is represented by a vector, typically 1000-4000 dimension, normalization with L1/L2 norm What does ...
3
votes
1answer
481 views

Huber loss prior in Bayesian context

Gaussian prior in Bayesian setting is equivalent to minimizing squared error, while Laplace prior minimizes the absolute error and leads to lasso regression. What (if any) prior distribution can be ...
1
vote
1answer
697 views

How do distribution functions (e.g. Gaussian, Bernoulli, Poisson, etc.) relate to deep learning?

I know that neural nets use activation functions, but where do distribution functions play into deep neural networks? For example, the h2o.deeplearning() function ...

15 30 50 per page