New answers tagged regularization
1
The best way to understand the role of the hyperparameter $C$ is to think in term of robustness of your classifier. In short, $C$ is the hyperparameter that you will have to tune to find the optimal trade off between two competing objectives, which are 1) to make sure that your classifier performs well on your training data and 2) that it will also ...
1
In fact, there are some works related to using regularization on decision tree splitting. For example, in [1], the author proposes a cost function to penalize redundant features, and thus get a more succinct feature set. The experimental results reveal that it can effectively get a more concise feature subset without loss of accuracy.
[1]. Deng H, Runger G. ...
1
You should use group lasso. When you have a categorical variable, with say, $k$ levels, usually represented in a regression model with $k-1$ dummys, that group of $k-1$ dummys together is the one variable, and should be included or not as a unit. Group lasso does that.
This have been explained many times on this site, for example Can I ignore coefficients ...
1
What is regularization, really?
Perhaps you are conflating L1/L2 regularization (aka. Lasso/ridge regression, Tikhonov regularization...), the most ubiquitous type, as the only type of regularization 🤔
Regularization is actually anything that prevents overfitting, that you can do to a learning algorithm [Wikipedia]. Dropout, batch normalization, early ...
1
The usual practice in penalized methods like LASSO, ridge or principal-component regression is to start by standardizing all predictors to zero mean and unit standard deviation. Otherwise the issue that you raise would lead to a distance measured in miles being handled differently than the same distance measured in millimeters. That's the default, for ...
0
Define all your restrictions, such as Lipshitz continuity, degree of the polynomial etc.
Define a loss function for smoothness, e.g. in terms of second order derivative over the interval $\int_a^b f''(x)^2$ or otherwise $\max_{x \in [a,b]} f''(x)^2$.
Estimate the parameters of the polynomial by minimising this loss function.
5
If $\lambda$ is a nonzero eigenvalue of an invertible matrix $A$, then $\lambda^{-1}$ is an eigenvalue of $A^{-1}$. To show this, note that if $Av=\lambda v$ then $v = A^{-1}(\lambda v) = \lambda A^{-1} v$ and thus $\lambda^{-1} v = A^{-1} v$.
0
A minimum in out-of-sample mean squared error (OOS MSE) is found only when the unregularized model is prone to overfitting.
The simple model for the happiness response variable using only the 6 features provided is on the high-bias side of the bias-variance tradeoff. It's not possible to overfit the data using a linear model here.
Expanding the feature space ...
2
We don't generally consider $\lambda$ as a parameter in the model you want to estimate. It doesn't have an interpretation outside of the model, in terms of your actual data. Instead, we consider $\lambda$ as a tuning parameter or a hyperparameter. This terminology means that $\lambda$ affects how you estimate $\beta$, but you aren't interested in $\lambda$ ...
1
I'll offer a partial answer since there has been little activity since you asked this.
A keyword here is hierarchical modeling. It is possible to specify a hierarchical model for your biological markers to shrink their coefficients toward one another while letting the other predictors vary independently. A Bayesian hierarchical model would be something like
$...
Top 50 recent answers are included
