4
$\begingroup$

My question might be a bit ambiguous, but I started to wonder what does the "effective parameters" mean in machine learning? I have heard few professors of machine learning in my university talk about effective parameters (the context was about k-nearest neighbors or Gaussian mixture models etc.).

Any clarification on what this might mean? Thank you for any help!

P.S. I have also seen this term in the book: The Elements of Statistical Learning 2nd edition, Trevor Hastie, page 15.

$\endgroup$
5
$\begingroup$

Some models (here: linear regression) have parameters $\beta$:

$$ \hat{y} = \sum_{i\in\{1..p\}} \beta_i x_i + \beta_0 $$

For the same number of input features, more complex models (here: basis expansion to a quadratic model) have more parameters:

$$ \hat{y} = \sum_{i,j\in\{1..p\} \atop i\le j} \beta_{ij} x_ix_j + \sum_{i\in\{1..p\}} \beta_{i} x_i + \beta_0 $$

In general, models with more parameters are more flexible (because there are more parameters to tune to fit the model to the data), but are more difficult to fit and thus more likely tend to overfit. Regularization helps overcome these problems by reducing the degrees of freedom for tuning, thus reducing the complexity a bit:

$$ \hat{y} = \sum_{i,j\in\{1..p\} \atop i\le j} \beta_{ij} x_ix_j + \sum_{i\in\{1..p\}} \beta_{i} x_i + \beta_0 \textrm{, with} \sum \beta^2 \textrm{ small} $$

We can express the 'model complexity' or 'degrees of freedom for tuning' in terms of 'effective parameters': For the model that you investigate, find the parameter-based model (without regularization) that has the same degree of freedom for tuning. The number of parameters for that model is the effective number of parameters for your model.

This generalization can also be extended further to models that are not based on parameters (e.g. k nearest neighbor). Then the degree of complexity is known as Vapnik–Chervonenkis (VC) dimension. Conveniently, for (non-regularized) linear regression, it is $p+1$, the actual number of parameters. Elements of Statistical Learning, section 7.9, gives more information about that (and they seem to use the terms 'VC', 'effective number of parameters' and 'model complexity' more or less interchangably).

$\endgroup$
  • $\begingroup$ I'm not familiar with Statistical Learning. What do you mean with "degrees of freedom for tuning"? What would be the degrees of freedom for tuning in the example you provided? $\endgroup$ – a.arfe Sep 6 '14 at 12:00
  • 1
    $\begingroup$ In the first two examples, you can tune all parameters independently, so your degree of freedom is the number of parameters. In the third (regularized) example, you have the additional restriction that the parameters are small on average (what small means depends on the regularization). So you're not free to change all parameters independently: If you set one parameter to a high value, you implicitly force other parameters to be smaller. Your degrees of freedom thus are smaller than without the restriction, this is quantified by VC or effective parameters. $\endgroup$ – dobiwan Sep 6 '14 at 17:22
2
$\begingroup$

"Effective parameters" can also be referred to as "effective degrees of freedom". In a linear model, we note that the leverages -- the amount the fitted value changes with the actual value $\partial \hat y_i \over \partial y_i $ -- can be added up to obtain the degrees of freedom of the model. This way of calculating degrees of freedom can then be applied to non-linear models.

For example suppose your "model" is to take the average of the $k$ nearest neighbours. For each observation, the fitted value is the average of $k$ values. The observation itself will be amongst these $k$ nearest points, and its own contribution will be $y_i/k$, hence the leverage for every observation is $1/k$, and for $n$ observations the effective degrees of freedom is $n/k$.

$\endgroup$
  • $\begingroup$ Note that the expression ∂y^i∂yi to compute the degrees of freedom is only valid when the model has additive Gaussian noise. With generic noise the result is the same but the derivation uses that knn is a linear smoother. $\endgroup$ – aripakman Apr 26 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.