Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
My question might be a bit ambiguous, but I started to wonder what does the "effective parameters" mean in machine learning? I have heard few professors of machine learning in my university talk about effective parameters (the context was about k-nearest neighbors or Gaussian mixture models etc.).
Any clarification on what this might mean? Thank you for any help!
P.S. I have also seen this term in the book: The Elements of Statistical Learning 2nd edition, Trevor Hastie, page 15.
Some models (here: linear regression) have parameters $\beta$:
$$ \hat{y} = \sum_{i\in\{1..p\}} \beta_i x_i + \beta_0 $$
For the same number of input features, more complex models (here: basis expansion to a quadratic model) have more parameters:
$$ \hat{y} = \sum_{i,j\in\{1..p\} \atop i\le j} \beta_{ij} x_ix_j + \sum_{i\in\{1..p\}} \beta_{i} x_i + \beta_0 $$
In general, models with more parameters are more flexible (because there are more parameters to tune to fit the model to the data), but are more difficult to fit and thus more likely tend to overfit. Regularization helps overcome these problems by reducing the degrees of freedom for tuning, thus reducing the complexity a bit:
$$ \hat{y} = \sum_{i,j\in\{1..p\} \atop i\le j} \beta_{ij} x_ix_j + \sum_{i\in\{1..p\}} \beta_{i} x_i + \beta_0 \textrm{, with} \sum \beta^2 \textrm{ small} $$
We can express the 'model complexity' or 'degrees of freedom for tuning' in terms of 'effective parameters': For the model that you investigate, find the parameter-based model (without regularization) that has the same degree of freedom for tuning. The number of parameters for that model is the effective number of parameters for your model.
This generalization can also be extended further to models that are not based on parameters (e.g. k nearest neighbor). Then the degree of complexity is known as Vapnik–Chervonenkis (VC) dimension. Conveniently, for (non-regularized) linear regression, it is $p+1$, the actual number of parameters. Elements of Statistical Learning, section 7.9, gives more information about that (and they seem to use the terms 'VC', 'effective number of parameters' and 'model complexity' more or less interchangably).
"Effective parameters" can also be referred to as "effective degrees of freedom". In a linear model, we note that the leverages -- the amount the fitted value changes with the actual value $\partial \hat y_i \over \partial y_i $ -- can be added up to obtain the degrees of freedom of the model. This way of calculating degrees of freedom can then be applied to non-linear models.
For example suppose your "model" is to take the average of the $k$ nearest neighbours. For each observation, the fitted value is the average of $k$ values. The observation itself will be amongst these $k$ nearest points, and its own contribution will be $y_i/k$, hence the leverage for every observation is $1/k$, and for $n$ observations the effective degrees of freedom is $n/k$.
