0
$\begingroup$

I'm trying to do measurements based on image processing : first I do some image processing to find detect the pixels that changed between the background and my image, and then I perform a linear regression to find the number of objects from the number of changing pixels.

The issue is that I have rather few samples (24 for now) because it is very time-consuming to get them.

I'm trying to find the optimal parameters (3 for the image processing + 2 for the regression) and to estimate the variance of the residuals. For finding the optimal parameters, I do a grid search on the 3 image processing parameters, and for each I compute the regression (and estimate the variance of residuals).

The problem is that I'm not sure how best to estimate this variance : - If I were to measure a fixed value + noise, I know that I take the sum of the squares of the residuals, and I divide by the number of samples minus 1 - If I make a linear regression, from what I understand I have to divide by the number of samples minus 2. - But I'm not sure about my situation : should I divide by N-2 or by N-5? (ie do the parameters of the grid search count as lost degrees of freedom the same way as the parameters of the regression?)

Thank you very much in advance Felix

PS : just to make sure, is it at all correct to try to estimate the variance this way or should I go for separated learning/testing sets and/or to cross validation?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I am not an expert, but the degrees of freedom is related only to the regression parameters, not hyperparameters on grid search. When thinking about degrees of freedom I like to make an analogy with simple mean and variance estimation. Since we have N data points, we use it to estimate mean, thus when calculating variance we have lost our freedom by one degree. In regression context, it is the same, we use data points to estimates the parameters, not the hyperparameters. Your samples does not have anything to do with your learning rate, or penalty criteria, this is something related in how your model will behave to estimate parameters. In conclusion, you should use N-2 if you have two features.

It is my first time answering a question, if I do not made myself clear enough, please, let me know.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Just to complement: I forgot to mention, but as you have such a few instances to train your model, it is good to try using bagging to test your performance as well. $\endgroup$
    – Victor Oliveira
    Commented Mar 1, 2019 at 12:58
  • $\begingroup$ Thanks for your answer. I'm not sure however if we can really consider the image processing parameters as hyperparameters, because we are learning them too (it's not like choosing an arbitrary value as for the number of clusters in k-means for exemple). $\endgroup$
    – felix
    Commented Mar 4, 2019 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.