Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

I'm working on an algorithm that estimates two parameters of its input data. I have a representative set of samples with the true parameters, to act as a ground truth. As this algorithm uses a threshold that has to be adjusted manually by the user, I was wondering if it could be possible to use my "ground truth" samples to find the best threshold for my training set. How can I express an error metric for two dependant variables, since for each threshold value and sample I'll have two parameters estimated by the algorithm.

I would really appreciate any hint.

Thanks in advance.

share|improve this question
What "you mean by adjusted manually by user"? Also I am not clear about the dependent variables part( you have mentioned parameters are independent). One soultion is you can consider Mean squared error (MSE) as a metric. – vinux Dec 29 '11 at 14:16
The two parameters that the algorithm estimates are visualized by the user, and he can try with different thresholds until he thinks the estimation is ok. It's an "intuitive" procedure. Both parameters are dependent of the threshold (I've corrected the question). How can I express both parameters with MSE? – Federico Dec 29 '11 at 14:30
How about the covariance matrix of the predictions made in each of the two dimensions? I've never done/seen this but it's just something that occurred to me that you may look into. Also, you may check that the predictions are unbiased in each dimension – Macro Jan 28 '12 at 17:16

1 Answer

up vote 0 down vote

Since you know the true parameters, you could calculate the difference of estimated value and true parameter. Sum of squares of this difference would be a good error metric. i.e. say $ \hat\theta_1, \hat\theta_2,\dots,\hat\theta_n$ are estimates and $\theta_1,\theta_2,\dots, \theta_n $ are the true values. Then take $E = \sum_{i=1}^{n}(\hat\theta_i-\theta_i)^2$. You could consider sum (or weighted sum) of E=E1+E2 in case of two parameters.

share|improve this answer
This may be reasonable when the parameter estimates are almost independent and have almost the same variances, but that's a rare situation. Consider OLS fitting of a line, with two estimates (intercept and slope). Frequently the variance of the intercept is huge compared to that of the slope: do you really want to recommend penalizing differences in intercepts the same as differences in slopes, then? – whuber♦ Dec 29 '11 at 17:04
I thought about this. That is the reason I put the weighted sum in the bracket. I guess MAPE ( mean absolute percentage error also a candidate for error metric. – vinux Dec 29 '11 at 17:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.