1
$\begingroup$

I'm working on an algorithm that estimates two parameters of its input data. I have a representative set of samples with the true parameters, to act as a ground truth. As this algorithm uses a threshold that has to be adjusted manually by the user, I was wondering if it could be possible to use my "ground truth" samples to find the best threshold for my training set. How can I express an error metric for two dependant variables, since for each threshold value and sample I'll have two parameters estimated by the algorithm.

I would really appreciate any hint.

Thanks in advance.

Improve this question
$\endgroup$
3
  • $\begingroup$ What "you mean by adjusted manually by user"? Also I am not clear about the dependent variables part( you have mentioned parameters are independent). One soultion is you can consider Mean squared error (MSE) as a metric. $\endgroup$
    – vinux
    Commented Dec 29, 2011 at 14:16
  • $\begingroup$ The two parameters that the algorithm estimates are visualized by the user, and he can try with different thresholds until he thinks the estimation is ok. It's an "intuitive" procedure. Both parameters are dependent of the threshold (I've corrected the question). How can I express both parameters with MSE? $\endgroup$
    – Federico
    Commented Dec 29, 2011 at 14:30
  • $\begingroup$ How about the covariance matrix of the predictions made in each of the two dimensions? I've never done/seen this but it's just something that occurred to me that you may look into. Also, you may check that the predictions are unbiased in each dimension $\endgroup$
    – Macro
    Commented Jan 28, 2012 at 17:16

1 Answer 1

Reset to default
1
$\begingroup$

Since you know the true parameters, you could calculate the difference of estimated value and true parameter. Sum of squares of this difference would be a good error metric. i.e. say $ \hat\theta_1, \hat\theta_2,\dots,\hat\theta_n$ are estimates and $\theta_1,\theta_2,\dots, \theta_n $ are the true values. Then take $E = \sum_{i=1}^{n}(\hat\theta_i-\theta_i)^2$. You could consider sum (or weighted sum) of E=E1+E2 in case of two parameters.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This may be reasonable when the parameter estimates are almost independent and have almost the same variances, but that's a rare situation. Consider OLS fitting of a line, with two estimates (intercept and slope). Frequently the variance of the intercept is huge compared to that of the slope: do you really want to recommend penalizing differences in intercepts the same as differences in slopes, then? $\endgroup$
    – whuber ♦
    Commented Dec 29, 2011 at 17:04
  • $\begingroup$ I thought about this. That is the reason I put the weighted sum in the bracket. I guess MAPE ( mean absolute percentage error also a candidate for error metric. $\endgroup$
    – vinux
    Commented Dec 29, 2011 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.