1
$\begingroup$

I have constructed a continuous variable by using two different methods. Now I want to know the variable created under which way is the best and should be included in the model. Some preliminary results show that the variables constructed using both methods can significantly predict the outcome (p=0.03 vs. p=0.04). The incremental R-squared added by the variable constructed by both ways is the same. The AIC results for the model with the variable created using method one is 8000.2 and 8000.9 for the model with the variable constructed using method two.

Is there any other method I can use to compare which variable is relatively better?

$\endgroup$

2 Answers 2

0
$\begingroup$

There will not be a "data driven way" to select the method from a single sample, for the method which is best today might not be best tomorrow with a new sample.

My advice would be to perform some sort of calibration on your methods. Much like lab work, you should have some sort of control for each method and then ideally compare the method's estimate against the truth. Then, my advice would be to select the method with the lowest residual variance because errors in variables can attenuate the estimated effect towards the null.

If this is not possible, you could just take the average between the two methods and use that as the variable into the model.

$\endgroup$
0
$\begingroup$

Since both methods perform similarly well via p-value and AIC, another aspect you can consider is their abilities to generalize to new data. Cross validation will give you a good measure of this, since iteratively training on a susbet of data while testing on the remainder emulates this process and also gives a measure of how sensitive the metrics are to outliers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.