1
$\begingroup$

I have two datasets consist of different features. I did multiple linear regression on these datasets separately and calculated MSE and r-squared scores. After that, I combine these features and the target values and doing multitask regression. I compare these two methods score and there is an improvement.

My question is, how can I test statistical significance of newly added features ?

Summary of datasets

    Performed multiple linear regression on first 2 dataset.
    1. dataset: features:(158,40) target: (158,1) mse score: 0.143
    2. dataset: features:(158,14) target: (158,1) mse score: 0.207

    Perform Multitask learning on third experimental dataset.
    3. dataset(combined): features:(158,54) target: (158,2) 
         mse score for first dataset: 0.05
         mse score for second dataset: 0.09
$\endgroup$
0
$\begingroup$

You shouldn't compare the models here because they don't come from the same dataset. It's unfair to compare your third model with your first/second, because the data used in your third model has everything, and thus should perform better than your two individual models.

You can bootstrap if you can make your data overlapping, (How to compare models from different but related datasets?) has the details. But I'm not sure that's exactly what you're looking for.

You should use the merged data set to fit different models. Your models will be nested and comparable.

$\endgroup$
1
  • $\begingroup$ I think that these datasets are comparable. For example, in the first dataset I try to predict "A" and in the second dataset I try to predict "B". From my domain knowledge, A and B is related so when I combined them and predict them together with multi-task learning, the success is increase both of them. My purpose is, proving the relationship between A and B statistically. Because, for example if there is another dataset which try to predict "C", when I combine with it "A", maybe it also increase the success but there is no causation.Shorty, I want to prove the causation between the datasets. $\endgroup$
    – Batuhan B
    Jan 14 '17 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.