1
$\begingroup$

I am performing data imputation on a large matrix [100000,34] of past measurements that contains missing values (rows are time-steps and columns are stations). So far I've used several machine-learning techniques for the imputation and I want to compare between them. I do it using 10-fold CV on the non-missing values (i.e., in each fold, additional 10% of the non-missing values are set as nan, and then I perform the imputation on the entire matrix and evaluate on the 10% I put aside). I also do repeated-holdout (Monte Carlo cross-validation) so I can sample different missing interval lengths for evaluation.

Eventually, I want to find the best performing algorithm FOR IMPUTATION and to impute my entire matrix. I am not using the imputed matrix or features for prediction at all (i.e, I don't have a target).

What bothers me is that I don't set aside a testing-set for the final evaluation of the chosen imputation model, and I'm not sure that this is the right workflow. In my intuition, it shouldn't matter much since my testing set is the real missing values in the matrix and my only goal is to choose the algorithm that will impute it best. Nevertheless, everywhere I read people highlight how important this stage is.

So, Is this stage critical to my application? I would appreciate some advice from experienced machine learners...

Thanks

$\endgroup$
2
  • $\begingroup$ It might make you feel better if you chose the winner based on the average of CV scores... the holdout is the holdout so you can see that you're stable but choose your model using the CV scores. If the holdout was vastly different I might wonder if you're even able to simulate the "missing" properly. Maybe true missing isn't random? $\endgroup$ – Josh Jul 14 '20 at 20:58
  • $\begingroup$ I used CV for hyperparameter tuning (for example, one of the estimators is Random Forest and it has many parameters). Later, I use the repeated holdout for choosing the winner, since that way I am able to estimate the performance of the tuned models per different missing interval lengths (I extract different sizes of missing chunks for this validation). I think the results are pretty consistent. $\endgroup$ – iditbela Jul 15 '20 at 5:51
1
$\begingroup$

This is not necessarily an answer, but it was too long for the comment section.

In a simple scenario one would like to impute features to predict a target. Which imputation method works best is indeed a question of experimentation, a hyperparameter if you will; to be found out using for example k-fold cross-validation. The final verdict on the whole procedure though would be gotten by applying the imputation, the model to predict the target in order to find out the final metric on the target. It would also answer the question if the imputation helped at all; maybe dropping observations is better.

If there is no target, i.e. the targets are the imputed values, then one can still hold out a test set and report the accuracy of the imputation method per feature by indeed masking some known values.

Depending on the workflow, it might be advisable to mark each value as imputed or not. The fact that the value was missing might be a signal of itself, more so than the imputed value. Some digging into the causes of the missing values could give more confidence in the imputation. A weather station not reporting in a hurricane might have a very specific meaning! Missing at random (MAR) and missing completely at random (MCAR) are some keywords to look for.

HTH

$\endgroup$
6
  • $\begingroup$ Thanks for the reply, you've raised some good points. Yes, in my case, I don't have a target. And yes, I can hold out a final test set. But should I? BTW, I'm doing a multivariate imputation (using IterativeImputer from scikit-learn) with different estimators (Bayesian Ridge, Ransom forest, KNN). So far CV was used for hyperparameter tuning of the estimators of the IterativeImputer and repeated holdouts for later validation per interval length. Other than that, MCAR is never a reasonable assumption, but I do assume MAR. $\endgroup$ – iditbela Jul 14 '20 at 19:18
  • $\begingroup$ You loose nothing by setting apart a test set, no? It is just used for reporting the error on of the imputation. It would mainly serve to validate the cross-validation procedure. $\endgroup$ – spdrnl Jul 14 '20 at 21:54
  • $\begingroup$ Well, I loose some data I can learn from and so I might end up underestimating the error of the imputation... Also, technically, when I perform an imputation of the matrix, the entire matrix in imputed. So any test-set set apart will necessarily be imputed. Notice I wrote that after the CV I do repeated holdouts so I can estimate performance per missing interval lengths. In that stage, I choose the winner. As I see it, in that stage (many repeated holdouts), I already have a good estimation on the performance of the different models per different missing chunk sizes. So isn't that enough? $\endgroup$ – iditbela Jul 15 '20 at 6:04
  • $\begingroup$ My point is, I can't see why the error of the imputation on a test-set I set aside would be any different than the estimation obtained by many repeated holdouts. $\endgroup$ – iditbela Jul 15 '20 at 6:22
  • $\begingroup$ Well, to make sure you did not overfit using the holdouts; as always. Imagine trying thousands of combinations using the holds. $\endgroup$ – spdrnl Jul 15 '20 at 13:14
0
$\begingroup$

We build a prediction model using one or machine learning algorithms for future use. The historic data used to train the model, for those you know the outcome already. So what you want to know at the end of the day, is how your prediction models perform on future data. Your test set or validation set are these 'future data'.

In general, the larger your training set is in terms of cases/observations the less likely overfitting is to occur - the situation you want to investigate using a separate test or validation set. On the other hand, the more flexible a prediction model is the higher the risk over overfitting.

For training classifiers some theoretical results are available. Confidence intervals for the error rate has been derived for linear and quadratic discriminant analysis. See chapter 10 in the book Discriminant Analysis and Statistical Pattern Recognition by G.J. McLachlan (Wiley), 2004. These confidence intervals provide bounds for the performance on future test data. For more complex learning algorithms, such theoretical results do not yet exist (to my knowledge, that is).

For nonparametric classifiers, you can only assess future performance using validation data.


Specifically, you perform imputation of missing values. As stated, different missing value mechanisms can be the case:

  • MCAR (Missing Completely At Random)
  • MAR (Missing At Random)
  • NMAR (Not Missing At Random)

MCAR means that the probability that a data point is missing (i.e. a '?' in your data matrix), the observed data you do have - they give no clue what so ever about the most likely value (range) of the missing data point. Strict independence between the missing data mechanism and the observed data variables is present.

MAR means that the (unknown) value of the missing data point itself does not influence in any way the probability that the data point has gone missing. However, one or more observed data values do statistically determine whether that particular data value is missing. Strict independence between the actual value of the missing data point and the probability that it is missing, that is the case with MAR.

NMAR means MAR, plus that the value of the absent value itself statistically influences the probability that it has gone missing.

Imputation of missing values can be successful using existing schemes (such as the EM-algorithm), when the missing data mechanism is MCAR or MAR. Nonetheless, many scientific studies have shown that missing value imputation by the EM-algorithm tends to yield imputed values closer to the (conditional) means of the distribution of the missing data than is the case in the true underlying distribution. Multiple imputation is one approach to overcome this problem - a real challenge when your data are not normally distributed.

Final conclusion and my advice is as follows. You model the missing data mechanism, perform imputation and build a predictive model afterwards. Hence, even more parameters are being fitted in your scheme than when just building a prediction model in 'one go'. More parameters combined with a complex scheme that models possible higher order relations between variables (namely the missing data mechanism) - these factors make validation by independent test sets even more necessary.

$\endgroup$
3
  • $\begingroup$ Thanks, but I'm not sure you're answering my specific question regarding the imputation I'm performing $\endgroup$ – iditbela Jul 14 '20 at 18:46
  • $\begingroup$ Answer has been expanded as to completely cover your question. $\endgroup$ – Match Maker EE Jul 14 '20 at 19:58
  • $\begingroup$ Thanks, did you see my comment to @spdrnl? I don't build a predictive model at all. Sorry if my original question was not clear enough. $\endgroup$ – iditbela Jul 14 '20 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.