How to handle variable size input data (incomplete) to build/train a model for regression?

Question

Suppose you have the classical example of predicting house prices and you have as input features area size, built year, number of previous owners, city, number of floors, number of bedrooms, etc. But in this case, the expected outputs are the sale price, the rent price and annual tax.

Now suppose you have a lot of input data but one or more features are missing, also not all the outputs (labels) are present. For example:

| Area |  Year | owners | rooms || sale | rent | tax |
------------------------------------------------------
|   80 |  2001 |      X |     4 || 250k |    X | 200 |
|   70 |  2015 |      X |     2 || 125k | 1200 |   X |
   ...
|   60 |  1995 |      4 |     2 ||    X | 1000 | 100 |
|    X |     X |      2 |     3 || 300k | 1600 | 330 |
   ...
|    X |     X |      1 |     X || 220k |    X |   X |
|    X |  2005 |      2 |     X ||    X |    X | 150 |

A possible scenario I think for this could be gathering data from different sources like real state companies, Government entities, self-published post by owners, etc. (Different datasets with overlapping features and outputs)

I think that having some amount of this incomplete data is enough to generate a model to estimate the outputs.

I have found some approaches like filling the missing values with mean or with zero. Or discarding rows with missing values. But I'm looking more something like ignoring some part of the model when an input is not present or guess the value based on the dependency with other features, but this is not needed.

What kind of model could be used or what training approach should be used to address this type of problem?

NOTE: This question is closely related, but after 8 years I hope we have new approaches to try.

NOTE 2: I originally posted this question on StackOverflow and was deleted, but I post below the comments I got:

• You may try with a non-fully connected neural network, if some rows of your dataset are not fully filled, or you can try with dimensionality reduction to determine which columns to keep, if some of them are not fully filled. – Daniel R. Oct 4 at 16:54

• One option you didn't mention is to add a separate input for 'unknown' and let the NN learn what to do with it. But it seems more like a question for datascience.stackexchange.com – de1 Oct 4 at 17:54

• Thanks for your comments. @DanielR., do you have any resource on this non-fully connected NN? I don't consider dimensionality reduction yet because most of the rows have missing fields, if not all. – GVelascoh Oct 5 at 8:18

• @de1 do you mean something like having an additional binary input indicating if certain feature exist? If I have N features, use 2N features (N originals + N binaries)? Any resource on this approach? – GVelascoh Oct 5 at 8:18

• It was an intution, this paper might be related but didn't have time to study it: link There is also a discussion here: link As mentioned I think it might be better to move the question as stackoverflow is meant more for precise programming questions. Your question is more open ended, there may be current good approaches. But they depend onn your data and your model. New approaches may be developed in the future. A moving target. – de1 Oct 5 at 8:54

• I read this paper on sparse neural networks, link and at page 317 (3 of the pdf) they state that if a representation is both sparse and robust to small input changes, the set of non-zero features is almost always roughly conserved, and also that Different inputs may contain different amounts of information and would be more conveniently represented using a variable-size data-structure. You will have to build your model from scratch though, as I am not aware of any premade ones. Your NaN values would then count as 0's. – Daniel R. Oct 5 at 16:10

Answer 1

Imputation of some kind is a very common solution, but decision trees and tree-based methods such as boosted regression trees and random forests (in some implementations) can also handle missing values by splitting on NAs. That is, at a node, one split is created that specifically indicates whether the value is missing. The node may therefore take 3 values, for e.g. $X_1$ > 10, $X_1$ < 10, and $X_1$ = NA.

Answer 2

Do you want to predict multiple things? If so, you can

• use full information direct likelihood/joint model (but may be a more restrictive class of models), or
• use any model that can predict multiple outcomes (e.g. a multi-output neural network) with a loss function where missing observed values results in no loss (i.e. it doesn't matter what the model predicts for the non-observed outcomes, all incur the same/no loss), or
• multiple separate models (just some get more information) and some less (you primarily loose out in terms of information shared between the different outputs - e.g. if more or less higher sales price = higher rent = higher tax, or if there's joint model internal representations that can be transferred between prediction tasks - unless there's a systematic bias in the missingess, in which case the separate models could be biased).

If the problem is missing predictors, you typically want to impute. But how you impute depends on many things. For example whether you think that the missingness itself provides useful information (e.g. someone not providing information on a loan application could be a sign of hiding something, or perhaps the attribute doesn't exist such as "petrol take size in liters" for an electric car), or whether it just so happens to be unavailable without it providing any signal, and do you need to reflect the uncertainty in the missing values (may be particularly important for interpretability and/or when you want to avoid the missingness from being interpreted as containing information - like it happens when you impute an average and adding a missingness flag to the predictors)?

Answer 3

The general approach is to fill (pad) those values with some other neutral value. But you said you have tried that.

You're asking if you could shut down a part of the model that is responsible for some feature if that feature is not present in the example. I don't think it's possible.

guess the value based on the dependency with other features

This would mean to have a different model to predict the value of the feature based on other features right? But I think that approach is not very good if the number of features that are missing is high because you'd have to have a lot of models for a lot of combinations of features. Maybe something that you can do is have a different model based on the source where you have found the data.

I also guess, since you're asking this question, that the shared set of features is not big enough for your predictions to be relevant?