I want to fill in missing values in a dataset. I thought i would use regression but I'm unsure of it and of which regression to use as well. My dataset has many (10-20) features and most of them are highly correlated—but not linearly. For example there exist relations like $a=\frac{b^3c}{2}$ or $d=\frac{a^2c^3}{4\pi}$, and others like that.

There's quite a lot of missing values and so I'm filling out the ones that can be done by direct formula. But for others, where I don't have the exact relation or if more than one of the inputs is missing (hence, I can't calculate), I want to find an association rule. Which would be the best regression method to apply in this case to fill these values by using multiple features to determine a single feature?

Updated info: The data is astrophysical and there are physics equations (interrelations of mass, radius, period, temperature, etc.) that relate some of them (like the examples I have given), but the exact values of associations/formulas are still under question for the sub-domain that I want to apply it on (binary star systems, not just single stars). These associations definitely exist.

Hence, I want to apply regression using multiple features (trying absolutely all combinations including changing the number of features considered) to match a single feature's known values. Then I would use the association with the least error to impute missing values of that feature. Which exact regression model do I use? I was checking exponential regression, but I'm not at all sure if its correct.

  • $\begingroup$ Are these relations ($a=(b^3)c/2$ or $d=(a^2)(c^3) / 4pi$) you are talking about some kind of 'true'/known association, or something you've estimated from your data? In the first case, this would mean you need to find out those formula, but in the latter, you might want to check whether non-linearity is still the case when using more predictors to estimate the missing values. On a sidenote, I would urge you not to simply replace your missing values once, but look at multiple imputation techniques $\endgroup$ – IWS Mar 16 '17 at 8:02
  • $\begingroup$ The data is astrophysical and they are physics equations ( interrelations of mass, radius, period, temperature...etc) so they are true associations ( like the relations have to be that way..) but the exact associations/formulae are still under question for the sub-domain that i want to apply it on ( binary star systems and not just single stars).... $\endgroup$ – Pratham Agarwal Mar 16 '17 at 8:55
  • $\begingroup$ I guess that in that case you have to make some assumptions on which equation you believe applicable. If you still have doubt, you might want to do sensitivity analyses where you follow another paradigm of equations. $\endgroup$ – IWS Mar 16 '17 at 8:58
  • $\begingroup$ But lets say all equations that relate these features are definitely of d type that i've mentioned ( highly correlated non-linear like a=(b^3)c/2 .. proof of concept kinda since its physics ), and i want to apply regression using multiple features(trying absolutely all combinations including changing the number of features considered) to determine a single feature's values (then i would use the association with the least error to impute missing values of that feature..) which exact regression model do i use? i was checking exponential regression but i'm not at all sure if its correct $\endgroup$ – Pratham Agarwal Mar 16 '17 at 9:12
  • $\begingroup$ I am unsure how to estimate your missing data, but I would recommend you to edit this information as an update to your question. Context, what you've already tried and what kind of missing pattern you are experiencing is all useful information for someone who might know the answer, and should be part of your question's text for clarity's sake. $\endgroup$ – IWS Mar 16 '17 at 9:15

Gaussian processes work well. They'll give you as output not only a curve through your cloud of points, but:

  • it'll give you a curve of the uncertainty of the curve. ie, you'll have an approximate upper and lower bound for the curve
  • in areas where you have more data, the uncertainty will go down; in areas with less data the uncertainty will increase
  • the curve will be more or less confident about trading off bias and variance, depending on how much data there is for some particular region

You can look at Figure 2 from "Gaussian Processes for Regression: a Quick Introduction", by Ebden 2008, http://www.robots.ox.ac.uk/~mebden/reports/GPtutorial.pdf , for example, see below. The shaded pink area represents the uncertainty. The black dots are data points, and the black curve is the curve fit by the Gaussian Process:

enter image description here

There's also a recent article in Wired about Ghahramani's work on using GPs in machine learning https://www.wired.com/2017/02/ai-learn-like-humans-little-uncertainty/

  • $\begingroup$ Thank you for your help. I've updated my post based on the comments and given more information about the data, could you tell me if your method will still work ? $\endgroup$ – Pratham Agarwal Mar 16 '17 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.