Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

256 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
820 views

Missing value imputation in huge dataset

I have a huge data (4M x 17) that has missing values. Two columns are categorical, rest all are numerical. Given the huge amount of data, running any imputation method runs forever. What should I do? ...
6
votes
0answers
850 views

Canonical correlation analysis on a MICE data set

I am looking to do a canonical correlations analysis (CCA) in R, using the CCA package, on a multiply imputed dataset (obtained from the mice package). I know that ...
5
votes
0answers
1k views

Adjusting Standard Error for Imputed/Generated Regressors

This is my first question, so I hope this is a valid question. I am surprised that I have seen only few questions (and no answer helping me out) referring to the adjustment of variance estimators in ...
5
votes
0answers
266 views

How to do multiple imputation for spatial models?

I'm trying to estimate various spatial models such as spatial autoregressive regression (SAR), Spatial Durbin Model (SDM), and Spatial Error Model (SEM) but have missing data throughout my variables. ...
4
votes
0answers
24 views

Auxiliary variables for imputation in survey data

I have a situation where I have a survey that contains two parts, everyone answers all of the questions on the first part, and a sample (10 percent) of the people are selected to answer some further ...
4
votes
1answer
411 views

Are VAE used for missing data imputation in multivariate time series? If not, what is used?

Multivariate time series are, to the best of my understanding, one of the few cases where Deep Learning still hasn't had its AlexNet moment. I'm especially interested to the case where most of the ...
4
votes
0answers
542 views

Multiple imputation of glm binomial size parameter

Suppose we have a generalized linear model with a binomial response $y_i\sim \mathrm{bin}(n_i,p_i)$ where $p_i$ is determined by the linear predictor in the usual way via some link function. Is there ...
4
votes
0answers
231 views

Is missing outcome in survival analysis a problem?

I would like to look at survival in time to event data. So individuals either have an event, or are censored. My problem is the sensitivity for detecting an event differs between arms. I.e. in the ...
4
votes
0answers
485 views

Cross validation for multivariate imputation

I am currently using the mice: Multivariate Imputation by Chained Equations in R (JSS 2011 45(3)) package. Consider the following example. I am using Sites B to Z and mice() to help infill missing ...
3
votes
0answers
20 views

Theory behind Multivariate Imputation with Chained Equations

Can anyone provide a reference to the theory that supports multivariate imputation with chained equations (MICE). I know Rubin has provided this for MI but MICE is a Gibbs sampler (I have never seen ...
3
votes
0answers
293 views

Imputation and nested cross-validation

I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
3
votes
0answers
250 views

How to use MICE in R to fill missing values in test set?

It seems that MICE does not have a "predict" function which allows to use a fitted mids object to predict the missing values in test data set. I can certainly ...
3
votes
0answers
722 views

Imputing missing values and SVD

Similar questions have been asked a lot of times but I have not found an answer that gives an intuitive explanation as to why this works. For reference I have read the answers here and here. As I ...
3
votes
0answers
1k views

How to use pooled results from multiple imputation?

I've been reading some posts about data imputation using multiple imputation, specifically the MICE R package. I get the main idea of creating multiple datasets with imputed data. The part that is not ...
3
votes
0answers
55 views

Bad data imputation: How to impute specific bad data and replace it with realistic ones?

I conducted an experiment with multiple human participants to analyze some air traffic scenarios. Some data has turned out to be very unrealistic. Take a look at the following pics which shows the ...
3
votes
1answer
191 views

Should I impute Missing Laboratory Data? - Confused about MAR-MNAR

I am trying to model hospital readmission using both categorical and nominal variables. Laboratory data comprises a big chunk of the nominal variables, the problem is not every patient has laboratory ...
3
votes
0answers
755 views

General practice to impute missing values

There are multiple resources and answers on type of imputations and packages that can help in imputing the missing values or how to use a particular package. But there are little to no resources ...
3
votes
0answers
46 views

Should I select features for imputation

When imputing a missing value (in my case using MICE) - should I use all the variables in the dataset or should I use only the variable which correlate most with the missing values I want to impute? ...
3
votes
0answers
5k views

Best methods to deal with missing categorical data?

So I've read about imputation already, but what can I do when the data is categorical, where there is no mean or median? For example, if the categories are Male/Female. Would assigning missing ...
3
votes
0answers
124 views

Random Forest Models: creating correlated features

I'm trying to understand how correlated (multicollinear) predictors affect predictive power and / or variable importance in tree models, e.g. Random Forest models. Particularly, I'd like to know if ...
3
votes
0answers
858 views

Tobit model (or survival analysis) for imputation of censored data in R

I have some data with missing values. For the missing observations I know a range, in which the true values are. I want to use a tobit model to predict the variable with these missing values. The ...
3
votes
0answers
27 views

Imputation of a (weird) multivariate outcome

I am working with a dataset in which the outcome of interest is a vector of dates of particular events: (date_1,date_2,date_3,...,date_n). Some of these outcome vectors are completely missing, but I ...
3
votes
0answers
419 views

Multiple Imputation and Matrix Completion

It is quite common that data sets will contain missing values in them. Suppose we want to try to fill in the missing values. For this we have techniques such as single/multiple imputation and matrix ...
3
votes
0answers
464 views

How to compare and validate imputation models?

I've seen a lot of interesting questions here about multiple imputation and also great answers that helped me a lot to impute my data. I've used Predictive Mean Matching, EMB and I would like to use ...
3
votes
0answers
647 views

Imputation with mice: recode variables before or after imputation?

I am using mice in R, a chained equations (sequential regression) algorithm, to impute a series of polytomous variables (e.g. ...
3
votes
0answers
97 views

Fuzzy Record Linkage of Spatial Datasets

I have two datasets describing real-estate properties Dataset 1 describes building characteristics; it includes the location of the entrance to the building along with building descriptions and ...
3
votes
0answers
389 views

How to impute data without missing at random?

Recently I got a global longitudinal data from several countries, and each county has one outcome variable and two predictors from 1995 to 2008. I found one of the predictors is always missing in each ...
2
votes
0answers
116 views

ARIMA with external regressors for district heating load time-series imputation

TL;DR ARIMA model does not work as expected. There is a load for district heating with some external regressors like temperature and wind speed. There is missing data which we like to impute. We work ...
2
votes
0answers
19 views

Imputation that takes into account both relationships among variables and spatial adjacency?

I have a dataset with 13 variables and 50 observations representing the U.S. states. The variables represent the land use intensity of different agricultural industries in each state. Of those 650 ...
2
votes
1answer
73 views

Missing data imputation when all variables have some missing data?

I am working with my resident survey data (n=8356) including 59 items, most of which are ordinal variables scored from 1 to 7, and others are continuous variables (e.g., age, residency length). ...
2
votes
0answers
259 views

Interpolation versus imputation for time series on chemical profiles of water wells

So I am working with some data on water wells and time series of chemical pollutant tests on those wells. There are 10 chemicals and 10 years in the data. My goal is to do some clustering on the wells ...
2
votes
4answers
93 views

imputation FOR random forests

I was wondering what imputation method you would recommend for data to be fed into a random forest model for a classification problem. If you google for "imputation for random forests", you get a lot ...
2
votes
0answers
43 views

What value to impute for informative NA values in R without misleading model

I'm building a model (random forest) in R to predict a rare event (scoring a goal in soccer). I have event-level data, which provides a log of all the actions (pass, tackle, foul, save, shot, goal) ...
2
votes
0answers
195 views

Partial imputation of missing dates

I'm working with dataframes (one for each of 185 locations) that shows sums of occurrences for each calendar date. There are no 0 values for occurrences in the entire dataset. There are several ...
2
votes
0answers
407 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
2
votes
0answers
345 views

Overcome NA's in Random Forest and SVM?

I am using clinical data for prediction purposes with SVM and RF. Two of my columns are as following: ...
2
votes
0answers
86 views

To impute or not - community consensus for reporting accuracy of an imputed model

I have a model generated using an imputed data set with imputation accuracy of 75%. If the model using imputed data has an accuracy of 80% What would be the community consensus to report the ...
2
votes
0answers
93 views

Impute missing population values in Census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
2
votes
1answer
44 views
2
votes
0answers
406 views

stan - 2 approaches to missing value imputation; which is better and why?

So, me and a colleague have to impute some data, x, given a categorical variable. We arrived at two different approaches: a) as in the tutorial: split x into x_obs and x_mis, and treat x_mis as ...
2
votes
0answers
314 views

Order in which Data Imputation, Outlier removal, Data Transformation should be done?

While preprocessing data we need to carry out the following steps: Missing value Imputation Outlier Detection Transformation of Data In what order should we perform these 3 steps while preprocessing ...
2
votes
0answers
943 views

pooling over imputation with LASSO in R

I've been trying to find an answer to my question, with no success. It could be I'm just not looking for it in the right way (I'm also learning about imputation and lasso as part of the process). So I ...
2
votes
0answers
61 views

Imputing missing counts in a time series (with variance?)

We're using a sonar to enumerate migrating fish. The date & time are recorded with each event that a fish passes the sonar station. We've noted two very strong time-varying components to the ...
2
votes
0answers
889 views

Imputing categorical 'string' data for missing values in python

I'm working on binary classifier model with 30% N/A values aka missing fields. I have one continuous feature and two categorical 'string' features missing in my data. For a continuous feature, I've ...
2
votes
0answers
547 views

Feature selection and subsetting when most features are missing many data points

I have a data set for which I would like to build a predictive model, and it contains quite a few features where most of the data is missing. The feautres are also highly correlated pair-wise and I ...
2
votes
0answers
554 views

Missing data MAR not MAR - multiple imputation

I am preparing my data for CFA and multiple regression. I want to see how different types of parental involvement influence children's motivation and academic results. I have a sample size of 5000 ...
2
votes
0answers
3k views

Imputation in monthly univariate time series

I have a time series of the amount of apples sold in a specific Region. The time series include monthly values of 10 years (2006-2016). However two months are missing (February 2009 and July 2014). ...
2
votes
0answers
245 views

How do I impute missing values of a dataset with little pairwise correction using regression

I have a dataset with 10 independent variables and one response variable. They are all in integer types. V1 - V10 are numeric values rounded to closest integer values which are always in range of [1, ...
2
votes
1answer
29 views

Imputation for Industrial Survey

What do you think the best imputation method is for industrial surveys? Methodology reports usually mention historical background, administrative data or hot-deck, but according to the theory, the ...
2
votes
0answers
85 views

Method for predicting price based on Geographical market, Product, and Company

I have a dataset which tracks the prices of 21 products, charged by 24 companies, in 150 different cities across the globe. However, the data set has missing values--that is, I might have Company X's ...

1
2 3 4 5 6