Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

240 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
730 views

Missing value imputation in huge dataset

I have a huge data (4M x 17) that has missing values. Two columns are categorical, rest all are numerical. Given the huge amount of data, running any imputation method runs forever. What should I do? ...
6
votes
0answers
831 views

Canonical correlation analysis on a MICE data set

I am looking to do a canonical correlations analysis (CCA) in R, using the CCA package, on a multiply imputed dataset (obtained from the mice package). I know that ...
5
votes
0answers
1k views

Adjusting Standard Error for Imputed/Generated Regressors

This is my first question, so I hope this is a valid question. I am surprised that I have seen only few questions (and no answer helping me out) referring to the adjustment of variance estimators in ...
5
votes
0answers
248 views

How to do multiple imputation for spatial models?

I'm trying to estimate various spatial models such as spatial autoregressive regression (SAR), Spatial Durbin Model (SDM), and Spatial Error Model (SEM) but have missing data throughout my variables. ...
4
votes
0answers
19 views

Auxiliary variables for imputation in survey data

I have a situation where I have a survey that contains two parts, everyone answers all of the questions on the first part, and a sample (10 percent) of the people are selected to answer some further ...
4
votes
2answers
5k views

Imputing missing values in Python using RandomForest model

I know some strategies of imputing the missing data, for example, using filling with zeros, using mean, median or the most frequent values. So what I don't quite understand till this point-how can ...
4
votes
0answers
523 views

Multiple imputation of glm binomial size parameter

Suppose we have a generalized linear model with a binomial response $y_i\sim \mathrm{bin}(n_i,p_i)$ where $p_i$ is determined by the linear predictor in the usual way via some link function. Is there ...
4
votes
0answers
201 views

Is missing outcome in survival analysis a problem?

I would like to look at survival in time to event data. So individuals either have an event, or are censored. My problem is the sensitivity for detecting an event differs between arms. I.e. in the ...
3
votes
1answer
295 views

Are VAE used for missing data imputation in multivariate time series? If not, what is used?

Multivariate time series are, to the best of my understanding, one of the few cases where Deep Learning still hasn't had its AlexNet moment. I'm especially interested to the case where most of the ...
3
votes
0answers
265 views

Imputation and nested cross-validation

I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
3
votes
0answers
184 views

How to use MICE in R to fill missing values in test set?

It seems that MICE does not have a "predict" function which allows to use a fitted mids object to predict the missing values in test data set. I can certainly ...
3
votes
1answer
354 views

Forecast (impute) missing discrete values in multiple time series

I'm looking to forecast (impute) missing discrete values in multiple time series in order to reach a target volume in a consolidated time serie. The context: I have salesmen that are selling ...
3
votes
0answers
914 views

How to use pooled results from multiple imputation?

I've been reading some posts about data imputation using multiple imputation, specifically the MICE R package. I get the main idea of creating multiple datasets with imputed data. The part that is not ...
3
votes
0answers
53 views

Bad data imputation: How to impute specific bad data and replace it with realistic ones?

I conducted an experiment with multiple human participants to analyze some air traffic scenarios. Some data has turned out to be very unrealistic. Take a look at the following pics which shows the ...
3
votes
1answer
157 views

Should I impute Missing Laboratory Data? - Confused about MAR-MNAR

I am trying to model hospital readmission using both categorical and nominal variables. Laboratory data comprises a big chunk of the nominal variables, the problem is not every patient has laboratory ...
3
votes
0answers
718 views

General practice to impute missing values

There are multiple resources and answers on type of imputations and packages that can help in imputing the missing values or how to use a particular package. But there are little to no resources ...
3
votes
0answers
45 views

Should I select features for imputation

When imputing a missing value (in my case using MICE) - should I use all the variables in the dataset or should I use only the variable which correlate most with the missing values I want to impute? ...
3
votes
0answers
4k views

Best methods to deal with missing categorical data?

So I've read about imputation already, but what can I do when the data is categorical, where there is no mean or median? For example, if the categories are Male/Female. Would assigning missing ...
3
votes
0answers
124 views

Random Forest Models: creating correlated features

I'm trying to understand how correlated (multicollinear) predictors affect predictive power and / or variable importance in tree models, e.g. Random Forest models. Particularly, I'd like to know if ...
3
votes
0answers
801 views

Tobit model (or survival analysis) for imputation of censored data in R

I have some data with missing values. For the missing observations I know a range, in which the true values are. I want to use a tobit model to predict the variable with these missing values. The ...
3
votes
0answers
27 views

Imputation of a (weird) multivariate outcome

I am working with a dataset in which the outcome of interest is a vector of dates of particular events: (date_1,date_2,date_3,...,date_n). Some of these outcome vectors are completely missing, but I ...
3
votes
0answers
407 views

Multiple Imputation and Matrix Completion

It is quite common that data sets will contain missing values in them. Suppose we want to try to fill in the missing values. For this we have techniques such as single/multiple imputation and matrix ...
3
votes
0answers
414 views

How to compare and validate imputation models?

I've seen a lot of interesting questions here about multiple imputation and also great answers that helped me a lot to impute my data. I've used Predictive Mean Matching, EMB and I would like to use ...
3
votes
0answers
620 views

Imputation with mice: recode variables before or after imputation?

I am using mice in R, a chained equations (sequential regression) algorithm, to impute a series of polytomous variables (e.g. ...
3
votes
0answers
97 views

Fuzzy Record Linkage of Spatial Datasets

I have two datasets describing real-estate properties Dataset 1 describes building characteristics; it includes the location of the entrance to the building along with building descriptions and ...
3
votes
0answers
478 views

Cross validation for multivariate imputation

I am currently using the mice: Multivariate Imputation by Chained Equations in R (JSS 2011 45(3)) package. Consider the following example. I am using Sites B to Z and mice() to help infill missing ...
2
votes
0answers
17 views

Imputation that takes into account both relationships among variables and spatial adjacency?

I have a dataset with 13 variables and 50 observations representing the U.S. states. The variables represent the land use intensity of different agricultural industries in each state. Of those 650 ...
2
votes
1answer
58 views

Missing data imputation when all variables have some missing data?

I am working with my resident survey data (n=8356) including 59 items, most of which are ordinal variables scored from 1 to 7, and others are continuous variables (e.g., age, residency length). ...
2
votes
0answers
174 views

Interpolation versus imputation for time series on chemical profiles of water wells

So I am working with some data on water wells and time series of chemical pollutant tests on those wells. There are 10 chemicals and 10 years in the data. My goal is to do some clustering on the wells ...
2
votes
4answers
86 views

imputation FOR random forests

I was wondering what imputation method you would recommend for data to be fed into a random forest model for a classification problem. If you google for "imputation for random forests", you get a lot ...
2
votes
0answers
43 views

What value to impute for informative NA values in R without misleading model

I'm building a model (random forest) in R to predict a rare event (scoring a goal in soccer). I have event-level data, which provides a log of all the actions (pass, tackle, foul, save, shot, goal) ...
2
votes
0answers
185 views

Partial imputation of missing dates

I'm working with dataframes (one for each of 185 locations) that shows sums of occurrences for each calendar date. There are no 0 values for occurrences in the entire dataset. There are several ...
2
votes
0answers
384 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
2
votes
0answers
306 views

Overcome NA's in Random Forest and SVM?

I am using clinical data for prediction purposes with SVM and RF. Two of my columns are as following: ...
2
votes
0answers
644 views

Imputing missing values and SVD

Similar questions have been asked a lot of times but I have not found an answer that gives an intuitive explanation as to why this works. For reference I have read the answers here and here. As I ...
2
votes
0answers
77 views

To impute or not - community consensus for reporting accuracy of an imputed model

I have a model generated using an imputed data set with imputation accuracy of 75%. If the model using imputed data has an accuracy of 80% What would be the community consensus to report the ...
2
votes
0answers
80 views

Impute missing population values in Census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
2
votes
1answer
44 views
2
votes
0answers
871 views

pooling over imputation with LASSO in R

I've been trying to find an answer to my question, with no success. It could be I'm just not looking for it in the right way (I'm also learning about imputation and lasso as part of the process). So I ...
2
votes
0answers
553 views

How to interpret OOBerror while doing data imputation with missForest

I am doing data imputation with the missForest function (in the missForest package in R) and everything seems to work just fine. The function is easy to use and, at first sight, imputed data look ...
2
votes
0answers
60 views

Imputing missing counts in a time series (with variance?)

We're using a sonar to enumerate migrating fish. The date & time are recorded with each event that a fish passes the sonar station. We've noted two very strong time-varying components to the ...
2
votes
0answers
881 views

Imputing categorical 'string' data for missing values in python

I'm working on binary classifier model with 30% N/A values aka missing fields. I have one continuous feature and two categorical 'string' features missing in my data. For a continuous feature, I've ...
2
votes
0answers
501 views

Feature selection and subsetting when most features are missing many data points

I have a data set for which I would like to build a predictive model, and it contains quite a few features where most of the data is missing. The feautres are also highly correlated pair-wise and I ...
2
votes
0answers
540 views

Missing data MAR not MAR - multiple imputation

I am preparing my data for CFA and multiple regression. I want to see how different types of parental involvement influence children's motivation and academic results. I have a sample size of 5000 ...
2
votes
0answers
3k views

Imputation in monthly univariate time series

I have a time series of the amount of apples sold in a specific Region. The time series include monthly values of 10 years (2006-2016). However two months are missing (February 2009 and July 2014). ...
2
votes
0answers
237 views

How do I impute missing values of a dataset with little pairwise correction using regression

I have a dataset with 10 independent variables and one response variable. They are all in integer types. V1 - V10 are numeric values rounded to closest integer values which are always in range of [1, ...
2
votes
1answer
29 views

Imputation for Industrial Survey

What do you think the best imputation method is for industrial surveys? Methodology reports usually mention historical background, administrative data or hot-deck, but according to the theory, the ...
2
votes
0answers
85 views

Method for predicting price based on Geographical market, Product, and Company

I have a dataset which tracks the prices of 21 products, charged by 24 companies, in 150 different cities across the globe. However, the data set has missing values--that is, I might have Company X's ...
2
votes
0answers
43 views

Multiple imputation for missing data

I'm using multiple imputation in my mediation analysis and was wondering if the variables that I use for the imputation have to precede the study variables? For example if my mediator is self-esteem ...
2
votes
0answers
537 views

Imputation introduces negative values when using imputePCA() from the missMDA package in R?

I am testing out various imputation methods on my data and would like to use imputePCA. It imputes the missing values with no error messages, but when I check the completeObs matrix some of the ...

1
2 3 4 5