Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
31 votes
3 answers
37k views

R caret and NAs

I very much prefer caret for its parameter tuning ability and uniform interface, but I have observed that it always requires complete datasets (i. e. without NAs) even if the applied "naked" model ...
user avatar
  • 681
28 votes
5 answers
64k views

Imputation of missing values for PCA

I used the prcomp() function to perform a PCA (principal component analysis) in R. However, there's a bug in that function such that the ...
user avatar
24 votes
6 answers
5k views

What are the disadvantages of using mean for missing values?

I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section. ...
user avatar
  • 241
21 votes
1 answer
38k views

XGBoost can handle missing data in the forecasting phase

Recently I have reviewed XGBoost algorithm and I have noticed that this algorithm can handle missing data (without requiring imputation) in the training phase. I was wondering if XGboost can handle ...
user avatar
21 votes
3 answers
3k views

How to combine confidence intervals for a variance component of a mixed-effects model when using multiple imputation

The logic of multiple imputation (MI) is to impute the missing values not once but several (typically M=5) times, resulting in M completed datasets. The M completed datasets are then analyzed with ...
user avatar
  • 331
18 votes
5 answers
6k views

A 6th response option ("I don't know") was added to a 5-point Likert scale. Is the data lost?

I need a little bit of help salvaging the data from a questionnaire. One of my colleagues applied a questionnaire, but inadvertently, instead of using the original 5-point Likert scale (strongly ...
user avatar
18 votes
3 answers
4k views

Methods to work around the problem of missing data in machine learning

Virtually any database we want to make predictions using machine learning algorithms will find missing values ​​for some of the characteristics. There are several approaches to address this problem, ...
user avatar
  • 165
16 votes
2 answers
14k views

How to fill in missing data in time series?

I have a large set of pollution data that has been recorded every 10 minutes for the course of 2 years, however there are a number of gaps in the data (including some that go for a few weeks at a time)...
user avatar
  • 163
16 votes
1 answer
17k views

How do the number of imputations & the maximum iterations affect accuracy in multiple imputation?

The help page for MICE defines the function as: ...
user avatar
  • 335
16 votes
5 answers
49k views

KNN imputation R packages

I am looking for a KNN imputation package. I have been looking at imputation package (http://cran.r-project.org/web/packages/imputation/imputation.pdf) but for some reason the KNN impute function (...
user avatar
  • 2,112
16 votes
2 answers
11k views

Imputation of missing data before or after centering and scaling?

I want to impute missing values of a dataset for machine learning (knn imputation). Is it better to scale and center the data before the imputation or afterwards? Since the scaling and centering ...
user avatar
  • 303
15 votes
3 answers
12k views

How to handle with missing values in order to prepare data for feature selection with LASSO?

My situation: small sample size: 116 binary outcome variable long list of explanatory variables: 44 explanatory variables did not come from the top of my head; their choice was based on the ...
user avatar
  • 375
15 votes
1 answer
821 views

Pooling calibration plots after multiple imputation

I would like advice on pooling the calibration plots/statistics after multiple imputation. In the setting of developing statistical models in order to predict a future event (e.g. using data from ...
user avatar
  • 2,584
14 votes
2 answers
15k views

Using Kalman filters to impute Missing Values in Time Series

I am interested in how Kalman Filters can be used to impute missing values in Time Series Data. Is it also applicable if some consecutive time points are missing? I cannot find much on this topic. Any ...
user avatar
  • 233
13 votes
2 answers
633 views

using neighbor information in imputing data or find off-data (in R)

I have dataset with assumption that nearest neighbors are best predictors. Just a perfect example of two-way gradient visualized- Suppose we have case where few values are missing, we can easily ...
user avatar
  • 3,513
12 votes
5 answers
10k views

How to perform imputation of values in very large number of data points?

I have a very large dataset and about 5% random values are missing. These variables are correlated with each other. The following example R dataset is just a toy example with dummy correlated data. <...
user avatar
  • 2,098
12 votes
3 answers
12k views

How to perform SVD to impute missing values, a concrete example

I have read the great comments regarding how to deal with missing values before applying SVD, but I would like to know how it works with a simple example: ...
user avatar
  • 233
11 votes
2 answers
62k views

Which is better, replacement by mean and replacement by median?

I'm doing a project that involves replacing missing values in a set of data (first time doing this). This involves using two methods replacement by mean and ...
user avatar
  • 111
11 votes
1 answer
7k views

Multiple imputation for missing count data in a time series from a panel study

I am trying to tackle a problem which deals with the imputation of missing data from a panel data study(Not sure if I am using 'panel data study' correctly - as I learned it today.) I have total death ...
user avatar
  • 295
11 votes
1 answer
1k views

Imputation of a censored variable

I have a medical dataset with approx 200 variables. One of the variables is a bio-marker (concentration of a particular enzyme). It's distribution is right skew, and the problem is that values above a ...
user avatar
  • 49.5k
11 votes
3 answers
5k views

Combining two time-series by averaging the data points

I would like to combine the forecasted and backcasted (viz. the predicted past values) of a time-series data set into one time-series by minimizing the Mean Squared Prediction Error. Say I have time ...
user avatar
  • 295
10 votes
2 answers
6k views

Using multiple imputation for Cox proportional hazards, then validating with rms package?

I've been researching the mice package, and I haven't yet discovered a way to use the multiple imputations to make a Cox model, then validate that model with the rms package's ...
user avatar
  • 817
10 votes
2 answers
19k views

Imputation with Random Forests

I have two questions on using random forest (specifically randomForest in R) for missing value imputation (in the predictor space). 1) How does the imputation algorithm work - specifically how and ...
user avatar
  • 7,660
10 votes
1 answer
2k views

Why is this multiple imputation low quality?

Consider the following R code: ...
user avatar
  • 185
10 votes
3 answers
1k views

What is the advantage of imputation over building multiple models in regression?

I wonder if someone could provide some insight into if an why imputation for missing data is better than simply building different models for cases with missing data. Especially in the case of [...
user avatar
  • 5,019
10 votes
1 answer
264 views

Does imputation introduce unacceptable bias?

I have recently come to know about imputation techniques, which, in short, "guess" realistic values with which to replace missing values in a dataset. My big issue with this is that we are ...
user avatar
9 votes
4 answers
1k views

Can I delete missing data?

I have a dataset (4,898 X 17,000) that follows 4898 mothers, fathers, and their children over a period of 15 years. The interviews have been conducted at baseline (when the child was born), year-1, ...
user avatar
9 votes
1 answer
13k views

How does the mice imputation function work?

I was wondering if anyone had experience using the mice function, as described in mice: Multivariate Imputation by Chained Equations in R (JSS 2011 45(3))? I have a dataset with a number of variables, ...
user avatar
  • 1,087
9 votes
2 answers
4k views

Missing rates and multiple imputation

Is there a limit which is the least acceptable when using multiple imputation (MI)? For example can I use MI if the missing values in a variable are the 20% of the cases while and other variables ...
user avatar
  • 297
9 votes
2 answers
232 views

Imputation to account for systematic error in survey responses

I have a large survey in which students were asked, among other things, their mother's level of education. Some skipped it, and some answered wrongly. I know this, because there a sub-sample of the ...
user avatar
8 votes
4 answers
4k views

Imputing missing observation in multivariate time series

Suppose I have a dataframe consisting of six time series. In this dataframe, some observations are missing, meaning at some timepoints all time series contain a NA-value. In R, one possible imputation ...
user avatar
  • 1,560
8 votes
1 answer
2k views

Can I replace NAs based on response variable?

My data consists of 1 response variable 'Age' and 1 feature (beta). The feature contains some missing values (NA) so I want to replace them. I've been replacing them with the median of the feature. ...
user avatar
8 votes
1 answer
2k views

How to use restricted cubic splines with the R mice imputation package

I am wondering how to integrate restricted cubic splines (such as in the rms package) in the imputation models within R mice imputation package. Context: I am doing biomedical research and have ...
user avatar
  • 2,584
8 votes
1 answer
8k views

Should data be normalized before or after imputation of missing data?

I am working on a metabolomics data set of 81 samples x 407 variables with ~17% missing data. I would like to compare a number of imputation methods to see which is best for my data. Is there a ...
user avatar
  • 145
8 votes
1 answer
666 views

Missing data at random

How does one tell if a dataset is missing data at random? I've been reading up on how to impute missing values, and was wondering what techniques can be used to tell if data is really missing at ...
user avatar
8 votes
2 answers
17k views

How to combine multiple imputed datasets?

I need a single imputed dataset (e.g. to create a country group dummy from the imputed country per capita income data). R offers packages package for creating multiple imputed data (e.g. Amelia) and ...
user avatar
8 votes
2 answers
670 views

Imputing a missing variable based on common variables with another data set

I have 2 data sets: $A$ and $B$. The variables are common to both data sets with the exception of two, which are both missing in A. Let's call those two additional variables: $b_1$ and $b_2$. We ...
user avatar
8 votes
1 answer
1k views

Guassian Process for Data Imputation

I recently came across Gaussian Processes in Gelman et al. (2013), and I am trying to learn more about their potential application for use in imputing time series data. The data of interest is a ...
user avatar
7 votes
1 answer
5k views

Hot deck imputation, ''it preserves the distribution of the item values'', how can that be?

I read in this link, under section 2, first paragraph about hot deck that ''it preserves the distribution of item values''. I do not understand that, if one and the same donor is used for a lot of ...
user avatar
7 votes
2 answers
19k views

Fast missing data imputation in R for big data that is more sophisticated than simply imputing the means?

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of ...
user avatar
  • 1,269
7 votes
1 answer
274 views

Dealing with missing data due to variable not being measured over initial period of a study

I was recently consulting a researcher in the following situation. Context: data were collected over four years at around 50 participants per year (participants had a specific diagnosed clinical ...
user avatar
7 votes
3 answers
4k views

Uncertainty in random forest imputations from R missForest package

I am in the process of imputing missing values for my data set that contains approximately 20 variables and 3,000 observations. Most of the missing data values are contained in 2 of the variables (one ...
user avatar
  • 506
7 votes
1 answer
418 views

How can I apply a Pareto tail to a truncated distribution?

Many income surveys (especially older ones) truncate key variables, such as household income, at some arbitrary point, to protect confidentiality. This point changes over time. This reduces inequality ...
user avatar
  • 71
7 votes
1 answer
586 views

Is Structurally Missing Data a subset of Missing at Random Data?

I'm quite familiar with MCAR, MAR and MNAR (NMAR) data but I have just come across a new (for me) term: Structurally Missing Data (SMD). According to this page, Structurally missing data is data ...
user avatar
  • 183
7 votes
0 answers
998 views

Missing value imputation in huge dataset

I have a huge data (4M x 17) that has missing values. Two columns are categorical, rest all are numerical. Given the huge amount of data, running any imputation method runs forever. What should I do? ...
user avatar
7 votes
0 answers
890 views

Canonical correlation analysis on a MICE data set

I am looking to do a canonical correlations analysis (CCA) in R, using the CCA package, on a multiply imputed dataset (obtained from the mice package). I know that ...
user avatar
  • 681
6 votes
1 answer
13k views

Imputation by regression in R [closed]

Say I have below example data, where rows are observations and columns are variables, and NAs stand for missing values. ...
user avatar
  • 1,269
6 votes
2 answers
4k views

Should I use missing value using imputation or listwise or pairwise deletion methods?

I have 60,000 data and around 45% of them is missing and the missing values are random. Can I simply use listwise or pairwise deletion or do I have to use imputation? If imputation is recommended ...
user avatar
6 votes
1 answer
4k views

Imputation methods for time series data

I have some network data which measures the noise levels in a cellular network. On a typical mast there are generally 3 sectors or antennas which point in different directions. Within one of these ...
user avatar
  • 549
6 votes
2 answers
234 views

At what point in analysis do you perform imputation for missing variables?

There is a dataset with 30 variables and over 5 million observations. We plan to use a subsample of the data for analysis. Around .02 - 2.5% of EACH variable are missing. I plan imputation in Stata ...
user avatar
  • 211

1
2 3 4 5
13