Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
29
votes
3answers
33k views

R caret and NAs

I very much prefer caret for its parameter tuning ability and uniform interface, but I have observed that it always requires complete datasets (i. e. without NAs) even if the applied "naked" model ...
27
votes
5answers
54k views

Imputation of missing values for PCA

I used the prcomp() function to perform a PCA (principal component analysis) in R. However, there's a bug in that function such that the ...
24
votes
6answers
3k views

What are the disadvantages of using mean for missing values?

I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section. ...
21
votes
3answers
3k views

How to combine confidence intervals for a variance component of a mixed-effects model when using multiple imputation

The logic of multiple imputation (MI) is to impute the missing values not once but several (typically M=5) times, resulting in M completed datasets. The M completed datasets are then analyzed with ...
17
votes
5answers
5k views

A 6th response option (“I don't know”) was added to a 5-point Likert scale. Is the data lost?

I need a little bit of help salvaging the data from a questionnaire. One of my colleagues applied a questionnaire, but inadvertently, instead of using the original 5-point Likert scale (strongly ...
16
votes
2answers
14k views

How to fill in missing data in time series?

I have a large set of pollution data that has been recorded every 10 minutes for the course of 2 years, however there are a number of gaps in the data (including some that go for a few weeks at a time)...
16
votes
5answers
46k views

KNN imputation R packages

I am looking for a KNN imputation package. I have been looking at imputation package (http://cran.r-project.org/web/packages/imputation/imputation.pdf) but for some reason the KNN impute function (...
16
votes
3answers
4k views

Methods to work around the problem of missing data in machine learning

Virtually any database we want to make predictions using machine learning algorithms will find missing values ​​for some of the characteristics. There are several approaches to address this problem, ...
15
votes
1answer
13k views

How do the number of imputations & the maximum iterations affect accuracy in multiple imputation?

The help page for MICE defines the function as: ...
15
votes
1answer
583 views

Pooling calibration plots after multiple imputation

I would like advice on pooling the calibration plots/statistics after multiple imputation. In the setting of developing statistical models in order to predict a future event (e.g. using data from ...
14
votes
1answer
30k views

XGBoost can handle missing data in the forecasting phase

Recently I have reviewed XGBoost algorithm and I have noticed that this algorithm can handle missing data (without requiring imputation) in the training phase. I was wondering if XGboost can handle ...
14
votes
4answers
11k views

How to handle with missing values in order to prepare data for feature selection with LASSO?

My situation: small sample size: 116 binary outcome variable long list of explanatory variables: 44 explanatory variables did not come from the top of my head; their choice was based on the ...
14
votes
2answers
12k views

Using Kalman filters to impute Missing Values in Time Series

I am interested in how Kalman Filters can be used to impute missing values in Time Series Data. Is it also applicable if some consecutive time points are missing? I cannot find much on this topic. Any ...
13
votes
2answers
529 views

using neighbor information in imputing data or find off-data (in R)

I have dataset with assumption that nearest neighbors are best predictors. Just a perfect example of two-way gradient visualized- Suppose we have case where few values are missing, we can easily ...
12
votes
5answers
10k views

How to perform imputation of values in very large number of data points?

I have a very large dataset and about 5% random values are missing. These variables are correlated with each other. The following example R dataset is just a toy example with dummy correlated data. <...
12
votes
2answers
7k views

Imputation of missing data before or after centering and scaling?

I want to impute missing values of a dataset for machine learning (knn imputation). Is it better to scale and center the data before the imputation or afterwards? Since the scaling and centering ...
10
votes
2answers
48k views

Which is better, replacement by mean and replacement by median?

I'm doing a project that involves replacing missing values in a set of data (first time doing this). This involves using two methods replacement by mean and ...
10
votes
2answers
17k views

Imputation with Random Forests

I have two questions on using random forest (specifically randomForest in R) for missing value imputation (in the predictor space). 1) How does the imputation algorithm work - specifically how and ...
10
votes
1answer
7k views

Multiple imputation for missing count data in a time series from a panel study

I am trying to tackle a problem which deals with the imputation of missing data from a panel data study(Not sure if I am using 'panel data study' correctly - as I learned it today.) I have total death ...
10
votes
3answers
9k views

How to perform SVD to impute missing values, a concrete example

I have read the great comments regarding how to deal with missing values before applying SVD, but I would like to know how it works with a simple example: ...
10
votes
3answers
1k views

What is the advantage of imputation over building multiple models in regression?

I wonder if someone could provide some insight into if an why imputation for missing data is better than simply building different models for cases with missing data. Especially in the case of [...
10
votes
1answer
1k views

Imputation of a censored variable

I have a medical dataset with approx 200 variables. One of the variables is a bio-marker (concentration of a particular enzyme). It's distribution is right skew, and the problem is that values above a ...
10
votes
3answers
4k views

Combining two time-series by averaging the data points

I would like to combine the forecasted and backcasted (viz. the predicted past values) of a time-series data set into one time-series by minimizing the Mean Squared Prediction Error. Say I have time ...
9
votes
2answers
5k views

Using multiple imputation for Cox proportional hazards, then validating with rms package?

I've been researching the mice package, and I haven't yet discovered a way to use the multiple imputations to make a Cox model, then validate that model with the rms package's ...
9
votes
1answer
2k views

Why is this multiple imputation low quality?

Consider the following R code: ...
9
votes
1answer
13k views

How does the mice imputation function work?

I was wondering if anyone had experience using the mice function, as described in mice: Multivariate Imputation by Chained Equations in R (JSS 2011 45(3))? I have a dataset with a number of variables, ...
9
votes
2answers
222 views

Imputation to account for systematic error in survey responses

I have a large survey in which students were asked, among other things, their mother's level of education. Some skipped it, and some answered wrongly. I know this, because there a sub-sample of the ...
9
votes
1answer
120 views

Does imputation introduce unacceptable bias?

I have recently come to know about imputation techniques, which, in short, "guess" realistic values with which to replace missing values in a dataset. My big issue with this is that we are ...
8
votes
1answer
1k views

How to use restricted cubic splines with the R mice imputation package

I am wondering how to integrate restricted cubic splines (such as in the rms package) in the imputation models within R mice imputation package. Context: I am doing biomedical research and have ...
8
votes
2answers
16k views

How to combine multiple imputed datasets?

I need a single imputed dataset (e.g. to create a country group dummy from the imputed country per capita income data). R offers packages package for creating multiple imputed data (e.g. Amelia) and ...
8
votes
2answers
4k views

Missing rates and multiple imputation

Is there a limit which is the least acceptable when using multiple imputation (MI)? For example can I use MI if the missing values in a variable are the 20% of the cases while and other variables ...
8
votes
1answer
959 views

Guassian Process for Data Imputation

I recently came across Gaussian Processes in Gelman et al. (2013), and I am trying to learn more about their potential application for use in imputing time series data. The data of interest is a ...
7
votes
1answer
1k views

Can I replace NAs based on response variable?

My data consists of 1 response variable 'Age' and 1 feature (beta). The feature contains some missing values (NA) so I want to replace them. I've been replacing them with the median of the feature. ...
7
votes
2answers
18k views

Fast missing data imputation in R for big data that is more sophisticated than simply imputing the means?

I need a package for missing data imputation in R. But since I am dealing with big data, the number of missing data entries can also be high. The packages which impute using mean or median are of ...
7
votes
1answer
563 views

Missing data at random

How does one tell if a dataset is missing data at random? I've been reading up on how to impute missing values, and was wondering what techniques can be used to tell if data is really missing at ...
7
votes
1answer
267 views

Dealing with missing data due to variable not being measured over initial period of a study

I was recently consulting a researcher in the following situation. Context: data were collected over four years at around 50 participants per year (participants had a specific diagnosed clinical ...
7
votes
1answer
402 views

How can I apply a Pareto tail to a truncated distribution?

Many income surveys (especially older ones) truncate key variables, such as household income, at some arbitrary point, to protect confidentiality. This point changes over time. This reduces inequality ...
7
votes
1answer
7k views

Should data be normalized before or after imputation of missing data?

I am working on a metabolomics data set of 81 samples x 407 variables with ~17% missing data. I would like to compare a number of imputation methods to see which is best for my data. Is there a ...
7
votes
2answers
607 views

Imputing a missing variable based on common variables with another data set

I have 2 data sets: $A$ and $B$. The variables are common to both data sets with the exception of two, which are both missing in A. Let's call those two additional variables: $b_1$ and $b_2$. We ...
7
votes
1answer
360 views

Is Structurally Missing Data a subset of Missing at Random Data?

I'm quite familiar with MCAR, MAR and MNAR (NMAR) data but I have just come across a new (for me) term: Structurally Missing Data (SMD). According to this page, Structurally missing data is data ...
6
votes
4answers
3k views

Imputing missing observation in multivariate time series

Suppose I have a dataframe consisting of six time series. In this dataframe, some observations are missing, meaning at some timepoints all time series contain a NA-value. In R, one possible imputation ...
6
votes
1answer
11k views

Imputation by regression in R [closed]

Say I have below example data, where rows are observations and columns are variables, and NAs stand for missing values. ...
6
votes
2answers
4k views

Should I use missing value using imputation or listwise or pairwise deletion methods?

I have 60,000 data and around 45% of them is missing and the missing values are random. Can I simply use listwise or pairwise deletion or do I have to use imputation? If imputation is recommended ...
6
votes
1answer
3k views

Imputation methods for time series data

I have some network data which measures the noise levels in a cellular network. On a typical mast there are generally 3 sectors or antennas which point in different directions. Within one of these ...
6
votes
1answer
7k views

What is the proper way to use rfImpute? (Imputation by Random Forest in R)

Is it right to use rfImpute to impute missing feature values on the whole data set and then use other regression/classification techniques on the new data set created? Or, is an rfImpute model ...
6
votes
1answer
164 views

Getting an average measurement based on two raters for cases where data is missing for one rater

Context: I'm investigating behaviour in a clinical study involving children. I had both parents and teachers completing questionnaires to inform an understanding of the same underlying constructs, ...
6
votes
1answer
7k views

Python packages for numerical data imputation [closed]

I am working with multivariate numerical data with a lot of missing values (so dropping all entries or columns with missing data is not an option). Is there a Python package for data imputation? I ...
6
votes
2answers
1k views

Multiple Imputation for Mixed Effects models

I stumbled accross this related question from 2010, and I wonder if there has been any progress on using multiple imputation for mixed effects models ? I prefer to use R, though Stata is also ...
6
votes
1answer
2k views

Can you perform a multiple imputation on data that is missing not at random (MNAR)?

Is there a way to identify if your data is MNAR, MAR, or MCAR? And when performing multiple imputation, should you include all predictor variables even if only 1 or 2 variables have missing values? (...
6
votes
1answer
161 views

Creating fuzzy values for binary data

I am doing a logistic regression where my dependent variable is whether or not a person owns a particular product. Among the variables in the model is an indicator of marital status, 1 for married, 0 ...

1
2 3 4 5
12