Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
0
votes
1answer
1k views

Imputation during pre-processing in data mining

I'm trying to comprehend the most appropriate way to utilize machine learning imputation (e.g., KNN) as a pre-processing step in data mining, and I've run into a few questions that don't have clear ...
5
votes
3answers
4k views

Uncertainty in random forest imputations from R missForest package

I am in the process of imputing missing values for my data set that contains approximately 20 variables and 3,000 observations. Most of the missing data values are contained in 2 of the variables (one ...
1
vote
1answer
14k views

Data imputation with preProcess in caret returns less observations than expected

I wonder why preProcess function from R's caret package used for imputation of dataset's missing values returns less observations than in original dataset? For example: ...
0
votes
0answers
42 views

filling missing data with other than mean values [duplicate]

What all options are available for filling missing data ? One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy. So how do we deal with ...
2
votes
0answers
839 views

Filling missing values for categorical feature [closed]

In case of continuous variable, the missing values can be filled by the mean of other values, but in case of categorical feature how should we fill in the missing values?
6
votes
1answer
233 views

Use Available Pairs Method for Missing Data in OLS

I have renewed interest in handling missing covariate data in OLS using the pairwise covariance matrix estimator, i.e., using all available pairs of variables in computing variances and covariances. ...
0
votes
1answer
143 views

Removing rows with missing values vs. variables

I am dealing with a real dataset with a large amount of missing values. The easiest and fastest way to deal with the missing values is to get rid of them by running ...
1
vote
0answers
69 views

Data imputation for a dataset where all values are 1 or N/A

I have a dataset which contains relations between jobs and skills required for these jobs. It is a matrix with value 1 if a skill is required for a job, and N/A otherwise. N/A instead of zero because ...
2
votes
0answers
84 views

Group wise imputations allowed?

What is the correct way of imputing missing values? We are trying to predict the label of persons belonging to group A or B. The alternative hypothesis is that there are differences in parameters for ...
0
votes
0answers
16 views

How do I distribute answers lacking geo information from a poll?

My data table looks like this: Region Answer1 Answer2 NaN 20 40 Region1 15 17 Region2 18 19 ... So is it possible to distribute answers for which region is not ...
2
votes
1answer
389 views

MICE: what does returned df mean?

In MICE, the object returned by pool() has a component, df, which is included in the summary of the pooled analysis. In my analysis I have about 55,000 cases, but the returned df is higher for most ...
4
votes
5answers
13k views

How do we decide on how to fill missing values in data?

I have a data set with NA values in many predictor variables. How do we impute the best values ? I have 302 variables in total. Out of them 236 belong to some abstract category, 37 to other, 9 to ...
1
vote
1answer
793 views

Impute with the Mean or Median? Instrumental Variables

I am using instrumental variables and I have missing data. In r, I don't believe you can use the MICE package with the AER package. Therefore, I am going to impute with either the mean or median ...
1
vote
1answer
618 views

Imputation and linear regression analysis paradox

Missing values, especially in small datasets, can introduce biases into your model. There are several data imputation methods (MICE, Amelia II), which use EM algorithms to "fill in" the missing values....
1
vote
0answers
60 views

Advice on imputation of multiple time series

Background In the first year of the study 60 streams had temperature data loggers installed (temperature measured every 30 seconds). The second year only 30 of these same streams had data loggers. ...
1
vote
1answer
97 views

How to handle data with 2 variables that have same missingness pattern?

I've not had much academic coursework on imputation, and I can't find anything online or in any texts regarding how one could handle missing data where there are two (or more, possibly?) variables ...
2
votes
1answer
652 views

Regression analysis with non-integer event rates

I am working as part of a team on a large dataset which has been subject to imputation analysis. One of my colleagues has pointed out that the when carrying out regressions that can provide odds ...
0
votes
1answer
938 views

Imputation in R: How to impute univariate data in R?

I am trying classification(2 classes) using Random Forest. Classes are - Red, Green. My dataset contains 1 numeric attributes(called X), and 51 binary attributes to classify a document into red and ...
1
vote
1answer
109 views

Independent variable has a known non-causal relationship with the dependent variable; is it still okay to regress?

To further elaborate on my question, assume that I have a time series dataset of Tax X and Tax Y, where in Tax X is paid by 100% of the sample while Tax Y is paid by 75%. Both taxes differ with ...
2
votes
0answers
615 views

Poisson Regression - missing data, imputation, distribution of fitted values

I'm working on a school project concerning Poisson regresion. I'm trying to build a model for number of cars in household base on American Community Survey. Among explanatory variables are value of ...
1
vote
0answers
85 views

What is the computational complexity of mean imputation?

I'm trying to figure out what the complexity of mean imputation is (in $\mathcal{O}$ notation). Assuming a data set with $m$ observations and $n$ features, I would say the complexity is $\mathcal{O}(...
1
vote
0answers
18 views

Estimating the opinion of a user by looking at opinions of other users

First of all, a bit of background: i am not a statistics expert but i am an enthusiast about data analysis. I have this list of "items" and for each item i have a list of "users" and the vote that ...
3
votes
4answers
2k views

Missing data and imputation in general

Handling missing data is a bit confusing for me. My questions are: Is it better to calculate imputations than simply leave out NAs and leave it to the (appropriate) model to handle it? Is there a ...
29
votes
3answers
34k views

R caret and NAs

I very much prefer caret for its parameter tuning ability and uniform interface, but I have observed that it always requires complete datasets (i. e. without NAs) even if the applied "naked" model ...
11
votes
2answers
50k views

Which is better, replacement by mean and replacement by median?

I'm doing a project that involves replacing missing values in a set of data (first time doing this). This involves using two methods replacement by mean and ...
0
votes
1answer
85 views

Imputation Procedures

I don't completely understand how to impute in the following situation. Consider the following example. In this example we have a dataframe of students. For each student we have an IQ score and a ...
14
votes
2answers
12k views

Using Kalman filters to impute Missing Values in Time Series

I am interested in how Kalman Filters can be used to impute missing values in Time Series Data. Is it also applicable if some consecutive time points are missing? I cannot find much on this topic. Any ...
0
votes
0answers
568 views

Multiple imputation of conditional variables

I need to impute the missing values of a dataset of medical data in which several variables only make sense if another variable has a specific value. In the questionnaire the data come from they were ...
1
vote
0answers
168 views

Multiply imputing data, but using just one of the imputed data sets

All, I have a question about what's practical when it comes to presenting results of multiply imputed data. I'm well-versed on the difference among MCAR/MAR/MNAR and approaches to imputing the data ...
13
votes
2answers
7k views

Imputation of missing data before or after centering and scaling?

I want to impute missing values of a dataset for machine learning (knn imputation). Is it better to scale and center the data before the imputation or afterwards? Since the scaling and centering ...
2
votes
0answers
421 views

Imputing missing gaps in irregular time series

I am currently working with time series data that was sampled at irregular time intervals. There are some gaps of missing data, i.e. a bunch of subsequent observations are missing every now and then. ...
1
vote
1answer
2k views

Amelia (error if I include categorical vars)

If I include a categorical variable in amelia (e.g., p3[,4]), I am getting the following error: ...
3
votes
0answers
423 views

Multiple Imputation and Matrix Completion

It is quite common that data sets will contain missing values in them. Suppose we want to try to fill in the missing values. For this we have techniques such as single/multiple imputation and matrix ...
5
votes
0answers
273 views

How to do multiple imputation for spatial models?

I'm trying to estimate various spatial models such as spatial autoregressive regression (SAR), Spatial Durbin Model (SDM), and Spatial Error Model (SEM) but have missing data throughout my variables. ...
0
votes
0answers
2k views

R MICE imputation failing

I am really baffled about why my imputation is failing in R's Mice 2.22 package. I am attempting a very simple operation with the following data frame: ...
2
votes
0answers
89 views

Taking only a single data set from a multiple imputation?

If I use a package like R's mice to do multiple imputation, then only select the first of the resulting imputations and use that as a single imputation -- ignoring ...
5
votes
1answer
791 views

Missing data not at random - Advice needed on method

I have been developing a logistic regression model based on retrospective data from a national trauma database of head injury in the UK. The key outcome is 30 day mortality (denoted as "Survive" ...
2
votes
1answer
834 views

How to handle missing data in a small $n$ large $k$ machine learning scenario?

I have a sample size $N=130$ and $1000$ variables. I am using machine learning techniques (SVM) for analysing the data. Some variables in the dataset have values that are so huge that they must be ...
2
votes
0answers
45 views

Imputation for small cells?

I have a question about the data requirements for imputation. Specifically, is there a rule of thumb about what proportion of the data have to be non-missing for the imputation to be "valid?" I am ...
0
votes
1answer
26 views

Group variables seeing the class

I need to perform a classification model (logistic regression, pnn or neural network). I'm doing the part of data preparation in R. I have a nominal variable, that has more than 800 levels. I need to ...
2
votes
1answer
6k views

What are the pros and cons of using median imputation to handle missing value?

I have to choose between median or mean imputation to handle missing values. I feel median imputation will work better because it is a number that is already present in the data set and is less ...
2
votes
0answers
135 views

Imputation of missing values for doing PCA in R [duplicate]

I have a dataset with approximately 4000 rows and 150 columns. I want to predict the values of a single column (= target). The data is on cities (demography, social, economic, ... indicators). A lot ...
9
votes
1answer
2k views

Why is this multiple imputation low quality?

Consider the following R code: ...
2
votes
2answers
424 views

Measuring longitudinal data where individuals have missing observations

We have a longitudinal panel of X users with their online spending patterns and are trying to measure certain metrics within the panel. We have time series information about the users such as their ...
2
votes
1answer
95 views

Semantics rules? A classification challenge

Suppose we make interviews on a large number of households in which we ask, among other things, the sex and age of the individuals living in the household, and also who among these individuals is the ...
1
vote
0answers
59 views

Are pooled results from multiple imputation equivalent to a posterior mean?

I am fairly new to multiple imputation and trying to be sure I understand the approach. Say I have a data set with missing values, so I create 5 imputed data sets using multiple imputation by ...
4
votes
3answers
825 views

Data imputation for meta analysis using mice package in R

I have a data-set with 32 effect size estimates- only 11 of which report a value for the continuous moderator of interest (the samples anxiety level). A complete case analysis (restricted to the 11 ...
4
votes
2answers
1k views

Single-Imputation on Age Needed?

I am hoping someone can help me with this answer. I did a Single Imputation on my data set for age (<5% missing). My adviser asked the following "It’s strange to me to impute a demographic ...
1
vote
1answer
60 views

Data imputation for statistical analysis

Here is the situation: I have an individual level data set $X$ where each row is a person $i$ and each column denote characteristics of $i$. The problem is that my data is missing an important ...
1
vote
0answers
454 views

Random Forest for data imputation

Currently I am using Random Forest approach for Missing Values Imputation from missForest package in R. I faced the following problem: the algorithm works much longer than any other imputation ...