Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
0
votes
0answers
43 views

Do I impute missing values with the response?

I have a dataset with missing values in both predictors and the response. As far as I know, the data are missing not at random, so I cannot simply use listwise deletion. Instead, I employed the EM ...
1
vote
0answers
55 views

Hot Deck Imputation

I am currently facing a set of data with missing values. I would like to impute these values with a Hot Deck. I have read upon the Hot Deck methods and decided to use Nearest Neigbour Hot Deck (NNHD) ...
1
vote
1answer
40 views

What to assume if exact lower limit data values are not available

I have some data - about 100 values, maximum value 100. However, values below 10 are just written as <10, exact values for these are not available. How can I impute these data points. What value ...
4
votes
1answer
213 views

When performing imputation on categorical variables, does the data lose meaning?

Suppose I have a data set with several variables where one of my variables is categorical. For instance, a rating from 1 to 10. Suppose it has missing values. I want to impute this data via ...
3
votes
2answers
61 views

Dataset having about 1% missing fraction, missing completely at random. How would I address the missingness of the dataset?

I have a dataset comprising of 1% missing fraction where data is missing completely at random. How would I address the missingness of the dataset with a multiple imputation technique? For example, is ...
1
vote
0answers
113 views

Are my data a good candidate for EM imputation followed by exploratory factor analysis?

I am doing Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. I have missing data that prevent me getting factor scores, so I am imputing data. I am using ...
0
votes
0answers
24 views

Does missing completely at random and imputing the mean together affect the bias or efficiency of the OLS estimator?

Suppose I am given a response variable $y$ and two predictors $A$, $B$. The correlations between these variables are given by $\rho_{AB} = 0.4395$, $\rho_{Ay} = 0.3141$, and $\rho_{By} = 0.9587$. ...
1
vote
0answers
34 views

Filling in Missing Data for Biological Experiment

I am trying to implement a semi-supervised learning model with biological data. In my case, I'm using features from DNA. I have a number of experiments each with many observations. Each observation ...
0
votes
1answer
44 views

Dataset that has approximately 1% missing values completely at random?

I have a dataset with approximately 1% of values missing completely at random. I have thought about using the Multiple Imputation technique but I am not sure if this would be the best solution. ...
1
vote
0answers
11 views

Doing multi-value imputation to maintain the same distribution

I currently learning value imputation. The popular methods that I've seen such as mean, median, arbitrary value etc, impute all missing values with a single calculated value. Each of these methods can ...
0
votes
1answer
353 views

Imputation: why impute missing test set values from train set?

I've consulted other posts here regarding imputation and the recommendation seems to be to not impute on unpartitioned data. Data should be partitioned and missing values in both train and test set ...
3
votes
1answer
115 views

Why is a sample covariance not even semi-positive definite with missing data?

I am trying to estimate a sample covariance when I have less observations $n$ than variables $p$ ($n<p$). This will serve later on as basis for a shrinkage estimator. We know (see this post) that ...
1
vote
0answers
75 views

Imputation of a single long period of missing values in a multivariate time series

I am working with a multivariate time series (with daily data) of approximately 40 years. It consists of the data of two stations (dataset A & dataset B) measuring both the river level and the ...
0
votes
1answer
38 views

Baseline carried forward missing data assumptions

Researchers assessed the effectiveness of a range of weight management programmes for weight loss. A randomised controlled trial study design, incorporating eight treatment arms, was used. Each ...
4
votes
0answers
2k views

How to implement MICE in data imputation using sklearn IterativeImputer? [closed]

I'm interested in learning how to implement MICE in imputing missing values in my datasets. I'd like to use sklearn IterativeImputer for the following reason (source from sklearn docs): Our ...
1
vote
0answers
95 views

Multiple imputation for count data

Date, Transect, Year, Species, Tr.Length, Month, numb.on.transect I am working with historical sea-survey data.... boat transects counting birds on water. only there are some transects ...
1
vote
0answers
26 views

Missing Data imputation on one continuos column which depends on another feature and which does make sense only when such feature is positive

For each row (open contract) of my dataset, I have got a certain number of orders. I have created some features related to such orders; let's take for instance the average and the std deviation of the ...
1
vote
0answers
17 views

Correcting for known imputation bias

I have a survey sample which includes income values by type of income for a significant number of high-income households. Some of the income data is measured, i.e. present in the responses of the ...
1
vote
0answers
298 views

How to impute a categorical variable with MICE but prevent it from taking some values?

I have a categorical variable, var1 , that can take on values of W, B, A, M, N or P. There are some NAs that I want to impute using the mice package in R, but I know that the missing values cannot be "...
2
votes
0answers
19 views

Imputation that takes into account both relationships among variables and spatial adjacency?

I have a dataset with 13 variables and 50 observations representing the U.S. states. The variables represent the land use intensity of different agricultural industries in each state. Of those 650 ...
3
votes
1answer
210 views

How should I calculate a confidence interval for a gap-filled data set?

I have a data set of half hourly CO2 and CH4 fluxes (emission/uptake) from a landscape along with a set of environmental variables (Photon flux density (PPFD), temperature etc.). I'm using Neural ...
2
votes
1answer
19 views

Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?

I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
1
vote
1answer
47 views

How can I adjust for negative eigenvalues?

I wish to run a path analysis from a pooled correlation matrix that I have imputed using the maximum-likelihood procedure. There was considerable missing data. The resulting correlation matrix is: <...
0
votes
1answer
123 views

Correlation between two matrices and linear regression

I have two matrices with numeric data and a lot of missing values. The missing values are not completely coherent in both matrices. My aim is to fill the missing values so I can perform further ...
0
votes
0answers
22 views

Longitudinal analysis with high level of attrition

I have an initial dataset of 500 individuals at Time 1; 350 at Time 2; 150 at Time 3; 150 at Time 4. It's a bit of a headscratcher for me to figure out how to deal with this high level of loss to ...
1
vote
0answers
26 views

How to deal with features that are only possible for some sample in Tree based algorithms?

If I have a data set where, for instance, I am predicting something about Bank customers. Some customers have mortgages and some don't. For those with mortgages I want to include how long until ...
1
vote
0answers
143 views

Interpolate/Impute Time Series Data from another Time Series

I have a dataset of multiple lakes with water level elevations through time. The observations are not regularly spaced and have many large gaps. Further, some of the older observations may be of lower ...
0
votes
0answers
89 views

Item/Question left off Likert Scale. What can I do about “missing” responses?

I am using a certain 7 item Likert scale in my research and have collected my data - however I've now realised that I left one of the items (Item 7) off my initial survey. I have since collected all ...
0
votes
1answer
45 views

How to impute missing values not at random?

My data consists of 202 cases, each stand for a single interview. The variables reflect the interviewers' and interviewees' behaviours during four different parts of the interview: p1, g1, pA, gA. in ...
1
vote
1answer
164 views

EM Imputation and missing age data

I've completed an EM imputation to replace a small amount of missing data. One of the missing data was within my age variable (age range of 20 - 54). My question is... the new score that has been ...
1
vote
1answer
30 views

Any suggestion for imputing missing values in the following case?

I am doing a 0-1 classification problem. One of the features has a really high missing ratio(over 95%). So usually I will drop it. But for the rest 5%, almost all corresponding dependent variable is ...
4
votes
0answers
70 views

How much missing data can you replace with data imputation? [duplicate]

I have a data set with 40 participants that underwent 20 different tests. For most of these tests all the data is there. For 2 of the 20 tests, 2 out of the 40 values are missing. However, for 5 of ...
1
vote
3answers
66 views

Best approach for dealing with continuous predictors with missing data in random forests

I was thinking about a problem I'm facing: I have wage data that I want to add to my model, but it's incomplete (data for about 70% of my observations). So, I was ...
2
votes
1answer
73 views

Missing data imputation when all variables have some missing data?

I am working with my resident survey data (n=8356) including 59 items, most of which are ordinal variables scored from 1 to 7, and others are continuous variables (e.g., age, residency length). ...
1
vote
2answers
42 views

Pre-processing - Filling missing values in supervised learning across test/train datasets

I have 2 datasets, testdf and traindf, they both contain missing values in some features. When filling in the missing values ...
1
vote
3answers
1k views

Binary Classification in Imbalanced Data; Oversampling and Imputation

Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle ...
0
votes
1answer
850 views

Pooling F-Values in Multiple Regression in a Multiply Imputed Database

When working with a dataset created via multiple imputation, SPSS pools some values but not others. For example, in multiple regression, I can get coefficients, t-tests for the coefficients, t-values ...
0
votes
1answer
57 views

'runif imputation' in R with mice package

I would like to do some imputation in a situation like this: Knowing that a value is missing is highly informative itself: if a certain variable has a missing value, it must be between zero and some ...
1
vote
0answers
34 views

Multivariate linear regression with missing points without imputation

I am currently trying to perform a vanilla multivariate linear regression- imagine the equations are $y_1 = a_0x_{0,1} + a_1x_{1,1}... a_nx_{n-1,1}$ $y_2 = a_0x_{0,2} + a_1x_{1,2}... a_nx_{n-1,2}$ .....
3
votes
1answer
410 views

What is the difference between Imputation and Prediction?

Background I am currently working on a project that has me stumped about the difference between imputing a "missing value" and predicting an "unknown value". So far I would understand imputation as ...
1
vote
0answers
110 views

Using residuals as noise and EWMA to fill in missing values?

For some time series data on booking transactions, I am attempting to use EWMA (exponential weighted moving average) + randomized residuals to fill in the missing data. That is, after calculating the ...
2
votes
0answers
259 views

Interpolation versus imputation for time series on chemical profiles of water wells

So I am working with some data on water wells and time series of chemical pollutant tests on those wells. There are 10 chemicals and 10 years in the data. My goal is to do some clustering on the wells ...
0
votes
1answer
407 views

Does MICE work with 100% correlated missing values?

I have a dataset with missing values which I would like to impute by using Multiple Imputation by Chained Equations (MICE). The important characteristics of the dataset is that, for the columns ...
2
votes
4answers
93 views

imputation FOR random forests

I was wondering what imputation method you would recommend for data to be fed into a random forest model for a classification problem. If you google for "imputation for random forests", you get a lot ...
1
vote
1answer
173 views

Does one impute auxiliary variables (multiple imputation)

Auxiliary variables may have themselves missing values. According to the following website, one (can) include(s) auxiliary variables also as variables to be imputed. Is this common practice? https://...
1
vote
0answers
32 views

Regression when dependent and independent variables come from different datasets

I am trying to figure our the most robust way to combine two different sets and run a regression. The first dataset gives me an outcome value for each of several categorical treatment variables, each ...
1
vote
1answer
46 views

MissForest for SurveyData

Hello fellow data scientist, I currently reading the paper by Stekhoven & Brühlmann about MissForest. I was wondering how to deal with variables that are restricted by domain knowlege. I.e. no ...
2
votes
4answers
455 views

Are there any clustering algorithms that do not exclude/impute missing data?

From my understanding, clustering algorithms require complete data. Based on this, if there are missing values in my dataset I have two options: Impute missing information using some sort of ...
1
vote
0answers
92 views

Pool redundancy analysis results from multiple imputations of missing data

I have a dataset that includes missing values, and I would like to carry out redundancy analysis using multiple imputation to fill in the missing values. So far, I have successfully created multiple ...
0
votes
1answer
47 views

Dealing with measurements falling outside of the theoretical range/boundaries of the data

Imagine I am measuring a bounded variable (with a maximum possible value above which the data doesn't make sense) and I end up with the following dataset with my measurements and measurement errors as ...

1 2
3
4 5
12