Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
4
votes
0answers
63 views

How much missing data can you replace with data imputation? [duplicate]

I have a data set with 40 participants that underwent 20 different tests. For most of these tests all the data is there. For 2 of the 20 tests, 2 out of the 40 values are missing. However, for 5 of ...
1
vote
3answers
65 views

Best approach for dealing with continuous predictors with missing data in random forests

I was thinking about a problem I'm facing: I have wage data that I want to add to my model, but it's incomplete (data for about 70% of my observations). So, I was ...
2
votes
1answer
58 views

Missing data imputation when all variables have some missing data?

I am working with my resident survey data (n=8356) including 59 items, most of which are ordinal variables scored from 1 to 7, and others are continuous variables (e.g., age, residency length). ...
1
vote
2answers
39 views

Pre-processing - Filling missing values in supervised learning across test/train datasets

I have 2 datasets, testdf and traindf, they both contain missing values in some features. When filling in the missing values ...
1
vote
3answers
767 views

Binary Classification in Imbalanced Data; Oversampling and Imputation

Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle ...
0
votes
1answer
536 views

Pooling F-Values in Multiple Regression in a Multiply Imputed Database

When working with a dataset created via multiple imputation, SPSS pools some values but not others. For example, in multiple regression, I can get coefficients, t-tests for the coefficients, t-values ...
0
votes
1answer
48 views

'runif imputation' in R with mice package

I would like to do some imputation in a situation like this: Knowing that a value is missing is highly informative itself: if a certain variable has a missing value, it must be between zero and some ...
1
vote
0answers
32 views

Multivariate linear regression with missing points without imputation

I am currently trying to perform a vanilla multivariate linear regression- imagine the equations are $y_1 = a_0x_{0,1} + a_1x_{1,1}... a_nx_{n-1,1}$ $y_2 = a_0x_{0,2} + a_1x_{1,2}... a_nx_{n-1,2}$ .....
3
votes
1answer
259 views

What is the difference between Imputation and Prediction?

Background I am currently working on a project that has me stumped about the difference between imputing a "missing value" and predicting an "unknown value". So far I would understand imputation as ...
1
vote
0answers
89 views

Using residuals as noise and EWMA to fill in missing values?

For some time series data on booking transactions, I am attempting to use EWMA (exponential weighted moving average) + randomized residuals to fill in the missing data. That is, after calculating the ...
2
votes
0answers
174 views

Interpolation versus imputation for time series on chemical profiles of water wells

So I am working with some data on water wells and time series of chemical pollutant tests on those wells. There are 10 chemicals and 10 years in the data. My goal is to do some clustering on the wells ...
0
votes
1answer
258 views

Does MICE work with 100% correlated missing values?

I have a dataset with missing values which I would like to impute by using Multiple Imputation by Chained Equations (MICE). The important characteristics of the dataset is that, for the columns ...
2
votes
4answers
86 views

imputation FOR random forests

I was wondering what imputation method you would recommend for data to be fed into a random forest model for a classification problem. If you google for "imputation for random forests", you get a lot ...
1
vote
1answer
99 views

Does one impute auxiliary variables (multiple imputation)

Auxiliary variables may have themselves missing values. According to the following website, one (can) include(s) auxiliary variables also as variables to be imputed. Is this common practice? https://...
1
vote
0answers
29 views

Regression when dependent and independent variables come from different datasets

I am trying to figure our the most robust way to combine two different sets and run a regression. The first dataset gives me an outcome value for each of several categorical treatment variables, each ...
0
votes
1answer
35 views

MissForest for SurveyData

Hello fellow data scientist, I currently reading the paper by Stekhoven & Brühlmann about MissForest. I was wondering how to deal with variables that are restricted by domain knowlege. I.e. no ...
1
vote
4answers
308 views

Are there any clustering algorithms that do not exclude/impute missing data?

From my understanding, clustering algorithms require complete data. Based on this, if there are missing values in my dataset I have two options: Impute missing information using some sort of ...
1
vote
0answers
73 views

Pool redundancy analysis results from multiple imputations of missing data

I have a dataset that includes missing values, and I would like to carry out redundancy analysis using multiple imputation to fill in the missing values. So far, I have successfully created multiple ...
0
votes
1answer
43 views

Dealing with measurements falling outside of the theoretical range/boundaries of the data

Imagine I am measuring a bounded variable (with a maximum possible value above which the data doesn't make sense) and I end up with the following dataset with my measurements and measurement errors as ...
0
votes
1answer
59 views

Multiple Imputation in SPSS for RCT

I have conducted a randomised controlled trial design (2 groups - experimental and control) with data collection at two time points (T1 and T2). I want to use the Multiple Imputation Method in SPSS to ...
1
vote
0answers
339 views

missForest Data imputation vs. MICE using RF as imputation method?

Is the missForest package a special case of MICE using Random Forest as imputation (for just a single imputation)? The missForest algorithm is described here: https://academic.oup.com/bioinformatics/...
1
vote
0answers
165 views

Problems estimating a “Bayesian version of FIML”

I am anticipating that my question exposes some basic ignorance about how mcmc works, but here we go: In an attempt to deal with missing data I am trying to simultaneously estimate a regression model ...
4
votes
2answers
57 views

Best resources on imputation in R [duplicate]

This is my first question at stats. I need to impute some factors and numbers in my data set in R. What are my best options regarding packages and also a source to read more about the theory.
2
votes
0answers
43 views

What value to impute for informative NA values in R without misleading model

I'm building a model (random forest) in R to predict a rare event (scoring a goal in soccer). I have event-level data, which provides a log of all the actions (pass, tackle, foul, save, shot, goal) ...
1
vote
1answer
40 views

What type of imputation should I use?

I am learning how to handle missing values in a dataset. I have a table with ~1million entries. At the moment I am trying to deal with a small number of missing values. My data concerns a bicycle-...
0
votes
1answer
358 views

How to handle or impute large number of missing values?

I am trying to use this dataset to build a predictive model. The hubway.db file contains 3 tables. One of which is is bike_trips...
1
vote
2answers
46 views

Missing values in a variable depending on the values of another variable

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
1
vote
0answers
40 views

Missing data imputation

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
-1
votes
3answers
354 views

How can missing values in the target variable be substituted using Python?

I have a dataset with some missing values in the target variable (label). Can I use clustering to find those missing label values? What other methods can be applied to resolve such an issue in Python? ...
-1
votes
2answers
119 views

Machine Learning, Imputing values that should be blank

Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event. As an example say a teacher wants to predict the grades of his students. Some of ...
4
votes
2answers
334 views

Is it ever recommended to use mean/multiple imputation when using tree-based predictive models?

Everytime that I am making some predictive model and I have missing data I impute categorical variables with something like "UNKNOWN" and numerical variables with some absurd number that will never be ...
0
votes
0answers
254 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
1
vote
0answers
28 views

How do I create scale scores from item-level multiply imputed data in R?

I have been able to create 5 multiply imputed datasets using the mice package on R. I computed at the item level so now I have to calculate scale scores for each of the 5 imputed datasets before ...
0
votes
1answer
114 views

SPSS regression imputation

I have a dataset of 45 observations (participants), with variables on demographic data and standardized tests. Two standardized test variables are such that they have missing values on only one ...
0
votes
0answers
88 views

Can anyone help me with step by step procedure in MATLAB for missing data imputation using Singular Value Decomposition (SVD [duplicate]

I need step by step procedure for imputing data using SVD in Matlab. Please provide resources if any !
3
votes
1answer
295 views

Are VAE used for missing data imputation in multivariate time series? If not, what is used?

Multivariate time series are, to the best of my understanding, one of the few cases where Deep Learning still hasn't had its AlexNet moment. I'm especially interested to the case where most of the ...
1
vote
0answers
31 views

How do I impute data that is only partially missing?

I want to impute some missing data. I am interested in the number of months someone was unemployed between ages 18-21. This variable is bounded at 0-48. However, for some individuals, I have partial ...
7
votes
1answer
258 views

Is Structurally Missing Data a subset of Missing at Random Data?

I'm quite familiar with MCAR, MAR and MNAR (NMAR) data but I have just come across a new (for me) term: Structurally Missing Data (SMD). According to this page, Structurally missing data is data ...
1
vote
1answer
190 views

Polynomial regression for missing value imputation

I am trying to impute missing values by fitting higher degree polynomial. I have highly autocorralated time series meaning each value at t must be close to t-1. There are some noise and missing ...
0
votes
1answer
33 views

Recommended methods for replacing missing data?

We conducted a pre-post attitudinal survey measuring “Attitudes toward STEM” (28-items; α > .90) and “Multi-Ethnic Identity” (12-items; α > .90) among 50 middle schoolers. Students skipped items ...
1
vote
0answers
161 views

How to localize points from an incomplete distance matrix in R?

Suppose you have 3 shops and 2 supply units, and you only know the 6 pairwise (Euclidean, assuming 2D) distances between each shop and each supply unit, but not the pairwise distances between the ...
0
votes
1answer
454 views

Imputing and handling class imbalance

I have data with missing values. My $y$ is imbalanced (20% to 80%). a) is it at all possible to balance (e. via Smote) and Impute (e. via Mice) or will the results become too unreliable? b) if a) ...
2
votes
0answers
185 views

Partial imputation of missing dates

I'm working with dataframes (one for each of 185 locations) that shows sums of occurrences for each calendar date. There are no 0 values for occurrences in the entire dataset. There are several ...
0
votes
1answer
49 views

Missing data in regression

I am researching the effect of different marketing mix variables (e.g., price promotion, innovation) on the market share. More specifically, I want to analyze the effect of different marketing mix ...
1
vote
1answer
259 views

Predicting spendings overall and spendings for subcategories

I have a Dataset containing information about spendings of customers in various shops. There are 10 spending variables related to some categories (like spendings on clothing, spendings on hardware, ...
0
votes
1answer
122 views

Regression with missing Y’s

I use publicly available EU-Silc data to estimate the market price of social dwellings (subsidized dwellings). However my X variables are almost perfectly available,...
3
votes
0answers
265 views

Imputation and nested cross-validation

I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
0
votes
1answer
34 views

Can I correct for randomly missing data where missingness is has a known relationship to the error term?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the ...
1
vote
0answers
19 views

Imputed values into Shapefile / fortified shapefile

I currently have a shape file with approximately 20 numeric variables. Several of these variables have missing values. As this is a shapefile I do not think using the median or mean as a form of ...
3
votes
0answers
186 views

How to use MICE in R to fill missing values in test set?

It seems that MICE does not have a "predict" function which allows to use a fitted mids object to predict the missing values in test data set. I can certainly ...

1 2
3
4 5
11