Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

0
votes
0answers
10 views

fancyimpute - crashes my Python runtime/environment every time I use the package [on hold]

I'm trying to impute null values in my data using fancyimpute python package. First, as suggested by this stackoverflow question, I converted my dataframe to numpy array in order for ...
2
votes
0answers
25 views

What value to impute for informative NA values in R without misleading model

I'm building a model (random forest) in R to predict a rare event (scoring a goal in soccer). I have event-level data, which provides a log of all the actions (pass, tackle, foul, save, shot, goal) ...
1
vote
1answer
27 views

What type of imputation should I use?

I am learning how to handle missing values in a dataset. I have a table with ~1million entries. At the moment I am trying to deal with a small number of missing values. My data concerns a bicycle-...
0
votes
0answers
13 views

How to run multiple imputation by predictive mean matching in multivariate missing data?

I'm running predictive mean matching for multivariate missing data for my final project, but how does the algorithm of predictive mean matching work on some variables if there are missing value in the ...
0
votes
1answer
33 views

How to handle or impute large number of missing values?

I am trying to use this dataset to build a predictive model. The hubway.db file contains 3 tables. One of which is is bike_trips...
1
vote
2answers
41 views

Missing values in a variable depending on the values of another variable

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
0
votes
0answers
26 views

Missing data imputation

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
-1
votes
3answers
34 views

How can missing values in the target variable be substituted using Python?

I have a dataset with some missing values in the target variable (label). Can I use clustering to find those missing label values? What other methods can be applied to resolve such an issue in Python? ...
0
votes
0answers
16 views

SAS vs. Stata: Interpreting Tobit scale and sigma Coefficients

I'm trying to replicate a SAS script that imputes censored wage observations using a Tobit model in Stata. However, I'm confused ...
0
votes
2answers
38 views

Machine Learning, Imputing values that should be blank

Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event. As an example say a teacher wants to predict the grades of his students. Some of ...
4
votes
2answers
131 views

Is it ever recommended to use mean/multiple imputation when using tree-based predictive models?

Everytime that I am making some predictive model and I have missing data I impute categorical variables with something like "UNKNOWN" and numerical variables with some absurd number that will never be ...
0
votes
0answers
45 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
0
votes
0answers
13 views

How do I create scale scores from item-level multiply imputed data in R?

I have been able to create 5 multiply imputed datasets using the mice package on R. I computed at the item level so now I have to calculate scale scores for each of the 5 imputed datasets before ...
0
votes
0answers
22 views

SPSS regression imputation

I have a dataset of 45 observations (participants), with variables on demographic data and standardized tests. Two standardized test variables are such that they have missing values on only one ...
0
votes
0answers
19 views

Can anyone help me with step by step procedure in MATLAB for missing data imputation using Singular Value Decomposition (SVD [duplicate]

I need step by step procedure for imputing data using SVD in Matlab. Please provide resources if any !
2
votes
1answer
47 views

Are VAE used for missing data imputation in multivariate time series? If not, what is used?

Multivariate time series are, to the best of my understanding, one of the few cases where Deep Learning still hasn't had its AlexNet moment. I'm especially interested to the case where most of the ...
0
votes
0answers
18 views

Degrees of freedom after multiple imputation

Goodmorning everyone, In my research project, I made use of multiple imputation to replace missing values.SPSS lets me then run most of the tests on the imputed data set and provides output for 5 ...
0
votes
0answers
19 views

Large discrepancy between complete-cases and imputed data

I would like to conduct a survival analysis using a dataset with approximately 12,000 participants (1100 events). However, complete data are available for only 9500 participants (820 events). I have ...
1
vote
0answers
28 views

How do I impute data that is only partially missing?

I want to impute some missing data. I am interested in the number of months someone was unemployed between ages 18-21. This variable is bounded at 0-48. However, for some individuals, I have partial ...
3
votes
0answers
33 views

Is Structurally Missing Data a subset of Missing at Random Data?

I'm quite familiar with MCAR, MAR and MNAR (NMAR) data but I have just come across a new (for me) term: Structurally Missing Data (SMD). According to this page, Structurally missing data is data ...
1
vote
0answers
40 views

Polynomial regression for missing value imputation

I am trying to impute missing values by fitting higher degree polynomial. I have highly autocorralated time series meaning each value at t must be close to t-1. There are some noise and missing ...
0
votes
0answers
7 views

Model to Impute strata data according to higher level data

I've got regional level data and I want to impute said data on a county level strata (smaller strata with respect to the regional strata). I know I can do that if I have a series of variables on the ...
0
votes
0answers
35 views

nnet with predictors with missing values

I got about 10 Variables with customer spendings and another 10 variables with background information (like homecountry, average income etc.). I want to use nnet to impute missing values in the ...
0
votes
1answer
29 views

Recommended methods for replacing missing data?

We conducted a pre-post attitudinal survey measuring “Attitudes toward STEM” (28-items; α > .90) and “Multi-Ethnic Identity” (12-items; α > .90) among 50 middle schoolers. Students skipped items ...
1
vote
0answers
38 views

How to localize points from an incomplete distance matrix in R?

Suppose you have 3 shops and 2 supply units, and you only know the 6 pairwise (Euclidean, assuming 2D) distances between each shop and each supply unit, but not the pairwise distances between the ...
0
votes
1answer
88 views

Imputing and handling class imbalance

I have data with missing values. My $y$ is imbalanced (20% to 80%). a) is it at all possible to balance (e. via Smote) and Impute (e. via Mice) or will the results become too unreliable? b) if a) ...
0
votes
0answers
11 views

instance propagation

I have read about label propagation where you have aggregated labels (y_g) but instance level features (x_i). Is there also literature about problems where features are (also) aggregated, and we want ...
1
vote
0answers
50 views

Partial imputation of missing dates

I'm working with dataframes (one for each of 185 locations) that shows sums of occurrences for each calendar date. There are no 0 values for occurrences in the entire dataset. There are several ...
0
votes
1answer
39 views

Missing data in regression

I am researching the effect of different marketing mix variables (e.g., price promotion, innovation) on the market share. More specifically, I want to analyze the effect of different marketing mix ...
1
vote
1answer
237 views

Predicting spendings overall and spendings for subcategories

I have a Dataset containing information about spendings of customers in various shops. There are 10 spending variables related to some categories (like spendings on clothing, spendings on hardware, ...
0
votes
1answer
92 views

Regression with missing Y’s

I use publicly available EU-Silc data to estimate the market price of social dwellings (subsidized dwellings). However my X variables are almost perfectly available,...
3
votes
0answers
78 views

Imputation and nested cross-validation

I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
0
votes
1answer
28 views

Can I correct for randomly missing data where missingness is has a known relationship to the error term?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the ...
1
vote
0answers
12 views

Imputed values into Shapefile / fortified shapefile

I currently have a shape file with approximately 20 numeric variables. Several of these variables have missing values. As this is a shapefile I do not think using the median or mean as a form of ...
3
votes
0answers
77 views

How to use MICE in R to fill missing values in test set?

It seems that MICE does not have a "predict" function which allows to use a fitted mids object to predict the missing values in test data set. I can certainly ...
0
votes
1answer
44 views

Imputing nested time series data with R

Does anyone know what is the superior algorithm to impute data in time series? I had strong dropouts over time because it was free to participants how many times to participate in my study (otherwise ...
0
votes
1answer
46 views

Proper Imputation and bias-correction on degrading signal with Kalman Filtering?

A signal degrades in its quality. Some signals are far more robust to degradation while others are not. We will simulate degradation by randomly removing values from a function and then applying ...
2
votes
0answers
171 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
0
votes
0answers
75 views

can I fill missing values by using target variable?

I have a 3 column data with 2 features and 1 target variable. But the first features (numeric) have a large number of missing values. If I use kNN to fill in the missing values, I am wondering can I ...
0
votes
1answer
140 views

How does Excel interpolate / imputate missing values in time-series when fitting a line to a plot?

I have a scatter plot in Excel (upper part of the screenshot) of time-series data. In-between the values that I plot (to the left), are some missings. I fit a (linear) line to those values and display ...
0
votes
0answers
15 views

What is a good reference on the philosophy of data imputation?

I would like to read something interesting that addresses why and when data imputation is advisable. I have only been able to find technical stuff about particular imputation methods but I want to ...
1
vote
0answers
23 views

Shouldn't we consider larger standard errors for effect measures or outcomes that are converted in meta-analysis?

There are methods to convert effect measures in meta-analysis (pdf). There are also methods to convert outcomes; at least, I am aware of the conversion described in Furukawa et al. (2005) from ...
1
vote
0answers
25 views

How do I impute clustered data that is not time-series data?

The goal of my research is to understand whether MRI imaging characteristics can predict tumor pathology. The data consists of resected tumor samples, with multiple samples per patient. On the MRI, we ...
1
vote
2answers
131 views

Why are missing values MNAR harder to impute than MCAR or MAR?

Reading papers related to the imputation of missing values related to the -omics field, systematically imputation algorithms were less accurate when imputing MNAR compared to imputing MCAR. My ...
0
votes
1answer
42 views

Imputing values with linear regression, valid strategy or creating biases?

I am practicing on the titanic competition from kaggle. In the dataset the Age variable has a number of missing values and I am now left with the choice of what to ...
0
votes
0answers
20 views

Measurements to deterministic value

I have a number of measurements of two variables: the number of products, the weight. Sometimes the weight is missing and sometimes the number of products is missing. I want to use the given data to ...
0
votes
0answers
21 views

Imputing missing data with MICE where each observation has different levels

I have a set of observations that each consists of different levels. For example, I ask a $P$ individuals $N$ questions, each question with a possible $k_n$ discrete responses. This produces a table ...
0
votes
1answer
24 views

Imputing binary variable when no 0s, only 1s are available

I'm trying to impute missing values for a binary variable (values 0 and 1) with some challenging data (of about 1 million observations). The data can be divided into two groups: in group 1, we know ...
1
vote
1answer
272 views

Predictive Mean Matching as Single Imputation?

Multiple imputation is known to be advantageous compared to single imputation. However, in practice there are often non-statistical reasons why multiple imputation can not be used (e.g. the data ...
3
votes
2answers
280 views

data imputation of missing values in non-normally distributed explanatory variables

I have been told that mean imputation of missing values is inappropriate when the variables underlying distribution is non-normal. my variable is contiunous (but bound at 100) and most observations ...