Questions tagged [data-imputation]

Refers to a general class of methods used to "fill in" missing data. Methods used for doing this typically are related to interpolation (http://en.wikipedia.org/wiki/Interpolation) and require assumptions about why the data is missing (e.g. "missing at random")

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

How to incorporate MICE (i.e. impute missing data) into my logistic regression?

Can someone please guide me on how to properly impute missing data (I have missing data in both of my predictors) in my logistic regression analysis? I used the MICE to handle missingness but I am not ...
0
votes
0answers
13 views

How to dealing with is missing value? [closed]

I have some problem of R programming. Can anyone give some suggestion? I need to deal with the missing value as below. The orginal one, x y 1 9 2 9 3 10 4 NA 5 NA 6 12 7 11 8 8 9 NA 10 NA ...
0
votes
0answers
11 views

Imputing Missing Data from Clusters of Environmental Sensors

I am working with a large environmental dataset with 15-minute resolution timeseries data from a variety of sensors. We collected data for multiple years and experienced some data loss as sensors ...
1
vote
1answer
45 views

Does a larger sample size increase multi-collinearity between predictors, after imputation of missing data?

I have two datasets that have exactly the same 1701 predictors, but one has 936 subjects and the other has 547 subjects. (The initial rationale for creating these two different datasets was to see ...
1
vote
0answers
13 views

How do I identify what kind of missing-data I have?

If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely: MCAR (missing ...
1
vote
0answers
18 views

Where and how to show imputed data in manuscript?

What are your thoughts on where or how to show (non-)imputed data? Please regard this question as a more a general question. I am in the field of medical clinical research, where missing data is very ...
10
votes
4answers
860 views

Can I delete missing data?

I have a dataset (4,898 X 17,000) that follows 4898 mothers, fathers, and their children over a period of 15 years. The interviews have been conducted at baseline (when the child was born), year-1, ...
1
vote
0answers
24 views

Impute Missing Correlations from Incomplete Correlation Matrix in R

I am in need of a function that will take an incomplete correlation matrix and return a complete matrix with correlations imputed for missing values. I am working on models that utilize only summary ...
1
vote
1answer
27 views

Conceptual questions about standardization/preprocessing

I have a datset consisting of: float column (seems like continuous variable with some outliers in range $[1000, 20000]$. I plotted its density curve, looks like close to normal) integer column (seems ...
0
votes
0answers
15 views

Understanding implications multiple Imputation

so Ive been studying multiple imputation and I dont quite get it. Suppose I have two variables Y and X with Y has some missing values and the missing mechanism is MAR. Suppose I use stochastic ...
1
vote
1answer
36 views

How should I handle missing values for survey data (missing due to adding a new set of items mid collection)?

I designed a survey (mostly Likert scales) and began to collect responses. After collecting several responses I decided to include an additional scale comprised of 9 items. The newly added scale will ...
1
vote
1answer
69 views

Propensity score matching after imputation in R with mice

9I have a dataset (500 rows) with missing values in different variables for a propensity score analysis. First I created the propensity score matching by omitting the rows with missing values (about ...
0
votes
1answer
38 views

Bootstrap for Performance CI with Imputation and Train/Test Split

I am currently performing an analysis in which we are hoping to develop a risk score for a survival outcome using machine learning techniques. Currently, our process is as follows: Split randomly ...
1
vote
1answer
36 views

Imputing missing data using testing data

My supervisor has instructed another person in my lab to use both training and testing data to impute missing values for use building a machine learning model. The results of the analysis haven't ...
0
votes
1answer
29 views

How should I determine what imputation method to use?

I am attempting to learn various methods of imputation to deal with missing values in a dataset. I have a dataset storing marathon segment splits (5K, 10K, ...) in seconds and identifiers (age, gender,...
1
vote
0answers
64 views

Imputing missing observations when total value of missing observations is given

I have a panel dataset with country-time. Below is an example: ...
0
votes
0answers
12 views

Distribution of imputed values and multivariate analysis

Below you'll find two plots showing the Q-Q plot (top) and histogram (bottom) of two variables. The aim is to use them, together with others, with sPLS and other multivariate methods for regression ...
3
votes
1answer
78 views

Missing data in R

I am trying to perform multiple regression on a dataset. If I have a column in my dataset which has say 900 Positive and 1000 ...
0
votes
0answers
22 views

Multiple imputation with weakly correlated and many missing values

I am pretty new to Statistics and multiple imputation. I have a dataset on high school students and their parents with around 10.000 observations and 1000 variables. I am interested in the household ...
0
votes
0answers
9 views

Will R MICE imputation accuracy be harmed by removing all duplicates? [duplicate]

I'm working with a very large amount of data, using the PMM method via R Mice. Very high degree of incompleteness in a couple of columns as well. I'm removing all the duplicates entries before ...
1
vote
0answers
31 views

Imputation Methods for Multiple Variables

I am training Recurrent Neural Network models (including LSTMs) on a dataset that includes 6-10 variables. Each variable is a properly formatted numerical measurement (ie: length, pressure, ...
1
vote
0answers
33 views

Statistical analysis on values that are not exact but above or below a limit (e.g. >100, or <5)

I have a data set with a small number of bioreplicates (3-10) per sample, and I am assessing whether each sample is statistically different from the wildtype control using a Kruskal-Wallis test with ...
0
votes
1answer
26 views

Online multivariate time series: imputation / forecasting of one of the channels with limited measurements

Problem: Data: Online (continuous) stream of multivariate time series data (>5 channels) The measurements from one of the channels (Channel F in example below) are irregular and (very) infrequent. ...
2
votes
1answer
101 views

How to best impute missing values of county-level time series data using R?

I have a dataset consisting of mobility data at the county-level for the US for about one year. So the number of observations is >1m. Apart from the county code, the date, and the mobility index, ...
0
votes
0answers
21 views

Matrix Completion problem with different observation patterns

I'm stuck to figure out why the nuclear norm minimization problem using SVD is unable to impute missing values that occur in a checkerboard pattern. MOVIE 1 MOVIE 2 MOVIE 3 MOVIE 4 MOVIE 5 USER1 1 0 ...
2
votes
1answer
28 views

Survey Analysis: Weighted Randomization to fill Missing Values

I am developing a Survey Analysis Application that works with Categorical Data and looking for a way to treat missing data fairly without losing the original data variance. Thus, I came up with a way ...
0
votes
1answer
57 views

How to use mice package to impute the subject level variables?

I am trying to do multiple imputation on a data set like below. This is a dummy data set. There are no missing values for variables idnum and eye. But specific idnum should have same bmi and gender ...
0
votes
0answers
57 views

Imputation of values using Soft impute

I've currently done a bit of homework on the behavior of matrix completion and imputations. I have been assigned to research how different structures of missing values affect the imputation ...
1
vote
0answers
35 views

imputation in one feature using correlation and other feature

I have 10 records for feature 'X' and 'Y' and the correlation between them is -0.75 Missing data: and I have 5 records with 'X' -- > missing and 'Y' I have the value. Now I need to fill this 5 ...
1
vote
0answers
40 views

How to handle the missing values in newly listed stocks?

I am trying to test some asset pricing models on 10 portfolios for the period of 2010-2020. The problem is that three of these portfolios included stocks that are newly listed in 2017 and 2018, so I ...
2
votes
1answer
163 views

How does the kNN imputer actually work?

I've understood that the kNN imputer, being a multivariate imputer, is "better" than univariate approaches like SimpleImputer in the sense that it takes multiple variables into account, ...
0
votes
1answer
24 views

Build a model to predict exam score based on previous exam scores - handle missing data

I'm building a prediction model to predict students' exam scores of the current course, based on the exam scores of the previous courses they have taken. The thing is, not every student has the same ...
0
votes
0answers
20 views

Package MICE: To include or not include the independent variables, and how many, when impute a data set

I am new to data imputation. This question come from studying the mammalsleep data set in the MICE package. The imputation ...
0
votes
0answers
4 views

Predicting values in combined training/testing data after model selection

I've trained several predictive models to try to impute missing data on training data and evaluated the performance of the models on test data. Now that I've found a model that performed the best, is ...
0
votes
1answer
64 views

Should multiple imputation be used before or after analytic sample inclusion/exclusion criteria are applied?

I have a question about using multiple imputation. I have a dataset of approximately 16,000 people with over 15 years of follow-up. To address missing data and loss-to-follow-up, I'm using multiple ...
0
votes
0answers
24 views

Verify whether data is missing at random (for multiple imputation)

I'm currently working with a data set from the medical domain, in which several parameters contain missing values (up to 40% of the data). The missing data is mostly caused by some hospitals not ...
0
votes
0answers
17 views

Extending coverage of panel data for one variable in large dataset

I would like to extend the coverage of my panel data for one variable. This variable, population at a very small level, is NA for the first 8 of 20 years of the panel for all cross-sectional units; ...
1
vote
0answers
15 views

methods for filling out the upper half of a similarity matrix

I have some data and a method of finding the similarity between any two points, but this method is expensive computationally and I do not need a precise answer as I want to use this matrix for ...
1
vote
1answer
94 views

Assigning gender based on first name when have missing data?

I have some missing data for gender in a study (between 10-20%) and am wondering how to address it. There are methods that assign gender based on first name. However, this came up in discussions of ...
0
votes
0answers
43 views

OneHot Encoding: Nan value - remove row or represent as a list full of zeros?

In the analysis survey, I have responses without answers and I'm not sure what is a better approach. Most of the attributes are categorical and missing numerical attributes have to remove. What can I ...
0
votes
0answers
28 views

Compare correlation coefficients between groups from multiply imputed data sets

I have the correlation coefficients between two variables (scores on a screener and scores on the criterion measure) for students from different racial backgrounds, and I am interested to see whether ...
2
votes
0answers
147 views

ARIMA with external regressors for district heating load time-series imputation

TL;DR ARIMA model does not work as expected. There is a load for district heating with some external regressors like temperature and wind speed. There is missing data which we like to impute. We work ...
2
votes
1answer
166 views

Machine Learning on MICE-imputed data

I'm working on a project with medical data where some of it is missing. We decided to impute the data using MICE and I found enough literature about how to choose $m$ (the number of imputations) and $...
1
vote
1answer
16 views

Preferred way to sum different time series together (in software)

Is there a canonical/best approach to computationally summing different time series together? What I mean by that is the operation $$ \sum_i{s_i(t)} = S(t) $$ where $s_i(t)$ is the $i$-th time series, ...
1
vote
0answers
17 views

How to analyze usefulness of imputation output in R

I am working with a dataset with 3,500 observations and includes a Body Mass Index variable. There are around 300 NA values for the BMI variable which I have imputed using multiple imputation. ...
0
votes
1answer
125 views

Can I use the na_kalman function in R (imputeTS) to replace missing data in a bivariate time series if the function is applied to only one series?

I am doing a bivariate time series analysis. There is missing data in one of the series (approximately 5% of the data ist missing). Is it appropriate to use the na_kalman function i R (package: ...
0
votes
0answers
19 views

Should I use data imputation to input into a multivariate regression?

I am using survey data (has complex survey design) with demographic factors such as age, gender, and race to predict respondent's Body Mass Index (BMI) using a multivariate regression model. However, ...
0
votes
0answers
24 views

Inclusion of missing data with adapted Winsorizing approach for small sample?

We have a dataset of patients with a rare condition (n=~10 for each of 2 groups). The participants completed a battery of standardized tests that use a mean of 100 and a standard deviation of 15. In ...
0
votes
0answers
150 views

Shall a one-hot coding applied before or after the missing values imputed?

For the case when the categorical data is handled, it is suggested to one-hot encode the values so as to digitize the value of the data. Many examples taking color (green, red, blue) as an example so ...
1
vote
0answers
9 views

methodological question: matching/imputation based on two datasets

I struggle to find the right method for what I want to do using two household surveys. I have two datasets: X dataset with socio-econ info (A1) and Z info Y dataset with socio-econ info (A2) The Y ...

1
2 3 4 5
12