Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [missing-data]

When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing an analysis or test.

2
votes
0answers
25 views

What value to impute for informative NA values in R without misleading model

I'm building a model (random forest) in R to predict a rare event (scoring a goal in soccer). I have event-level data, which provides a log of all the actions (pass, tackle, foul, save, shot, goal) ...
0
votes
0answers
12 views

How to penalize for empty fields in DataFrame?

I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns (10 circuit names) and for each of those columns I have the standard deviation in ...
0
votes
1answer
23 views

how can I perform confirmatory factor analysis when covariance matrix contains missing values?

I have dataset of test items which were administered in blocks. As a result, not all students answered each test item and there are some pairs of items for which no observations are shared, ...
0
votes
0answers
24 views

Workflow in data preparation with Box-Cox transformation

I have a dataset with both missing values and outliers in continuous features. I would like to perform Box-Cox transformation on every continuous feature to reach the best distribution. Box-Cox works ...
2
votes
0answers
15 views

Correlating Two Time Series with Gaps in Data and of Different Lengths

I am attempting to correlate the time series from 4 separate tilt monitors that sample every 5 minutes. The time series have slightly different base times and end times, and the resulting arrays are ...
1
vote
1answer
27 views

What type of imputation should I use?

I am learning how to handle missing values in a dataset. I have a table with ~1million entries. At the moment I am trying to deal with a small number of missing values. My data concerns a bicycle-...
0
votes
0answers
21 views

Handling Missing Values in the context of Time Series Data

I'm doing a study on one dataset that contains 70 financial ratios for all U.S. companies across eight different categories (Valuation, Liquidity, Profitability, and etc) from 1970 to 2018 monthly. ...
0
votes
0answers
13 views

How to run multiple imputation by predictive mean matching in multivariate missing data?

I'm running predictive mean matching for multivariate missing data for my final project, but how does the algorithm of predictive mean matching work on some variables if there are missing value in the ...
0
votes
1answer
33 views

How to handle or impute large number of missing values?

I am trying to use this dataset to build a predictive model. The hubway.db file contains 3 tables. One of which is is bike_trips...
0
votes
1answer
23 views

Instrumental Variables with Missing Observations in the Endogneous Regressor

I have 10k observations for a dependent variable $Y$ and an endogenous regressor $X$ with many missing observations (90%). I also have an instrument $Z$ without missings. I know that the values for $X$...
0
votes
0answers
7 views

Why some missing values are recorded as -1? [duplicate]

I have a lot of missing values in a dataset in a velocity column. Some of the missing values are just blank cells, some are recorded as NaN, but for some column (velocity), the missing value was ...
0
votes
0answers
8 views

Missing values while using fixed effects model

I have household-level panel data from two time periods. I found out that a good portion of my households have missing values in some variables that I need to control for, in the second time period. ...
1
vote
2answers
41 views

Missing values in a variable depending on the values of another variable

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
1
vote
0answers
28 views

R-squared vs MSE, why the discrepancy?

I am carrying out a project where I am imputing missing data. I am trying to compare an imputed dataset with a baseline dataset by measuring MSE and R-squared. These metrics are measured by ...
0
votes
0answers
7 views

Direct standardisation with missing values on ages

I am aiming to perform a direct standardisation to calculate the prevalence rate of a certain condition in an area. What I did was to merge different routinely collected health data to be able to ...
0
votes
1answer
21 views

Handling missing data in Sequence Analysis (TraMineR) within the observation window

I´m using sequence analysis. I have a question about how to deal with missing data within the observation window. The starting point of the analysis is when respondents leave secondary school (t0). I ...
1
vote
0answers
12 views

Getting pooled F-values and p-values for multiply imputed data in SPSS?

When working with a dataset created via multiple imputation, SPSS pools some values but not others. For example, in multiple regression, I can get coefficients, t-tests for the coefficients, t-values ...
0
votes
0answers
26 views

Missing data imputation

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
0
votes
0answers
29 views

Quality assessment for multiple imputation when joint distribution is not multivariate normal

I have a dataset with 100+ columns and 1000+ observations with significant amount (>60%) of data missing and fraction of missing data in individual columns varying from 10% to 90%. Data in none of the ...
0
votes
0answers
20 views

Multiple Imputation Vs Pool

I have simple question after running multiple imputation what the purpose of pooling? Suppose if i run a multiple imputation using method cart , after running this imputation technique i get very ...
0
votes
1answer
40 views

Preprocessing - dealing with natural NaN values

I am wondering how to deal with a variable having what I call natural NaN values. For example, a measure of duration between 2 events. If one event did not occure the variable has no value. For that ...
0
votes
0answers
24 views

What test to use to compare two sets of data. One only knowing mean and quantity

I have two methods that generate either a pass or a fail on input data. I want to know if statistically there is any difference between the two? For the first method I only have the mean (0.75) and ...
1
vote
1answer
46 views

Fraction of Missing Information with linear mixed models

I have a daily diary dataset (140obs for 110 persons) which I've analysed using a random slopes and intercept linear mixed model (using FIML). The model has 1 dependent variable, and 5 fixed effects. ...
0
votes
0answers
17 views

How many missings are too many to impute data with the AMELIA II package?

I have a very large longitudinal dataset. I only want to impute 3 variables individually using the AMELIA package in R. The problem is that some individuals have a lot of missing values, so I want to ...
0
votes
0answers
11 views

How does missing data affect analyses using svyset?

I am running some simple statistics using Stata and its svyset command. Does listwise deletion of cases during analyses affect the svyset? Are results still going to be weighted properly and the ...
2
votes
1answer
101 views

Bootstrap, Rubin's rules, and uncertainty of sub-estimates?

Can someone provide an intuition for why, when using bootstrap to calculate the variability of an estimate (say a regression coefficient $\beta$) we don't need to incorporate the uncertainty of each ...
0
votes
1answer
15 views

How does H2o handles missing values in DRF? [closed]

Just wanted to confirm that the h2o's implementation of RF (DRF) handles the missing values for both categorical and numerical features the same i.e., as a separate category?
0
votes
0answers
13 views

How to deal with values in predictive modeling where missingness is a predictor?

I have been reading about how to deal with missing values in predictive modeling and majority of the suggested solutions deal with either imputation or deletion. But what to do with missing values ...
0
votes
2answers
34 views

(Multinomial) Logistic regression with missing values

I want to do a (multinomial) logistic regression to predict 5 different physical activity classes based on different variables extracted for each subject. However, I have one variable (i.e., time ...
4
votes
2answers
131 views

Is it ever recommended to use mean/multiple imputation when using tree-based predictive models?

Everytime that I am making some predictive model and I have missing data I impute categorical variables with something like "UNKNOWN" and numerical variables with some absurd number that will never be ...
1
vote
1answer
79 views

How can I better predict with (g)lmer with missing values?

Suppose I'm building a mixed model in R, and I want to use that model to predict new data for which I might not know the value of all the features. Or in some cases, it might not be so much that I don'...
1
vote
1answer
24 views

Penalized multinomial regression with missing values

I would like to fit some models on a dataset where I have a lot of missing values. I am especially interested in comparing models fit with and without imputed values, because the dataset has so many ...
2
votes
1answer
45 views

use of measurements for patients who withdrew from study in multilevel model

I am working on a study where I have pre and post measurements and am interested in the impact of a treatment. I plan to use multilevel models. My question: Some subjects withdrew from the study and I ...
3
votes
1answer
154 views

How does mice::mice work?

The idea of multiple imputation seems to be based on the decomposition $$ p(\theta \mid y_{\text{obs}}) = \int p(\theta \mid y_{\text{obs}}, y_{\text{mis}})p( y_{\text{mis}} \mid y_{\text{obs}}) \text{...
1
vote
1answer
21 views

Should I report the pseudo $R^2$ value for full or final logistic regression model after removing NA's & running stepwise selection?

I'm working with a logistic regression model in r. model <- glm(response~., family="binomial", data) and I'm using ...
0
votes
0answers
30 views

analysis of categorical data for behaviour experiment

I have been stuck on how to analyse my data for many weeks. My dataset consist of 13 variables all of which are categorical. 3 variables details the continent, population and type of treatment of my ...
0
votes
0answers
15 views

Missing Data for Binary Dependent Variable

I have a Binary dependent variable (0/1) with panel data of three years(1 2 3). I want to measure the determinants of a woman choosing abortion using ordinary probit or logit. The problem is that no ...
1
vote
1answer
31 views

Will I break my model if I replace missing values with `Unknown`?

I have a classification problem with about 10,000 records. I have twenty predictors and I have data for most of the predictors. Some of the predictors provide valuable information, but I only have ...
1
vote
2answers
37 views

Missing values in questionnaire: What's next?

I'm doing a quantitative research for my paper. Recently, I went to fieldwork for collecting data using questionnaire form. But, when I'm doing a data entry, I faced a missing value for a few ...
1
vote
0answers
15 views

Calculating target mean to validate if I should drop column with missing values is correct?

I am working on the KDD 2009 Cup Data Set (The Small one) and I have a question about preprocessing data. It has a lot of columns with null values, some of them have more than 90% of missing. Reading ...
1
vote
1answer
19 views

MASE and handling nan-values

I'd like to ask advice on how to correctly compute Mean Absolute Scaled Error (2006, Hyndman, Rob J., and Anne B. Koehler.) over the following example: ...
0
votes
0answers
30 views

Treating outliers when missing values are in large amount

I am having a hard time coming up with a strategy to treat missing values and outliers in a dataset where more than 75% of values are missing. The present values are somehow extremely variable. If I ...
0
votes
0answers
22 views

SPSS regression imputation

I have a dataset of 45 observations (participants), with variables on demographic data and standardized tests. Two standardized test variables are such that they have missing values on only one ...
2
votes
0answers
17 views

LASSO Cox Model after multiple imputation

I want to develop a predictive survival model on a data set with about 8000 subjects and 38 covariates. About 4% of subjects had the event of interest. There are 21 variables with missing values, ...
0
votes
0answers
24 views

Appropriate way to deal with missing value in both training and test data

In the medical data, it's normal that there are lots of missing value. Now I am dealing with the data with tens of numerical features and many of them have lots of missing value for sure. The ...
2
votes
1answer
47 views

Are VAE used for missing data imputation in multivariate time series? If not, what is used?

Multivariate time series are, to the best of my understanding, one of the few cases where Deep Learning still hasn't had its AlexNet moment. I'm especially interested to the case where most of the ...
0
votes
0answers
18 views

Degrees of freedom after multiple imputation

Goodmorning everyone, In my research project, I made use of multiple imputation to replace missing values.SPSS lets me then run most of the tests on the imputed data set and provides output for 5 ...
2
votes
0answers
37 views

How to fill NaN values that exist because there are no measures of certain features?

I'm currently doing a ML project (the goal is simply to clean the data set and apply some of the models we learned , like Random Forests, Ensemble learning, etc, and test the results) for a class and ...
1
vote
1answer
65 views

How to handle missing data in machine learning [closed]

i know how to find and fill the missing values. But i am not sure when to fill the values with min., max. , mean, median or mode. Can someone help me to understand on what basis i can decide , i have ...
1
vote
0answers
28 views

How do I impute data that is only partially missing?

I want to impute some missing data. I am interested in the number of months someone was unemployed between ages 18-21. This variable is bounded at 0-48. However, for some individuals, I have partial ...