Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [missing-data]

When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing an analysis or test.

1
vote
1answer
53 views

Missing data when actually predicting - Additional model legit?

This is a theoretical question, but I have already stumbled upon this issue a few times: My learning data are not complete, but I manage to handle the missing values. Now it's time for actually ...
0
votes
0answers
4 views

Correct for known frequency of random sampling in prediction

When finding the expectation of a parameter $p$, depending on another parameter $h$, one likely wants to apply the formula $\hat{p}=E[p]=\sum_k E[p|h=k] P(h=k)$. Now, often the problem allows one to ...
0
votes
1answer
18 views

MissForest for SurveyData

Hello fellow data scientist, I currently reading the paper by Stekhoven & Brühlmann about MissForest. I was wondering how to deal with variables that are restricted by domain knowlege. I.e. no ...
0
votes
0answers
11 views

Difference in R lmer coefficient estimates due to specification of data set with missingness [closed]

My outcome variable has about 60% missingness and is the only variable with missingness in my model. My primary analysis imputes the outcome while my secondary analysis is based only on complete cases....
0
votes
0answers
13 views

Interpreting a 2 x 3 design with two empty cells

My psychology grad student and I have a study conducted in three different countries using two different method variants. Neither country nor method variant is a predictor of focal interest, but might ...
1
vote
1answer
33 views

Can i use autoencoder for predicting time series missing data?

I have time series data set of current and voltage at a regular interval of time there are some missing value . can i use autoencoder to predict the missing value?
0
votes
1answer
32 views

Using lme4/ linear mixed model to model longitudinal data in which age at first test and duration between measurements vary

I need specify a model using lmer() with data (n > 3500) in which there is a lot of variability in the age of the (human) subjects and also the duration between measurements (5 measurements total but ...
4
votes
0answers
40 views

Recent Example of the Consequences of Bad Data in Business/Commercial Domain

Preface I work at an ecommerce saas company. I have been asked to perform an analysis on the relationship between behaviors of potential customers during free trials and their conversion to paying ...
1
vote
4answers
46 views

Are there any clustering algorithms that do not exclude/impute missing data?

From my understanding, clustering algorithms require complete data. Based on this, if there are missing values in my dataset I have two options: Impute missing information using some sort of ...
0
votes
0answers
8 views

Estimation of outcomes when sampling coverage is limited

Set-up: Suppose I have communities $i = 1, \cdots, I$ with individuals $j = 1, \cdots, J_i$, and these $\{J_i\}$ and $I$ are known. I have individual-level covariates $X_{ij}$ and binary outcome $Y_{...
0
votes
0answers
14 views

Pool redundancy analysis results from multiple imputations of missing data

I have a dataset that includes missing values, and I would like to carry out redundancy analysis using multiple imputation to fill in the missing values. So far, I have successfully created multiple ...
2
votes
1answer
8 views

Must a subject participate at least twice to be considered in longitudinal analysis?

I am currently analyzing longitudinal data that was repeatedly measured (four times) but 1000 participants were lost to follow up after the first survey. Shall I include or exclude these ...
0
votes
0answers
25 views

How to treat missing data involving multiple entries

I'm developing a model used to predict the probability of a client changing telephone companies based on their daily usage. My dataset has information from two weeks (14 days). My datasets include in ...
0
votes
1answer
26 views

Missing data and unbalanced data set in 2-factors design

I'm new to statistics and I'm trying to understand what to do with my data! I have two factors : tree genotype (10 levels) and soil type (3 levels). For one genotype I have only 2 replicates in soil1 ...
0
votes
0answers
11 views

Multiple Imputation in SPSS for RCT

I have conducted a randomised controlled trial design (2 groups - experimental and control) with data collection at two time points (T1 and T2). I want to use the Multiple Imputation Method in SPSS to ...
0
votes
0answers
16 views

How to do when both response and covariates are missing?

I have some sort of longitudinal data and want to do regression analysis. The problem is that there are missing data both in response variable(y) and covariates(X). (sometimes only y observed, ...
0
votes
2answers
47 views

Logistic regression with missing data: which rows/columns to eliminate? What is the most simple method?

I have a large dataset (501 rows and 39 columns) with a lot of missing data. I have already deleted all the rows where the (binary) response variable is missing as well as three columns that were ...
0
votes
1answer
41 views

Anomaly detection in time series data from multiple sensors [closed]

I've build a classification model based on 15 features coming in real time from 15 sensors. The window time is 60 seconds, means that the classification model needs 60 records from each sensor (the ...
1
vote
0answers
37 views

Problems estimating a “Bayesian version of FIML”

I am anticipating that my question exposes some basic ignorance about how mcmc works, but here we go: In an attempt to deal with missing data I am trying to simultaneously estimate a regression model ...
0
votes
0answers
23 views

Recommender system with extra variables

I would like to to create a recommendation engine that makes use of a utility matrix (user-item interactions) as well as supplementary features (user features, item features and time-based features). ...
0
votes
0answers
6 views

How to handle partial observations of the variable of interest when training a time series model?

I have the following time series data: $\{ t_i, X_i, Y_i \}$ where $i$ is the index, $t_i$ is the timestamp, $X_i$ the measured value of the external variable and $Y_{i}$ the value of the variable ...
2
votes
0answers
37 views

What value to impute for informative NA values in R without misleading model

I'm building a model (random forest) in R to predict a rare event (scoring a goal in soccer). I have event-level data, which provides a log of all the actions (pass, tackle, foul, save, shot, goal) ...
0
votes
0answers
13 views

How to penalize for empty fields in DataFrame?

I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns (10 circuit names) and for each of those columns I have the standard deviation in ...
0
votes
1answer
32 views

how can I perform confirmatory factor analysis when covariance matrix contains missing values?

I have dataset of test items which were administered in blocks. As a result, not all students answered each test item and there are some pairs of items for which no observations are shared, ...
0
votes
0answers
28 views

Workflow in data preparation with Box-Cox transformation

I have a dataset with both missing values and outliers in continuous features. I would like to perform Box-Cox transformation on every continuous feature to reach the best distribution. Box-Cox works ...
2
votes
0answers
23 views

Correlating Two Time Series with Gaps in Data and of Different Lengths

I am attempting to correlate the time series from 4 separate tilt monitors that sample every 5 minutes. The time series have slightly different base times and end times, and the resulting arrays are ...
1
vote
1answer
30 views

What type of imputation should I use?

I am learning how to handle missing values in a dataset. I have a table with ~1million entries. At the moment I am trying to deal with a small number of missing values. My data concerns a bicycle-...
0
votes
0answers
22 views

Handling Missing Values in the context of Time Series Data

I'm doing a study on one dataset that contains 70 financial ratios for all U.S. companies across eight different categories (Valuation, Liquidity, Profitability, and etc) from 1970 to 2018 monthly. ...
0
votes
0answers
20 views

How to run multiple imputation by predictive mean matching in multivariate missing data?

I'm running predictive mean matching for multivariate missing data for my final project, but how does the algorithm of predictive mean matching work on some variables if there are missing value in the ...
0
votes
1answer
50 views

How to handle or impute large number of missing values?

I am trying to use this dataset to build a predictive model. The hubway.db file contains 3 tables. One of which is is bike_trips...
0
votes
1answer
30 views

Instrumental Variables with Missing Observations in the Endogneous Regressor

I have 10k observations for a dependent variable $Y$ and an endogenous regressor $X$ with many missing observations (90%). I also have an instrument $Z$ without missings. I know that the values for $X$...
0
votes
0answers
7 views

Why some missing values are recorded as -1? [duplicate]

I have a lot of missing values in a dataset in a velocity column. Some of the missing values are just blank cells, some are recorded as NaN, but for some column (velocity), the missing value was ...
0
votes
0answers
14 views

Missing values while using fixed effects model

I have household-level panel data from two time periods. I found out that a good portion of my households have missing values in some variables that I need to control for, in the second time period. ...
1
vote
2answers
44 views

Missing values in a variable depending on the values of another variable

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
1
vote
0answers
61 views

R-squared vs MSE, why the discrepancy?

I am carrying out a project where I am imputing missing data. I am trying to compare an imputed dataset with a baseline dataset by measuring MSE and R-squared. These metrics are measured by ...
0
votes
0answers
10 views

Direct standardisation with missing values on ages

I am aiming to perform a direct standardisation to calculate the prevalence rate of a certain condition in an area. What I did was to merge different routinely collected health data to be able to ...
0
votes
1answer
26 views

Handling missing data in Sequence Analysis (TraMineR) within the observation window

I´m using sequence analysis. I have a question about how to deal with missing data within the observation window. The starting point of the analysis is when respondents leave secondary school (t0). I ...
1
vote
0answers
20 views

Getting pooled F-values and p-values for multiply imputed data in SPSS?

When working with a dataset created via multiple imputation, SPSS pools some values but not others. For example, in multiple regression, I can get coefficients, t-tests for the coefficients, t-values ...
0
votes
0answers
28 views

Missing data imputation

I'm working on a public procurement dataset where I have information on all the participants that presented offers in 358 tenders. I'm analysing relationships between all the companies of the dataset (...
0
votes
0answers
29 views

Quality assessment for multiple imputation when joint distribution is not multivariate normal

I have a dataset with 100+ columns and 1000+ observations with significant amount (>60%) of data missing and fraction of missing data in individual columns varying from 10% to 90%. Data in none of the ...
0
votes
0answers
24 views

Multiple Imputation Vs Pool

I have simple question after running multiple imputation what the purpose of pooling? Suppose if i run a multiple imputation using method cart , after running this imputation technique i get very ...
0
votes
1answer
41 views

Preprocessing - dealing with natural NaN values

I am wondering how to deal with a variable having what I call natural NaN values. For example, a measure of duration between 2 events. If one event did not occure the variable has no value. For that ...
0
votes
0answers
24 views

What test to use to compare two sets of data. One only knowing mean and quantity

I have two methods that generate either a pass or a fail on input data. I want to know if statistically there is any difference between the two? For the first method I only have the mean (0.75) and ...
1
vote
1answer
51 views

Fraction of Missing Information with linear mixed models

I have a daily diary dataset (140obs for 110 persons) which I've analysed using a random slopes and intercept linear mixed model (using FIML). The model has 1 dependent variable, and 5 fixed effects. ...
0
votes
0answers
22 views

How many missings are too many to impute data with the AMELIA II package?

I have a very large longitudinal dataset. I only want to impute 3 variables individually using the AMELIA package in R. The problem is that some individuals have a lot of missing values, so I want to ...
0
votes
0answers
12 views

How does missing data affect analyses using svyset?

I am running some simple statistics using Stata and its svyset command. Does listwise deletion of cases during analyses affect the svyset? Are results still going to be weighted properly and the ...
2
votes
1answer
109 views

Bootstrap, Rubin's rules, and uncertainty of sub-estimates?

Can someone provide an intuition for why, when using bootstrap to calculate the variability of an estimate (say a regression coefficient $\beta$) we don't need to incorporate the uncertainty of each ...
0
votes
1answer
23 views

How does H2o handles missing values in DRF? [closed]

Just wanted to confirm that the h2o's implementation of RF (DRF) handles the missing values for both categorical and numerical features the same i.e., as a separate category?
0
votes
0answers
18 views

How to deal with values in predictive modeling where missingness is a predictor?

I have been reading about how to deal with missing values in predictive modeling and majority of the suggested solutions deal with either imputation or deletion. But what to do with missing values ...
0
votes
2answers
47 views

(Multinomial) Logistic regression with missing values

I want to do a (multinomial) logistic regression to predict 5 different physical activity classes based on different variables extracted for each subject. However, I have one variable (i.e., time ...