Skip to main content

Questions tagged [missing-data]

When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing an analysis or test.

Filter by
Sorted by
Tagged with
1 vote
0 answers
15 views

Statistical Non-Response and Drop Out

Hello Stats Community! This is a question I have. In statistical studies, it is possible that there might be biases: Someone groups of people are more likely to be represented compared to others ...
user avatar
0 votes
1 answer
48 views

Drop-outs in RCT

I have conducted a RCT with 34 participants, experimental group and a control group. We took measurement data before randomisation at baseline (T0), and then follow-up measurements every 4 months (so ...
Mai's user avatar
  • 21
0 votes
0 answers
9 views

Multilevel/mixed model when group is missing for some of the data

A client has given me a dataset of corn samples gathered from different loads of corn that were delivered to a grain elevator. They are interested in whether the concentration of different fungal ...
qdread's user avatar
  • 295
2 votes
2 answers
131 views

Multiple Imputation for missing outcomes in Cox regression

Imagine an RCT with a time-to-event outcome which is analyzed using a Cox regression. There are four assessments (T1=before randomization, T2=3 weeks, T3=6 weeks, T4=12 weeks). Under the censoring at ...
Survival's user avatar
2 votes
0 answers
65 views

Is it valid to use "withdrew from study" in a mixed model to help with dropout bias in longitudinal data?

Consider the hypothetical dataset where a number of study participants have no change over time in a particular outcome, but their probability to drop out of the study is related to the baseline value ...
Evan's user avatar
  • 225
1 vote
0 answers
6 views

Comparing assessment methods in longitudinal design

I am currently trying to figure out what to do but I can´t seem to find a proper solution. My problem is the following: I have a longitudinal design. For each day I have a variable stating whether an ...
Laura's user avatar
  • 11
0 votes
0 answers
26 views

what are the consequences of re-encoding with 0 or -1?

I have a dataset of information about students and the last column is the target variable which is the final note. My goal is to make logistic regression and ordinal regression models to see whether ...
Moez Daly's user avatar
0 votes
0 answers
8 views

Handling valid missing data in prediction model features

I'm developing an elastic net model using caret, with k-nearest neighbours to handle missing data in the features. Some features are conditional on others such that a missing value is valid e.g., ...
megsk's user avatar
  • 3
1 vote
0 answers
32 views

ML Modelling advice where a feature is partially missing but highly informative when present

I am building a model to predict a customer purchase event on a website. Specifically for those customers who, overnight when the model is run, have not yet purchased. Prediction is important, but ...
Jon's user avatar
  • 141
1 vote
0 answers
14 views

How to deal with MNAR data based on failures in a trial?

Background I have a dataset in which participants had to perform a certain task within a time frame. The outcome of this task is not binary, but there was a criterion that should be reached within the ...
Dom42's user avatar
  • 115
2 votes
0 answers
39 views

Missing Data in experiments, MIPO, MIPO|X vs MCAR, MAR, MNAR [closed]

Hello I was reading Field Experiments by Alan Gerber and Donald Green and was introduced to the idea of missingness independent of potential outcomes (MIPO). And MIPO|X which is missingness ...
Vefeagins's user avatar
  • 518
0 votes
0 answers
16 views

Reasons for failure of convergence in multiple imputation in wide format of (longitudinally) repeated Likert items

Our group is working on a dataset of approximately 1000 patients with 10 complete variables at the time of an acute disease, who subsequently completed a questionnaire of 5 Likert items (questions ...
torwart's user avatar
  • 305
1 vote
0 answers
16 views

mice: Default ridge parameter

Some methods in mice like pmm and norm apply a default ridge parameter of 1e-5 when the underlying data is nearly multi-...
jerome's user avatar
  • 31
1 vote
1 answer
85 views

Difference between truncated and unseen data

I have 2 related questions. Assume that we want to build a model to study of some random discrete variable $x$ that follows some known distribution with PMF $P(x)$, yet with unknown parameters that we ...
Geo's user avatar
  • 37
1 vote
3 answers
105 views

Combining regression models based on missing data patterns

I have a dataset that contains a few patterns of missingness. For this dataset, I have a training set that is complete and contains all input features. My test set has complete observations for the ...
Squan Schmaan's user avatar
2 votes
0 answers
43 views

Cox Regression: handling immediate drop-outs

In an RCT with two groups, I‘m currently analyzing data using cox regression. While I‘m familiar with the concept of censoring, a rather substantial amount of participants (~32% and 35% in the groups) ...
Survival's user avatar
5 votes
1 answer
123 views

SEM with 50% missing data (due to distribution of items over various survey ballots/waves)

I want to test a multilevel mediation (particularly, test a number of hypotheses about possible mediators of the effect of social class on Right-Wing Authoritarianism) using largely categorical ...
ThomasV's user avatar
  • 51
0 votes
1 answer
16 views

Data imputation for Laboratory Experiments

I have a timeseries dataset containing several experiments that were conducted in batches and concatenated (each batch is 48hours, with data recorded every 2 hours). However, there is a target ...
paul okafor's user avatar
0 votes
0 answers
50 views

How to deal with missing values in a panel survey (for Propensity score matching analysis)

I would like to know what is the recommended method for data imputation for propensity score matching in panel survey data. This survey has 4 waves and I am examining the treatment effect between the ...
user23960363's user avatar
0 votes
0 answers
6 views

Comparing Two-Way Fixed Effects Model using Pooled OLS and Panel Data Fixed Effects Approach

I am working with an unbalanced panel dataset that has many missing values. I want to control for both firm-specific and time-specific effects in my analysis. I am considering two approaches: ...
Newbie's user avatar
  • 83
0 votes
0 answers
16 views

Missing features in decision tree based algorithms

I have a medium-sized dataset consisting of many features, some of which can contain missing values. I want to predict a variable using an algorithm that employs decision trees (specifically XGBoost, ...
umbal's user avatar
  • 75
0 votes
0 answers
23 views

Large Scale Missing Data & Imputation of Time Series Data in Neural Networks [duplicate]

I know there has already been a lot of discussion about this topic, but I have reasons to believe it still remains unanswered and lacks several justifications. Suppose we have an time series feature ...
LazyAnalyst's user avatar
0 votes
0 answers
9 views

Seeking Advice on Handling Systematic Missingness Due to Conditional Logic in Dataset for Binary Outcome Analysis

I am tackling a complex multivariate analysis project with a binary outcome variable. My dataset's unique challenge arises from structured missingness due to conditional logic responses, significantly ...
g y l's user avatar
  • 1
2 votes
1 answer
46 views

How find the "closest" (in a sense of data generating process) time series?

Suppose we have overall $m$ time series, each with $n$ observations. We also have another time series with $n-k$ observations ($k>0$). Given the shortest series, I want to find from $m$ series ...
Sane's user avatar
  • 353
0 votes
0 answers
24 views

Does survey R package allow me to do beta regression?

I have a complex survey dataset with a response (dependent variable) bounded between 0 and 1, where I have applied multiple imputation to the dataset to account for missing data. The response formally ...
user45765's user avatar
  • 1,446
0 votes
0 answers
8 views

Weighing by missing observations

I have data where each row represents an observation and each column represents a variable. Variables are binary and has missing values. I want to calculate a sum of the presence of these variables. I ...
clarrity's user avatar
1 vote
1 answer
38 views

Question on Best Transformation (Negative, Zero, Positive Values) + Missing Data

I have a dataset with $5000$ observations, and 10 explanatory and 1 response variable (binary 0 or 1), and my task is to make a logistic regression model for prediction (but also needs to provide some ...
Alex Smirnov's user avatar
7 votes
3 answers
609 views

How to analyze a dichotomous outcome with 50% missing data?

I am researching predictors of dropout from a training program. I want so to see if personality traits add incremental variance above well-established predictors like age, fitness, and education. So, ...
E_H's user avatar
  • 341
0 votes
0 answers
12 views

Interpretation question about the article "Finite mixture modeling of censored data using the multivariate Student-t distribution"

I am reading this article and I am struggling to understand the following passage from page 6: \begin{align*} L_{i}(\boldsymbol{\theta}|\textbf{V}_{i},\textbf{C}_{i}) = f(\textbf{V}_{i}|\textbf{C}_{i},...
user1234's user avatar
  • 235
0 votes
0 answers
16 views

Analyzing longitudinal data with deaths

I've been given some longitudinal data of animals given different glioblastoma treatments. In addition to conventional survival analysis, they want to look at animal weights and sizes of tumors along ...
Bryan's user avatar
  • 1,059
1 vote
1 answer
18 views

How to report participant number with missing times?

I have used a linear mixed model for a set of data with children participating in experiments 1-4 times. The number of times they participated varies. Some kids skipped or participated in one or two ...
Funi's user avatar
  • 11
0 votes
0 answers
56 views

Principal components analysis missing data SPSS

I have a large data set (27k) There are 63 variables of interest. I attempted PCA as a dimensionality reduction strategy. The items are scored on different metrics ranging from dichotomous variables ...
Danny's user avatar
  • 1
0 votes
1 answer
38 views

Double selection lasso in and NA's handling

I work in a team where everyone uses Stata, and I work in R. I have created an efficient workflow that allows me to export the results quickly. The problem I ran into was when implementing the double ...
Paula's user avatar
  • 131
0 votes
0 answers
19 views

Approach for multivariate outlier detection when treating missing values with FIML

I‘m calculating a simple regression with one predictor and one dependent variable. Missings treatment is done with full information maximum likelihood (FIML). Should I do outlier detection, i.e. ...
Madamadam's user avatar
  • 247
0 votes
0 answers
81 views

Shap values associated to missing features

I have a model trained with Xgboost on some training data X_train, described by 10 features (x$_{1}$,...,x$_{10}$) and some of them might exhibit some missingness, i.e. some x$_{i}$ = NaN. This is ok ...
MarcoC's user avatar
  • 3
0 votes
0 answers
35 views

imputation of NA on financial ratios

I'm a student doing a data science report on finance. I need to impute NA on some financial ratios. For example, ROAE. I have negative values and positive values. There are also NA, where they are ...
LUcas1234's user avatar
0 votes
0 answers
16 views

Identifying the type of missing data and the post hoc test that can be carried out for Skillings Mack test

I have a non-normal paired sample dataset. Each row represents a dog that has been tested for an experiment. Each dog was provided with three cues (treatments): 5s cue (aka only face cue), vocalone (...
Rohan Sarkar's user avatar
3 votes
0 answers
54 views

I get very different and inconsistent model fit indices and test statistics every time I run the cfa.mi models with the same imputed categorical data

I am trying to test the measurement invariance in my one-latent-factor model. I have 24 questions coded "correct" and "incorrect" and 3 different groups in my data. The sample ...
Enes Bayrakoglu's user avatar
2 votes
1 answer
41 views

Multilevel model with random subset of conditions per participant

I have a repeated measures experiment with 5 factors each with 3 different levels. I'm trying to figure out a way to reduce the demand on participants by making them not have to sit through every ...
Andrew Thomas's user avatar
2 votes
1 answer
149 views

Survival analysis with missing values for some variables

I'm performing a survival analysis. I did the univariant Cox regression model for all my variables (about 40). I even performed the univariant analysis in all those variables that had missing values ...
Arthurr's user avatar
  • 73
4 votes
1 answer
133 views

how to deal with treatment dropouts in experimental designs

I conducted a between-subjects experiment with one 3-level factor (high group vs low group vs control group). Because of dropouts from the treatments, the final distribution is control group: 85 ...
dondu's user avatar
  • 41
2 votes
1 answer
59 views

Using FIML (Full Information Maximum Likelihood) for simple correlation test on data with missings?

I have a dataset with missing values. Since my main hypothesis is just a simple regression, my adviser told me to use FIML regression. So I used lavaan package: ...
Madamadam's user avatar
  • 247
2 votes
0 answers
28 views

What are the steps to simulate data showing bias when missing data are MAR

I have been trying to simulate data that shows bias in the estimated regression coefficients when there is data Missing At Random. In this case I am not interested in the estimated coefficients ...
Lynchian's user avatar
  • 155
0 votes
1 answer
16 views

When using Weighted Estimating Equations (WEE) to estimate a linear regression model with missing data, what can do if missing probability is 1

When using Weighted Estimating Equations (WEE) to estimate a linear regression model with missing data. One way is to assume the missing at random and then compute the missing probability using some ...
Fangzhi Luo's user avatar
0 votes
0 answers
4 views

Estimating a time series' likelihood of a missing observation

I have this time series of seasonal monthly data, which sometimes has missing observations. The likelihood of an observation being missing is mostly dependent on the month, but also of the value had ...
BlackNinja's user avatar
7 votes
2 answers
573 views

Missing data and maximum likelihood

I've heard it said that maximum likelihood estimation is an alternative to imputation methods for missing data. Does that mean any model fitted using maximum likelihood such as logistic regression, ...
user167591's user avatar
4 votes
4 answers
204 views

Why is multiple imputation not used more widely in Data Science? [closed]

I posted this question a few days ago on datascience.SE because I thought it was more relevant there: Why is multiple imputation not used more widely in Data Science? I have a background in ...
Joe King's user avatar
  • 3,845
1 vote
0 answers
39 views

Multiple imputation: deleting cases before imputation

Note: The question has been edited to make it more focused, and the title has been changed to make it clearer. I have read questions/answers about how to select variables for imputation. This question ...
Verity's user avatar
  • 11
1 vote
1 answer
52 views

Getting individual level scores from factor analysis with lots of missing data

I have a setting where I'm doing factor analysis in a context where I have lots of rows and where ~90% of data is missing (it's a survey of a couple hundred thousand people, each person was asked a ...
DavidShor's user avatar
  • 1,491
2 votes
1 answer
46 views

Visit and order sequence for multiple imputation in mice r

I want to use the R package MICE for Multiple Imputation and I have a question concerning the order of my dataset - regarding the order of my variables on the one hand and the order of my cases on the ...
rNewbie's user avatar
  • 23

1
2 3 4 5
33