When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing a analysis or test.

learn more… | top users | synonyms (1)

2
votes
0answers
11 views

Inspecting mechanism for missing values in categorical data without prior knowledge

Scenario I am inspecting the Soybean data set, which has a quite a number of missing values for various categorical variables. Plan My plan is to eventually perform data imputation. However, ...
1
vote
1answer
50 views

What to do with missing values?

I have missing values for some of the variables in my data. I am using pooled OLS and have 144 observations. I have missing values for three of the variables. Less than 10% of the data for each ...
0
votes
0answers
29 views

How Can I Make Sure All My .CSV Data Gets Imported as NA instead of Blank in R? [migrated]

In my dataset, I'm using have four assessments I'm trying to predict: 1 [Good] to 4 [Bad]. My model seems to be working using the polr function to predict values ...
2
votes
1answer
24 views

Missing observations in a linear mixed model

Suppose you are measuring temperature $T_{ij}$ for $i =1, \dots ,4$ subjects and $j= 1, \dots ,4$ time points. For subject 1, suppose $T_{12}$ and $T_{14}$ were missing. Would you omit the entire ...
1
vote
1answer
23 views

Choosing a regression model based on missing values

I'm trying to predict weight change with an intervention from baseline variables. Literature search yields suggests several predictors. Univariate analyses with weight change as dependent and baseline ...
3
votes
2answers
37 views

Ignorability in Rubin's theory of missing data mechanisms

I am trying to understand Rubin's theory of bayesian inference with missing data, specifically how the missing data mechanism affects the inference on a superpopulation parameter. The theory is ...
4
votes
2answers
75 views

How do I run Ordinal Logistic Regression analysis in R with both numerical / categorical values?

Base Data: I have ~1,000 people marked with assessments: '1,' [good] '2,' [middle] or '3' [bad] -- these are the values I'm trying to predict for people in the future. In addition to that, I have some ...
1
vote
1answer
18 views

Does a strong checksum provide reasonable assurance of absolute data integrity?

Does a checksum ensure absolute data integrity? That is, if a piece of data several gigabytes long changes a single bit, the odds are dependable that the checksum will be different, particularly with ...
1
vote
1answer
25 views

Is it better to use data imputation for missing data or an analysis that is not affected by missing data (e.g., HLM/mixed effects modelling)?

I have treated two groups of 100 people with different treatments. I have pre-treatment and post-treatment data for most participants (as well as 1-month follow-up. I also have weekly data for some ...
1
vote
0answers
22 views

How to handle Missing Data [closed]

Am conducting a study where I teach my participants 10 lessons. After each lesson they have to answer questions to check understanding of the lesson. Some participants missed between 2-4 lessons. How ...
2
votes
4answers
111 views

Plotting averages when there are missing values

I have count data for a number of subjects in different groups I would like to compare. The averages of the cumulative sums are shown in the figure below. As you can see the Red group has a spike at ...
0
votes
0answers
7 views

Over-estimated response

I have a large survey data set, about 15 years of data. One binary variable (yes/no) in which I'm interested was over-estimated due to a data collection error for about 1 1/2 years. I'm not sure how ...
2
votes
1answer
30 views

Dealing with poorly estimated/missing explanatory variable values in GLMs

Context I am using generalised linear models to analyse some ecological data looking at the relationship between the population density of moth larvae and the prevalence (%) of viral mortality in the ...
1
vote
0answers
30 views

Unequal variance, one replication, empty cells RCBD ANOVA

I could use some advice on how to handle this situation. I have a Randomized Complete Block Design (RCBD) ANOVA with 5 blocks and 8 treatments, the response is dry weight of some plant, there is 1 ...
3
votes
1answer
73 views

How to deal with invalid data values such as with age (e.g., -99, 0, F1)?

I have a data set that consists of 15 age values. I want to clean the data before doing anything further. I have a few questions about data cleaning and data integrity. What is the best treatment if ...
2
votes
1answer
82 views

Way to correct sample selection bias with unknown selection?

I would greatly appreciate some advise on a statistical problem that haunts me. Suppose you wish to estimate the effect of $x$ on $y$, but the probability to observe $\{y_i, x_i\}$ also depends on ...
2
votes
0answers
30 views

How to code and analyze skiped questions with SPSS

sorry if my question may seem rather silly, but I'm not well up on statistics and need a piece of advice. I have some cases in my dataset where respondents have abandoned the survey half-way, so I ...
1
vote
1answer
38 views

Missing data: what's the best approach?

In a representative sample of country population I have very few missing data, around 3%. But when I checked the missing data among communities, I found that one of them has almost 30% of data lost. ...
0
votes
0answers
18 views

Simulating EM versus listwise deletion--unexpected results

I'm preparing a presentation on missing data strategies and conducted a simulation to compare listwise deletion (LD) to the EM. Here's what I expected based on the literature: Standard errors will ...
3
votes
1answer
131 views

Statistical analysis on several data sources - possible?

I have a formulation of a statistical problem in mind and haven't been able to find any literature/references about it. As professors that I asked also couldn't help, I thought I'd ask here. Consider ...
0
votes
1answer
50 views

How to deal with missing categorical data in logistic regression models?

I am participating in a Project with data from a complex survey. We are going to analyze data from a national fertility survey. Some of the questions in the questionnaire were only asked by a ...
0
votes
1answer
28 views

How to take into consideration gaps in time series?

I've been analysing what is the probability of that measurement going up or down during a week (e.g. 4 times out of 7, I have 60% chances of my measurement going up) everyday for the last 100 days, ...
3
votes
0answers
18 views

How to analyze data with a lot of missing data (including covariate, nesting and blocking)

I have data set that was haphazardly collected to test grazing impact on soil. To test the grazing impact, 6 replicate soil samples were collected from inside a fence and 6 replicates were from ...
0
votes
0answers
34 views

Data mining of time series

I have a dataset which consist of n time series variables $X_1$..$X_n$ , and a time serie output $Y$. I would like to mine the data to find if some functions (lagged or not) of the $X_i$ can predict ...
0
votes
0answers
41 views

Using MICE in R: is it possible to impute only sub-sections of the data?

When using the mice library in R to impute data I encounter the following problem. I have a data matrix with missing information ...
0
votes
1answer
30 views

Longitudinal panel dataset: Consequences of missing values

I am analyzing a longitudinal panel dataset using OLS. The data spans around 40 years, but for some variables data was unavailable for certain categories. In most cases, the data for given category ...
0
votes
0answers
21 views

Binary logistic regression - Categorical function leaving out a category? [duplicate]

I'm still learning logistic regression, so hopefully my question makes sense. I have 10 independent variables and one dependent. The dependent and 3 of the independent variables are dichotomous (with ...
2
votes
0answers
36 views

Does it make sense to impute missing covariate data when the imputed value is a function of other covariates in the regression model?

We are building a model that adjusts for standard covariates (e.g., age, gender) and for the outcome at baseline. It would be ideal to adjust for each subject's baseline value like so: $$ Y = ...
2
votes
0answers
22 views

Creating consensus from multiple methods of measuring the same entity with some missing values

Imagine we have C cars and D drivers, and each driver takes a large subset of these C cars in order to test the rate of fuel consumption for some fixed amount of fuel (let's assume that the number of ...
1
vote
1answer
42 views

How do I run multiple linear regression with a limited data set for each subject?

I'm trying to use limited data across a range of variables to make predictions. There are ten variables and each subject has three of the ten variables defined. It's approximately random which three ...
1
vote
1answer
30 views

Missing-data in a rather small group of participants

Hi all and thank you for your valuable help all through my research work :) I am not by any means expert in statistics, I am a psychiatrist doing my phD in the area om medical education and I have ...
0
votes
0answers
17 views

dealing with missing values via pro-rating

I'm aware there are a variety of approaches of dealing with missing values. I have been asked to analyse data that contains missing values. An author of a scale has recommended pro-rating when there ...
0
votes
1answer
57 views

Use raking to impute multivariate distribution

The Problem I have a detailed dataset of 6 variabes, but for all but one year, I only have marginal distributions of 5 variables, the rest is missing. From that, I would like to obtain a full ...
0
votes
0answers
22 views

Missing data interpolation

I have two time series at different time intervals. So, for 60 seconds, I have 1000 values for X and 50 for Y. All measurements are evenly spaced over the interval. I wish to "fill in" the blanks ...
0
votes
0answers
25 views

Predicting outcomes where there's no data in rpart

I have a data set with about ~2000 data points. Of these ~1000 actually have features/data. (All 2000 data points have an outcome) Where there's no data, there is very likely a signal. In other ...
0
votes
1answer
74 views

e1071 svm predict - missing predictions

I use the following code m <- svm(x_train, y_train) current_class_prediction <- predict(m, x_cv) but predict returns 999 predictions instead of 1000: ...
2
votes
1answer
38 views

How to handle incomplete data in Kalman Filter?

What are some typical approaches to handling incomplete data in the Kalman Filter? I'm talking about the situation where some elements of the observed vector $y_t$ are missing, distinct from the case ...
13
votes
3answers
257 views

Can I reconstruct a normal distribution from sample size, and min and max values? I can use mid-point to proxy the mean

I know this might be a little ropey, statistically, but this is my problem. I have a lot of range data, that is to say the minimum, maximum and sample size of a variable. For some of these data I ...
8
votes
0answers
46 views

Incorporating more detailed explanatory variables over time

I'm trying to understand how I might best model a variable where over time I've obtained increasingly detailed predictors. For example, consider modeling recovery rates on defaulted loans. Suppose we ...
0
votes
0answers
18 views

IVEware imputation results

I have a large survey dataset (~150k individuals) on which I used IVEware to impute missing data. Most of the variables missing data are missing very small percentages of data (<5%). However, ...
1
vote
2answers
47 views

What to do if samples aren't comparable?

I have an unbalanced panel data set, which covers a set of about 10 variables I am very interested in as controls in a regression. However, only a few of these variables are available in all waves: ...
8
votes
1answer
159 views

Why is Expectation Maximization algorithm guaranteed to converge to minimum, even local?

I have read a couple of explanations of EM algorithm (e.g. from Bishop's Pattern Recognition and Machine Learning and from Roger and Gerolami First Course on Machine Learning). The derivation of EM is ...
0
votes
0answers
29 views

wavelet missing data

I have multiple time series datasets with missing data (the data is missing due to it not being recorded on these dates), and wish to perform a wavelet transformation on the data (probably a MODWT, ...
2
votes
0answers
48 views

Treatment of missing values introduced by padding lagged variables

In the case of a linear regression with lagged independent variables, what are the techniques for dealing with the NA values introduced by padding lagged variables (since t < 0 values do not ...
0
votes
1answer
31 views

multiple imputation for a longitudinal study

i have an experiment wherein respondents were tested in two time points. however, respondents were tested at t1 and t2 OR t1 and t3 OR t1 and t4. Hence, data is missing at t2,t3,and t4 for 3/4 of ...
1
vote
1answer
38 views
1
vote
1answer
58 views

Recovering true data from multiple noisy versions

I am trying to find if there is any way to get the true data from multiple noisy versions, but the true data has a peculiar property. Problem Statement Consider a matrix $F=[f_1, f_2, ... , f_n]$ ...
1
vote
1answer
35 views

How to cope with missing values in sequential data before applying moving averages (and in general)?

I have a set datasets with sequential measurements. Since the size of these sets is quite big (>80000 measurements) I decided to simplify them by applying a Simple Moving Average (SMA) and selecting ...
2
votes
0answers
37 views

What are some of the ways to deal with missing data when measuring extreme poverty?

The UNDP have reported that the millennium goal of halving the percentage of people living below 1 USD (PPP) a day has been met (compared to 1990). I was looking at the data for that indicator and ...
3
votes
0answers
64 views

Simulate missing data for multivariate distribution?

(Following the private request from a more senior CV member, I am editing this question to make it more readable and comply with CV standards). I am looking for a method to simulate multivariate, ...