Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [missing-data]

When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing an analysis or test.

2
votes
0answers
27 views

How to fill NaN values that exist because there are no measures of certain features?

I'm currently doing a ML project (the goal is simply to clean the data set and apply some of the models we learned , like Random Forests, Ensemble learning, etc, and test the results) for a class and ...
1
vote
1answer
29 views

How to handle missing data in machine learning [closed]

i know how to find and fill the missing values. But i am not sure when to fill the values with min., max. , mean, median or mode. Can someone help me to understand on what basis i can decide , i have ...
0
votes
0answers
18 views

How do I impute data that is only partially missing?

I want to impute some missing data. I am interested in the number of months someone was unemployed between ages 18-21. This variable is bounded at 0-48. However, for some individuals, I have partial ...
0
votes
1answer
31 views

How to simulate discrete data from a Poisson distribution? [closed]

I want to use R to simulate discrete data with missing values from a Poisson distribution. I have tried this: simdata<-rpois(1000,2) but when I checked if there ...
1
vote
0answers
20 views

Dummy variable method for missing data in ML/predictive models

I'm looking for references on the use of zero-imputation with dummy-variable augmentation in the context of predictive models and MNAR missingness. Basically, the idea is that one imputes zero for ...
0
votes
0answers
16 views

SAS proc mianalyze edf

We have a cluster randomised trial with a small number of clusters, The primary endpoint is measured at follow-up and we have missing data. We proposed to conduct a linear mixed model including ...
0
votes
0answers
12 views

Can iterative PCA be applied to grouped data?

My data set consist of 156 individuals with fifteen variables. The variables consist of one body mass (dependent) variable and fourteen (independent) variables of different bird bone dimensions (of ...
3
votes
0answers
23 views

Is Structurally Missing Data a subset of Missing at Random Data?

I'm quite familiar with MCAR, MAR and MNAR (NMAR) data but I have just come across a new (for me) term: Structurally Missing Data (SMD). According to this page, Structurally missing data is data ...
0
votes
0answers
12 views

Combining many sets of data with different intervals

I have a dataset to analyse which goes somewhat beyond my stats knowledge. Rather than explain the actual data I've reworded into something equivalent (it's not actually about crime). What I have: ...
0
votes
1answer
32 views

Constructing 3-way ANOVA when design is not fully factorial

I have conducted an experiment measuring individual sizes as a function of two categorical variables (A and B), each with three levels (1, 2, 3). The combination A3:B3 is a control group. This ...
0
votes
0answers
8 views

missing data in explanatory variable due to periodicity of survey

I have a pooled data set of 20 states for the years 1994-2017. In order to run a regression I have used explanatory variable(number of self employed people) from a different survey. This is one of my ...
0
votes
1answer
9 views

Is the R package Multivariate Imputation by Chained Equations suitable to fill in repeated measures survey data?

I am dealing with a dataset that contains demographic covariates (constant across time) and dummy variables for subjects for six repeated measures in wide format. I.e. a row represents a single ...
0
votes
0answers
11 views

When binary predictor = 0, all other predictors = NA - what model structure do I need?

I have a genetics dataset which I want to build a model for. The dependent variable y is case or controls status (binary). The first independent variable x1 is whether or not they have a variant in ...
1
vote
0answers
27 views

Classification where classes do not use all features?

I am attempting a classification task whereby the features used to describe classes are not all being used. For instance, Class A does not use feature 2, class B does not use feature 4, and class C ...
0
votes
0answers
17 views

All items missing for various questionnaires

this is actually the first time I'm working on a big dataset and I really hope someone can give me some advice on how to handle missing data. I tried to find information regarding my problem but can't ...
0
votes
0answers
17 views

How to evaluate relationships with categorical variables?

I am working on the BigMart dataset. One of the variables has a substantial amount of missing values. I'd like to check whether these are missing completely at random. There are 12 variables, 4 are ...
0
votes
0answers
18 views

Why Weighted Non Negative Matrix (WNMF) is better than standard NMF?

I have tried so hard to find answer of my subject query but have not found a single helpful source so far. Please let me know with the help of basic example that why WNMF is better than NMF in case of ...
1
vote
0answers
27 views

Polynomial regression for missing value imputation

I am trying to impute missing values by fitting higher degree polynomial. I have highly autocorralated time series meaning each value at t must be close to t-1. There are some noise and missing ...
1
vote
1answer
96 views

glm returns NA as coefficient for logistic regression

I am fitting a logistic regression for the response variable- 0 or 1. There are 15 explanatory variables- 10 are continuous and 5 are categorical with 3 levels each. I checked collinearity among the ...
0
votes
0answers
29 views

How do I include new features that did not exist before into an existing model?

I have a binary classification model predicting sports result with features covering 10 years worth of matches. However, how would I feed new tracking data that is only limited to the last 3 years. ...
0
votes
1answer
74 views

Mann Kendall test in R with missing data

I am trying to run a Mann Kendall test on community dissimilarity for streams sampled roughly monthly, but with NA values for some months where conditions prevented data collection. From what I am ...
0
votes
0answers
17 views

How does survfit (survival package) deal with NAs in time?

I am a beginner in survival analysis so please excuse if this question is obvious. Following this tutorial I want to calculate survival probability for bird clutches using ...
0
votes
1answer
28 views

Recommended methods for replacing missing data?

We conducted a pre-post attitudinal survey measuring “Attitudes toward STEM” (28-items; α > .90) and “Multi-Ethnic Identity” (12-items; α > .90) among 50 middle schoolers. Students skipped items ...
1
vote
1answer
47 views

Autocorrelation with missing Data

How should missing observations be handled to create ACF-Plots? Let's assume we have a time series t = [1, 1, 1, 2, NaN, 3, 2, NaN, ...]. What schould be done with these missing data points? I have ...
4
votes
2answers
144 views

MANOVA with variables from different datasets

I conducted several experiments where I have one underlying independent variable (tree species, IV). Each of these experiments gave me one dependent variable (DV), like bark pH, rugosity or the water-...
0
votes
0answers
13 views

How to deal with the problem of censoring in tree-based machine learning?

Censoring occurs when the outcome of a unit is not observed, because the unit is lost to follow up in a longitudinal study. Let $Y_t$ be the survival time at $t$. Then a unit is censored at $t'$ if $...
0
votes
0answers
13 views

Data structure for share capital increase

I have the following dataset for share capital increases for the 2010-2017 period: ...
0
votes
0answers
14 views

Recursive least square with incomplete data

If we have a system $\mathbf{Y}_t = \mathbf{A}_0 + \mathbf{v}_t$, in which $\mathbf{v}_t$ is the noise. Using the first $t$ samples, we can estimate $\mathbf{A}_0$ by the arithmetic (sample) mean, ...
0
votes
1answer
48 views

Paired t-test following main effect of mixed model with missing data

I am analyzing data for a repeated measures study with missing data. For example here is a 3 X 3 experimental design with three conditions and 3-time measures: I am using a mixed model for the main ...
0
votes
1answer
38 views

Missing data in regression

I am researching the effect of different marketing mix variables (e.g., price promotion, innovation) on the market share. More specifically, I want to analyze the effect of different marketing mix ...
0
votes
0answers
21 views

MissMech package in R

I am exploring whether my data are MCAR. Using your MissMech package I tested successfully for MCAR with the Anderson-Darling k-sample test since my data are multivariate non normal. The p-value was ....
2
votes
0answers
34 views

Time Series Prediction Model for Home Prices

I am building a time series model to predict the zillow home prices for march 2019.I have data for each zip code from the year 1993 - 2018 and i have prices for every month.I was trying to use ARIIMA ...
0
votes
1answer
51 views

missing value problem [closed]

I have survival data. However,there are missing values in both categorical and numerical data list. That's, in each column, approximately more than two values are missing. Now, I want to obtain ...
0
votes
1answer
16 views

Polluting a dataset with “non-determinables”

Let's say I'm working on solving sudoku puzzles with machine learning. Now, plenty of good methods exist for solving sudoku algorithmically, no machine learning required, but let's play along to get ...
1
vote
1answer
170 views

Predicting spendings overall and spendings for subcategories

I have a Dataset containing information about spendings of customers in various shops. There are 10 spending variables related to some categories (like spendings on clothing, spendings on hardware, ...
1
vote
0answers
33 views

Handling N/As which are not missing values in a classification model

I am facing a problem of N/As in classification model and haven't found similar problems. My dataset contains data on scores of students sitting an entrance examination. The exam contains $8$ ...
3
votes
0answers
57 views

Imputation and nested cross-validation

I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
1
vote
0answers
19 views

Logistic regression where there are unreported failures

Hello: Let's say you have a large number of reported results from a cooperative game. The data consists of a number of independent variables, such as number of players, choice of opposition, etc., ...
1
vote
1answer
45 views

Treating a column containing null values for random forest when they should be null (not missing)

Suppose I have columns like this date_ago happened_or_not 0 3.0 1 1 1.0 1 2 NaN 0 3 NaN 0 4 3.0 1 5 5.0 1 6 NaN 0 7 NaN 0 8 2.0 1 Now the ...
1
vote
1answer
50 views

EM Algorithm and Pattern Mixture Modelling - Is this a correct understanding?

I'm quite new to both the EM algorithm and pattern mixture models (for NMAR data). I am hoping someone can confirm my understanding is correct, and if not, let me know how to do it correctly. The ...
0
votes
1answer
26 views

Can I correct for randomly missing data where missingness is has a known relationship to the error term?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the ...
0
votes
1answer
29 views

how can this missing observation model be extended to include cases where sigma is a function of other variables?

Richard McElreath's blog entry Algebra and the Missing Oxen describes a simple missing observation model in RStan. At the end of the blog, he says it can be extended easily to cases in which the ...
1
vote
1answer
24 views

How to handle variable size input data (incomplete) to build/train a NN for regression?

Suppose you have the classical example of predicting house prices and you have as input features area size, built year, number of previous owners, city, number of floors, number of bedrooms, etc. But ...
0
votes
1answer
54 views

Follow up medical study with missing data

I am analyzing some patient data for a medical study that has a duration of several years. Once a year, the patients are expected to visit the doctor, where they get four treatments, say A, B, C, D. ...
0
votes
1answer
38 views

How to deal with missingness of dependent variable in unbalanced probit model

I am trying to estimate a probit model on the probability of suicide over the next year in a population. Unfortunately for this research, suicide rates are very very low so the probability of suicide ...
2
votes
0answers
92 views

Best practices for dealing with missing data [closed]

There are a lot of threads on here about missing data, but I haven't found something that really gets at the best practices, and discussion of why to choose one approach over another. This is such a ...
5
votes
4answers
260 views

Joint probability of multivariate normal distributions with missing dimensions

Suppose I conduct two experiments, each measuring a subset of possible parameters. From experiment #1 I measure two parameters and estimate the multivariate normal distribution $$ \mathbf{X}_1=\left [...
1
vote
0answers
39 views

Time Series sampled at varying frequency - Employ Linear Mixed Model to compare trends?

I hope that this question has not be asked like this elsewhere, if so I could not find it during my google research.. I have the following problem: I have data sampled from different sensors("ID") ...
0
votes
0answers
22 views

Regression for Interval and Continuous Predictors - Continuous Target Variable

I have in my dataset a continuous target variable sales and two predictors- one continuous and one interval. The continuous variable is ...
2
votes
0answers
128 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...