Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

When the data present lack of information (gaps), i.e., are not complete. Hence, it is important to consider this feature when performing an analysis or test.

2
votes
0answers
30 views

Time Series Prediction Model for Home Prices

I am building a time series model to predict the zillow home prices for march 2019.I have data for each zip code from the year 1993 - 2018 and i have prices for every month.I was trying to use ARIIMA ...
0
votes
1answer
47 views

missing value problem [on hold]

I have survival data. However,there are missing values in both categorical and numerical data list. That's, in each column, approximately more than two values are missing. Now, I want to obtain ...
0
votes
0answers
11 views

How do you deal with MNAR in randomForests?

The problem is the following: In a survey, people are asked to note different qualities of a firm with a note between 0 and 10. For some questions, the interviewer knows that the interviewee can't ...
0
votes
1answer
15 views

Polluting a dataset with “non-determinables”

Let's say I'm working on solving sudoku puzzles with machine learning. Now, plenty of good methods exist for solving sudoku algorithmically, no machine learning required, but let's play along to get ...
1
vote
1answer
78 views

Predicting spendings overall and spendings for subcategories

I have a Dataset containing information about spendings of customers in various shops. There are 10 spending variables related to some categories (like spendings on clothing, spendings on hardware, ...
0
votes
0answers
20 views

Cross tab with missing values

I have a following kind of problem. I have several countries. For every country I know the cumulative value of the variable of interest after four periods. I also know the cross-country sum for every ...
0
votes
0answers
8 views

what statistical test to use in comparing class grades (A, B, C)? How to use GraphPad PRISM for doing so?

we've got 2 school classes. each class have 15 students. each of them had 3 tests, and got A, B, or C grades. (Note: 2 of the students just did one of the tests, so we have missing data). which ...
1
vote
0answers
30 views

Handling N/As which are not missing values in a classification model

I am facing a problem of N/As in classification model and haven't found similar problems. My dataset contains data on scores of students sitting an entrance examination. The exam contains $8$ ...
3
votes
0answers
47 views

Imputation and nested cross-validation

I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
1
vote
0answers
17 views

Logistic regression where there are unreported failures

Hello: Let's say you have a large number of reported results from a cooperative game. The data consists of a number of independent variables, such as number of players, choice of opposition, etc., ...
1
vote
1answer
25 views

Treating a column containing null values for random forest when they should be null (not missing)

Suppose I have columns like this date_ago happened_or_not 0 3.0 1 1 1.0 1 2 NaN 0 3 NaN 0 4 3.0 1 5 5.0 1 6 NaN 0 7 NaN 0 8 2.0 1 Now the ...
1
vote
1answer
45 views

EM Algorithm and Pattern Mixture Modelling - Is this a correct understanding?

I'm quite new to both the EM algorithm and pattern mixture models (for NMAR data). I am hoping someone can confirm my understanding is correct, and if not, let me know how to do it correctly. The ...
0
votes
1answer
24 views

Can I correct for randomly missing data where missingness is has a known relationship to the error term?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the ...
0
votes
1answer
24 views

how can this missing observation model be extended to include cases where sigma is a function of other variables?

Richard McElreath's blog entry Algebra and the Missing Oxen describes a simple missing observation model in RStan. At the end of the blog, he says it can be extended easily to cases in which the ...
1
vote
0answers
21 views

Subscript error from blau's index using the “diversity” function in R [migrated]

I am trying to get the blau's index using the "diversity" function in R. However, I constantly get the error using the code and data I suppose to be ok as below. ...
1
vote
1answer
16 views

How to handle variable size input data (incomplete) to build/train a NN for regression?

Suppose you have the classical example of predicting house prices and you have as input features area size, built year, number of previous owners, city, number of floors, number of bedrooms, etc. But ...
0
votes
1answer
52 views

Follow up medical study with missing data

I am analyzing some patient data for a medical study that has a duration of several years. Once a year, the patients are expected to visit the doctor, where they get four treatments, say A, B, C, D. ...
0
votes
1answer
32 views

How to deal with missingness of dependent variable in unbalanced probit model

I am trying to estimate a probit model on the probability of suicide over the next year in a population. Unfortunately for this research, suicide rates are very very low so the probability of suicide ...
2
votes
0answers
31 views

Best practices for dealing with missing data [closed]

There are a lot of threads on here about missing data, but I haven't found something that really gets at the best practices, and discussion of why to choose one approach over another. This is such a ...
5
votes
4answers
242 views

Joint probability of multivariate normal distributions with missing dimensions

Suppose I conduct two experiments, each measuring a subset of possible parameters. From experiment #1 I measure two parameters and estimate the multivariate normal distribution $$ \mathbf{X}_1=\left [...
1
vote
0answers
27 views

Time Series sampled at varying frequency - Employ Linear Mixed Model to compare trends?

I hope that this question has not be asked like this elsewhere, if so I could not find it during my google research.. I have the following problem: I have data sampled from different sensors("ID") ...
0
votes
0answers
15 views

Regression for Interval and Continuous Predictors - Continuous Target Variable

I have in my dataset a continuous target variable sales and two predictors- one continuous and one interval. The continuous variable is ...
2
votes
0answers
110 views

Missing data imputation that can handle large data

I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills ...
0
votes
0answers
21 views

can I fill missing values by using target variable?

I have a 3 column data with 2 features and 1 target variable. But the first features (numeric) have a large number of missing values. If I use kNN to fill in the missing values, I am wondering can I ...
0
votes
0answers
19 views

what the missing data value method should i use?

what the best missing value method should I use which contains 9.89%-10% missing data? is it still okay to use mean? anyway my data is at Missing not at Random (NMAR) category. thank you so much for ...
0
votes
2answers
33 views

Machine Learning Framework that accepts Inputs with missing data and Outputs predicted values for that missing data

Imagine a dataset of with values for 10 features for 100,000 samples. Some feature values are missing at random from some samples. I would like to use this incomplete data set to train a single model ...
0
votes
1answer
8 views

Sensitivity of Weibull survival model to early missing data

I have a few datasets of survival times that appears to be very well-modelled by Weibull distributions; I am mostly interested in whether $k$ is above, below or around 1. However, there is a real ...
1
vote
0answers
12 views

How do I check whether two variables overlap in the information they contain?

I'm looking for a method to determine whether two variables contain the 'same' information. The information I'm using has different types of 'missings' or special meanings represented by specific ...
0
votes
0answers
27 views

How to treat “not applicable” value?

Even though there already exist questions with a similar topic, I still have not found an answer to my question. I am working on e-transparency of nonprofit organizations. I try to build an index to ...
0
votes
1answer
28 views

Two separate linear models

I am trying to fit the following linear model: ...
1
vote
0answers
33 views

Complex neural network design - 250K+ rows / Lot of missing variables

For the past years I have been developing different type of Neural Networks with great success. For the past few weeks I have been working on a (big) project I am kind of struggling with. I hope by ...
0
votes
0answers
16 views

Trajectory modeling with small dataset and missingness

I am struggling with a small dataset (260 records) for trajectory modeling. At the 3rd time point, there are 101 cases that are missing critical data points due to attrition. Does it still make sense ...
3
votes
0answers
32 views

Regression on Numerical Variable where NA is Present Using R [closed]

I am working on building a multivariable regression using a dataset in R. That dataset contains a variable that is generally an integer, but at times can have NA values. Because the NA values make ...
0
votes
1answer
38 views

Calculate the implied correlation for missing cells in a correlation matrix in R

I have a correlation matrix in R. Many of the correlations are specified, but there are some that are "NA". eg, A __ B __ C A 100% NA 25% B NA 100% 50% C 25% 50% ...
2
votes
1answer
48 views

Why don't people impute missing exposure data in database studies?

Investigators doing studies in large databases (e.g., EMR) in which there is often a lot of missing data usually (in my experience) want to exclude all subjects missing the exposure or outcome of ...
0
votes
0answers
12 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
-1
votes
1answer
18 views

K-means in R: complete case analysis followed by nearest-neighbor assignment for partial data

I have a dataset of 3K observations with only 162 being a complete case. I have read here that it is possible to run knn on the complete cases and then conduct a nearest neighbour assignment for ...
0
votes
0answers
52 views

Appropriate multiple imputation method for longitudinal data (R package mice)

I'm analyzing a dataset from a longitudinal study aimed at finding if a set of predictors is associated with the trajectories of an outcome, which is measured each day for seven days. The dataset is ...
0
votes
0answers
66 views

Proof: comparison between the statistical efficiency of two MLEs

Suppose that we have a simple random sample of size $n$. Based on this sample, we construct two log-likelihood functions. The first one is, \begin{eqnarray*} l_1 = \sum_{i=1}^r{\bigg[\ln f_\beta(y_i|...
2
votes
1answer
52 views

What type of missingness is this?

I have some missing data for a particular item on a 5-item measure, which is called Attitudes Towards Ageing. Several participants have declined to respond to the item because of the wording of the ...
1
vote
0answers
29 views

How to predict with label missing

I have be given a set of data where most label data (dependent variable) is missing (NaN). Precisely on 100000 rows only 1412 contain a 1 or 0 (the others contain NaN). I was first thinking "let's ...
2
votes
1answer
113 views

Mixed models in R : Compare measurements over several time points with missing data in 3 populations [closed]

I have data that look like the example below. There are 3 different groups (g1, g2 and ...
1
vote
0answers
17 views

Missing measurements in nonlinear chemical processes

I am using an imputation method to handle missing measurements;TSR.The prediction model used is LW-PLS. Based on the my results, the RMSE increases when the percentage of missing measurements ...
1
vote
4answers
100 views

How to best code the N/A response of the Likert-type rating scale?

Say I have a dataset of people's opinion/"rating" on something, and they have to choose 1 out of 5 possible answers for each question - Very happy, ...
1
vote
0answers
52 views

How to apply a model built using Multiple Imputation to predict on dataset with missing data?

I understand that Professor Harrell recommends using the target variable in Multiple Imputation. An example using aregImpute of the rms package is in his lecture notes: http://hbiostat.org/doc/rms....
0
votes
0answers
17 views

SVM prediction with historical data that starts at different times

I am working with an SVM prediction model that uses historical data starting in 2016. I now have new data that I want to use with the SVM model but it does not start until 2017. How can I use the two ...
0
votes
0answers
12 views

How to model a field whose feature values can be all 0 in regression model

I want to predict price of products. For each product, I use one-hot encoding to model their features. These features come from a limited set of fields (i.e., product attributes). For example, a field ...
1
vote
2answers
38 views

Why are missing values MNAR harder to impute than MCAR or MAR?

Reading papers related to the imputation of missing values related to the -omics field, systematically imputation algorithms were less accurate when imputing MNAR compared to imputing MCAR. My ...
0
votes
1answer
25 views

Imputing values with linear regression, valid strategy or creating biases?

I am practicing on the titanic competition from kaggle. In the dataset the Age variable has a number of missing values and I am now left with the choice of what to ...
0
votes
0answers
19 views

Measurements to deterministic value

I have a number of measurements of two variables: the number of products, the weight. Sometimes the weight is missing and sometimes the number of products is missing. I want to use the given data to ...