1
$\begingroup$

I am training Recurrent Neural Network models (including LSTMs) on a dataset that includes 6-10 variables. Each variable is a properly formatted numerical measurement (ie: length, pressure, temperature, etc). The raw data does not have a measurement of each variable at each timestamp. I have aggregated the raw data into a Pandas Data Frame, so that each row corresponds to a timestamp when at least one variable was recorded, but many rows are missing values for one or more variables.

My first idea was to us KNN methods to impute values, but this method gets weaker with as we reduce the number of variables available to use for finding similar data point clusters. Imputing based on mean/medians seems like a poor idea due to the expected correlated fluctuation of values. I have run models with these rows (times when multiple variables are missing values) dropped, but I would like to see if a model would perform better with these imputed steps (as I assume it would).

${\bf My} \space{} {\bf Question:} $
I need to impute values for these cases when missing multiple variable values. Are there imputation methods for similar cases which the Cross Validated Community find stronger than KNN?

Improve this question
$\endgroup$
8
  • $\begingroup$ If you don't have all that many missing values, you could also just drop those rows. How many rows do you have? And how many rows have missing values? $\endgroup$
    – Adrian Keister
    Jul 22 '21 at 14:48
  • $\begingroup$ I can adjust my time frames, but most recently was working with a data set with around 1200 rows, about 47% had a raw value for each variable, so about 53% missing values $\endgroup$
    – jros
    Jul 22 '21 at 14:50
  • $\begingroup$ Ah, I see. That's too much missing data for the number of rows you have to make dropping feasible. $\endgroup$
    – Adrian Keister
    Jul 22 '21 at 14:51
  • $\begingroup$ I thought so, I appreciate hearing that someone else came to a similar conclusion $\endgroup$
    – jros
    Jul 22 '21 at 14:53
  • $\begingroup$ Would getting more data allow you to greatly change the ratio of rows to rows with missing data? Or how much control do you have over the data acquisition? $\endgroup$
    – Adrian Keister
    Jul 22 '21 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.