I am learning how to handle missing values in a dataset. I have a table with ~1million entries. At the moment I am trying to deal with a small number of missing values.

My data concerns a bicycle-share system and my missing values are start & end locations.

Data: missing starting stations, only 7 values

enter image description here

Data: missing ending station, 24 values altogether

enter image description here

I deleted the rows where both values were missing since I figured there wasn't enough information to make an accurate prediction. However, using duration and possibly timestamp I figure I can reasonably fill in the missing values.

I've identified these values as MCAR because I can't see any systematic relationships as to why they would be missing. Furthermore, I notice that because duration (measured in seconds) is relatively low (1000s = 16min) I can accurately guess which other stations have been reached under that time.

I know I could probably drop these values because they are relatively few. But if I wanted to fill them in, what is the best method to achieve this?


I don't think you can ever know that values are MCAR because you would have to know what those values were in the first place in order to determine that. Unless the missingness comes from malfunctioning equipment (and even then, there should be some hesitancy), you shouldn't conclude data is MCAR.

So you have some missing data on birth date, starting station, and ending station. For birth date, it might be reasonable to just impute with the mean birth year. You could also do something like MICE, where the remaining data are used to impute the missing data. Maybe older people tend to use the bike for a shorter amount of time on average. This, MICE would look at duration of the rental to impute age.

So far as stations are concerned, those are categorical. You could impute with the mode, or you could reason about the missingness. If you have rental duration, you could argue that bikes can only travel so fast, and so stations too far away are excluded from possible imputed values. If you are imputing end stations, you could look at the start stations and ask where most people from that station go, and vice versa if you're imputing start station.

In short, imputation algorithms are nice, but you really have to think about the problem you are trying to solve. You can rarely conclude that data is MCAR since knowing that would require the mechanism for the missingness.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.