1
$\begingroup$

I have a dataset of around 15000 subscribers to a Software, and I also have some meta data collected for each of these subscribers based on which I need to do some analysis.

I need to decide whether there is a difference between larger subscribers and smaller subscribers on using a specific third party service. However all of the meta data that contains the 'size of the subscriber' and 'third party service use' is missing for a big portion of subscribers (more than 30%).

However the data on the behaviour of the subscriber within the software is complete.

I was wondering what is the best way to have an unbiased analysis. Most of the methods mentioned online such as imputation , KNN are for handling smaller number of missing data.

Considering that the proportion of missing data is high in our case, is it a viable way to assume the complete dataset as the population and it’s corresponding meta data with large missing records as the survey data and try to use survey weighting methods for removing the bias in estimation and the exploratory analysis?

In other words I am aiming to compare the behaviour of the subscribers with complete data with the ones with missing data in the software and conclude that the subscribers that have similar behaviour in the software (Population variables) have similar size and usage of the third party service (Meta/Survey data).

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you missing ALL metadata for 30% of your dataset? If so, there's not much you can do with this data. Or are you missing different pieces of information from different points? $\endgroup$
    – mkt
    Commented Sep 24, 2019 at 10:43
  • $\begingroup$ I am missing all variables of the metadata for 30% of the dataset, but I have the subscribers behaviour in the software for that 30%. $\endgroup$
    – Zanboor
    Commented Sep 24, 2019 at 11:23
  • $\begingroup$ Please edit your question to explain this. And generally to add more detail for e.g. how many data points do you have? What is the goal of your analysis? $\endgroup$
    – mkt
    Commented Sep 24, 2019 at 11:29
  • 1
    $\begingroup$ Thanks a lot for the feedback @mkt, I just edited the question. $\endgroup$
    – Zanboor
    Commented Sep 24, 2019 at 12:13

1 Answer 1

Reset to default
1
$\begingroup$

There is no particular reason to avoid imputation with 30% missingness. Of course there will be some extra imprecision but there is no free lunch. To quote from a paper by Ian White and colleagues on multiple imputation by chained equations

It would seem wise to take special care if more than 30–50 per cent missing data are to be imputed.

They also refer to an example with 70% missing although that did run into some instructive problems. Their article is entitled "Multiple imputations using chained equations: issues and guidance for practice" and available here but from behind a paywall as far as I can determine.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.