0
$\begingroup$

I'm quite new in this field. I hope my question makes sense. I have a database that stores information for around 10.000.000 projects. Each project has several features (let's call them X) like number of lines of code, number of people, etc. In addition, there are some extra features (let's call them Y) that I cannot calculate directly on the 10.000.000 projects due to time and storage limitations. Therefore, I would like to take a representative sample of these projects and perform a qualitative analysis to calculate the features Y. The frequency distribution for each of the X features seems to follow an exponential distribution (I run the one-sample Kolmogorov-Smirnov test and I got a p-value < 0.001). In addition, I generated some random samples (with sizes of 100, 500, 3000 projects) to calculate the features Y, and even those features seem to follow a negative exponential distribution.

I would like to know if there is a formula to get the minimum sample size to say that my sample is representative enough. Can I use the concept of confidence level (95%) and interval of confidence (1.96) to calculate the minimum sample size even if my distribution is not normal?

$\endgroup$
  • $\begingroup$ By negative exponential dist you mean which is known as the exponential distribution which has probability density function with negative exponent lambda. However, I dont's get your reasoning with that K-S test. $\endgroup$ – Germaniawerks Mar 28 '14 at 10:03
  • $\begingroup$ If you apply random sample it will certainly be representative enough at that sample sizes. This is hard to achieve in other surveys where the true random sampling cannot be readily applied. $\endgroup$ – Germaniawerks Mar 28 '14 at 10:06
  • $\begingroup$ @Germaniawerks, thank you for answering. Yes, I meant exponential distribution. I think it's wrong to use the K-S test over all the projects, but I wanted to know which curve was close to the frequency distribution of the X features. $\endgroup$ – user2314405 Mar 28 '14 at 10:29
  • $\begingroup$ One more question, can I use the concept of confidence level (95%) and interval of confidence (1.96) to calculate the minimum sample size even if my distribution is not normal? $\endgroup$ – user2314405 Mar 28 '14 at 12:16
  • $\begingroup$ Yes, you can do it with any distribution with the mean defined and finite central moments by Central Limit Theorem. $\endgroup$ – Germaniawerks Mar 29 '14 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.