Open Source Project:
In short (and roughly described, only to the purpose of clarification) open source projects allow me to have access to the code of a certain 'program'.
Motivation and Background (so I can understand your answer)
I have a undergoing majors in CS, expect a very high level course in statistics and self-learning using books. Do also consider I am trying to be practical here as I will do most of the calculations in R (that doesn't means your explanation needs to be in R), but its just a pointer that I am concerned on understand what I am doing without going to every single detail.
- It might have flaws on the theory along the path, much I salvaged from books on my own. Please feel free to point my mistakes.
According to Blalock Jr, Social Statistics, Mc Gran Hill we have that The Central Limit Theorem is as follows:
If repeated random samples of size N are drawn from any population (of what-ever form) having a mean $\mu$ and a variance $\sigma^2$, then as N becomes large, the sampling distribution of sample means approaches normality, with mean $\mu$ and variance $\sigma^2/N$.
Now, one of my interests is analyzing open source data using some statistics rigor. I wanted to use the Central Limit Theorem to reason about data, as one of the most trouble some issues I have been dealing with is that they do not follow a normal distribution.
My background course in statistics used as textbook Probability and Statistics by Paul Meyer. There and probably anywhere else we have the definition of random variables, however with a much great emphasis.
The first part of the approach I wanted to use starts with a doubt at this point:
The first book approaches binomial distributions, t student distribution, chi square, ANOVA, and others after talking about hypothesis testing. Thus, as defined by the book my concern is:
- To define and follow the basic rules of an experiment and given the assumptions of one of these distributions.
- Consider one of them to be my hypothesis and the remaining I must assume to be true (which constitutes my model).
The second book however is more concerned on observing each distribution as a random variable and does not follow the reasoning of choosing a distribution to verify a hypothesis when introducing the distributions. In fact, hypothesis testing is introduced much later.
- Question 1: Are these two different approaches to analyze data or am I being mislead by the order of subjects being taught on different textbooks?
Let me assume at this point despite question 1 that I decided to define the experiment first as wanting to test my hypothesis on an open source data.
Pang, on Introduction to Data Mining book suggest that there are two types of data:
- Observational Data: Data was already available and we are trying to understand something out of it.
- Experimental Data: An experiment is settled so we can control each of the variables during the experiment, and thus based on my description of the book above of choosing one distribution assumption as hypothesis and the remaining as true seems appropriated.
- Question 2: Using hypothesis testing, given this definition would be inappropriate to approach open source projects given that they fall under observational data or is this tolerable?
Assuming that anything at this point poses a serious violation, I moved on the approach to believe that if observing, for example, the number of open defects or closed defects on a given open source project doesn't follow a normal distribution, I could obtain one out of the distribution of the mean of the samples and use it instead.
After this I would try to see which distribution seems more appropriated based on its assumptions from the parametric 'family', since the normal distribution constraint which is the most troublesome one would be gone due to the Central Limit Theorem.
The first book also suggests that N>100 is enough for a given sample to be considered a large N, so for a open source project this is requirement is easily match.
- Question 3: When we go about using the distribution of the means, can we actually use it for creating hypothesis or I would need to have the 'original data' to follow the normal distribution? (In this example open defects and closed defects).
- Question 4: Which would be better: Using the whole open source project to this point as one single huge sample or extract a smaller (which size?) sample out of it?
Question 3 is actually of the most particular of interest of mine since I am unable to use any parametric distribution for hypothesis testing with the 'raw data'.
Question 5: How much do I lose on being able to infer only from the distribution of the samples instead of, say, the real distribution of open defects and closed defects?
Question 6: Would be better to attempt data analysis from the open source code using non-parametric methods? Two possibilities that I have salvaged from books seemed possible here:
- Attempt to convert a certain observation from the data to Binomial distribution (leaving to a binary mapping) so that the normal distribution constraint does not apply.
- Going for ordinal scales using ranking, in exchange losing statistical inference power (How much, in this case does usually is?).