How are sample size and confidence related when calculating Sample mean? Maybe I'm getting ahead of myself and I should just keep reading my textbook, but I'm getting confused about something.
Let's say a friend gives me 10,000 integers that he claims are normally distributed. My goal is to estimate the parameter $\mu$ of that initial distribution. I do the following:
Divide the 10,000 into 100 groups of size 100, and compute the average of each group, to give me 100 averages $a_1, a_2, \ldots, a_{100}$. These numbers should also be normally distributed with the same mean $u$, by the Central Limit Theorem? Do I then tell me friend that my guess for $u$ is simply $A = (a_1+\cdots+a_{100})/100$ ? If so, then the size of the groups I picked (groups of size 100) has no impact on the returned value.
Am I way off? Can I say anything like "With P % confidence, the original $\mu$ was $A$"?
 A: The way that you have stated the problem, A is simply the mean of all 10,000 values--the best estimate you will have of the true value of u, regardless of the nature of the distribution from which they have been sampled. (I assume that the distribution has finite mean and variance.)
For confidence limits on the true value of u, you have two general choices. 
One is based on theory. The Central Limit Theorem states that, as the number of observations increases, the distribution of differences (over multiple samplings with the same number of observations) between the observed means and u will tend toward a normal distribution even if the underlying distribution is not normal. The so-called standard error of the mean (standard deviation of the original 10,000 values divided by the square root of the number of values) is an estimate of the standard deviation of that distribution among multiple samplings. Tables of normal distributions then provide estimated probabilities that the true mean value is any specified distance from the observed mean. 
This leads to a generally useful rule of thumb: the precision of an estimate is typically related to the square root of the number of observations, so that 10,000 observations is 10 times better than 100 observations.
If the underlying distribution is not normal, the the Central Limit Theorem is only a limiting case that might not hold well for a particular sample size from a distribution. There is a second, empirical, way to get estimates of confidence in mean values directly from your original observations, a technique called bootstrapping.
Bootstrapping is based on the idea that the best estimate you have of the true underlying distribution is your set of observed values. In your case, you take many independent random samples of 10,000--with replacement--from the 10,000 observed values. Thus any particular observed value might be taken more than once or not at all in a particular re-sampling. You then use the distribution of observed mean values among the re-samples to estimate your confidence in the original observed mean value.
