in concrete terms, what does the mean of a sample actually tell you about the population? Suppose you take a sample of size 3 with values 0, 0 and 1.  (3 is the minimum size to make the sample asymmetrical and make the question non-trivial.)  Ever since my first stats class, I've been vaguely confused because I didn't understand: why is it useful to calculate the mean of this sample, 1/3?
At first I naively thought that if this meant that if you assumed the background population had some standard deviation (e.g. 1), that 1/3 would be the value of the population mean that would maximize the probability of getting the observed outcome.  (Of course I mean the probability of getting the observed outcome within some arbitrarily small delta, since the probability of any specific outcome is zero.)  i.e. for my sample and assuming standard deviation 1, the value k that maximizes the value of f(k)*f(k)*f(1-k) where f is the standard normal distribution.  But presumably that's not exactly 1/3, is it?  (By the way, is there a technical term for this "observed outcome probability maximizer" value?)
[Edited to add: I was wrong.  I explicitly f(k)*f(k)*f(1-k) where f is the standard normal distribution, and it has a maximum at 1/3.  This is convenient, although it was far from obvious to me at the outset that this would be the case. This renders the rest of the question moot.]
But if 1/3 is not the value that maximizes the probability of the observed outcome, then what good is it?
Is it simply the case that before everyone had a computer in their pocket, calculating the "observed outcome probability maximizer" value was too hard, and calculating the mean was good enough?  And now that we all have pocket computers, we should just go with the "observed outcome probability maximizer" instead, and calculating the mean is obsolete?  If not, then what does "1/3" tell you about the background population?
 A: 
1/3 would the value of the population mean that would maximize the probability of getting the observed outcome

This is a more profound statement than you might realize, as it alludes to maximum likelihood estimation, a common estimation method in statistics. Being a little loose, there is a technical sense in which the calculated $\bar x$ is the most likely value of the population parameter in order for you to obtain the data you obtained (and it isn't always true, but it's true under nice-enough conditions, such as normality). That sounds good, right? And that's why statisticians like maximum likelihood estimation.
To address your final paragraph, you're onto something by saying that finding the "probability maximizer" (likelihood estimator) is a good approach, perhaps better than the sample mean, but the sample mean often is exactly that likelihood maximizer by being the maximum likelihood estimate!
Getting away from maximum likelihood estimation in particular, "estimating a population parameter" means "guessing the correct value of the population parameter". There are ways of guessing that we know have desirable properties ("good" estimates of the unknown value). Since, the usual sample mean $\bar X$ has a number of desirable properties, we often like $\bar x$ as our estimate of $\mu$.
