As mentioned in existing answers, the main issue with a small sample size is low statistical power. There are various rules of thumb regarding what is acceptable statistical power. Some people say 80% statistical power is reasonable, but ultimately, more is better. There is also generally a trade-off between the cost of getting more participants and the benefit of getting more statistical power.
You can assess statistical power of a t test using a simple function in R,
The following code provides the statistical power for a sample size of 15, a one-sample t-test, standard $\alpha=.05$, and three different effect sizes of .2, .5, .8 which have sometimes been referred to as small, medium, and large effects respectively.
p.2 <-power.t.test(n=15, delta=.2, sd=1, sig.level=.05, type='one.sample')
p.5 <- power.t.test(n=15, delta=.5, sd=1, sig.level=.05, type='one.sample')
p.8 <-power.t.test(n=15, delta=.8, sd=1, sig.level=.05, type='one.sample')
round(rbind(p.2=p.2$power, p.5=p.5$power, p.8=p.8$power), 2)
Thus, we can see that if the population effect size was "small" or "medium", you would have low statistical power (i.e., 11% and 44% respectively). However, if the effect size is large in the population, you would have what some would describe as "reasonable" power (i.e., 82%).
The Quick-r website provides further information on power analysis using R.