It should always be large enough! ;)
All parameter estimates come with an estimate uncertainty, which is determined by the sample size. If you carry out a regression analysis, it helps to remind yourself that the Χ2 distribution is constructed from the input data set. If your model had 5 parameters and you had 5 data points, you would only be able to calculate a single point of the Χ2 distribution. Since you will need to minimize it, you could only pick that one point as a guess for the minimum, but would have to assign infinite errors to your estimated parameters. Having more data points would allow you to map the parameter space better leading to a better estimate of the minimum of the Χ2 distribution and thus smaller estimator errors.
Would you be using a Maximum Likelihood estimator instead the situation would be similar: More data points leads to better estimate of the minimum.
As for point variance, you would need to model this as well. Having more data points would make clustering of points around the "true" value more obvious (due to the Central Limit Theorem) and the danger of interpreting a large, chance flucuation as the true value for that point would go down. And as for any other parameter your estimate for the point variance would become more stable the more data points you have.