I am reading through Christopher Bishop's Pattern Recognition and Machine Learning book and came across this part that I did not quite understand. 
As I understand it, this is claiming that the probability of a set of data is equal to the product of the p.d.f. of the normal distribution of the datapoints. However, the p.d.f. can be greater than one, especially if $\sigma$ is small. So if it happens that I sample a large amount of my data near the mean, isn't it possible to return a value greater than 1 for the probability of the dataset? I feel I have a misunderstanding somewhere since this shouldn't be possible.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You're quite correct; as many posts on site reiterate, density is not probability, and densities can certainly exceed 1.
However density is proportional to a probability. e.g. in the univariate case the probability of being in the interval $(x,x+\delta )$ would (as $\delta$ becomes very small) approach $p(x)\delta$ (this may be put more formally but my aim here is not to be formal).
Bishop is not correct to call that conditional density "the probability of the data set" though we might reasonably assume that's loose terminology rather than lack of understanding.
