You certainly see that there's an increasing slope in the three sample values. But what you probably want to consider is whether such a thing could have arisen by chance for unrelated variables in the presence of noisy data.
(Without some random noise you'd know the exact relationship from just two values)
Three data points whose deviation from no relationship is just due to noise could either be in increasing or decreasing order quite easily (for continuous data there are 6 equally-likely orders, one of which is purely increasing and one purely decreasing).
On the basis of a naive test like that there's little reason to conclude that there's really anything but noise causing the appearance of a relationship.
The p-value you got is simply based on a more sophisticated* version of that sort of thinking.
You mention that your y-values are observed changes in means. If you have the original data you may be able to do more than you could with three points**, but you'd need to consider more about how the data were obtained (e.g. whether some model with a random-effects term is needed)
* it's often more powerful in that it's more likely to pick up a relationship - if one is present in the underlying "population" of possible samples you might take - under the specific assumptions it makes. For example, it assumes a linear relationship, from which much of the increase in power arises, while my naive calculation only looked at the more general case of monotonicity.
** for some reason there's a very common tendency to lose much of the original information by taking averages, while not even retaining the standard deviations of the averaged values