The frequentist, likelihood and, to an even greater extent, Bayesian approaches to statistics are all based on probability. Without probability, it seems difficult to use a data sample ("seen" cases), to infer results about a more general population ("unseen" cases), and estimate the uncertainty inherent in such a process. Thus it would appear that all we can do is descriptive statistics, as well as some trivial inequalities (such as, if we observe a person of height 2.0 mt, we know that the maximum height in the population of all living persons is $\geq$ 2.0 mt).
Or is it? Are there mathematically rigorous approaches to the foundation of statistics, which do not rely on the theory of probability, and can you point me to references on the subject?