This is more of a philosophical question, but from a purely Bayesian standpoint how does one actually form prior knowledge? If we need prior information to carry out valid inferences then there seems to be a problem if we have to appeal to past experience in justifying today's priors. We're apparently left with the same question regarding how yesterday's conclusions were valid, and a kind of infinite regress seems to follow where no knowledge is warranted. Does this mean that ultimately prior information must be assumed in an arbitrary way, or perhaps based on a more "frequentist" style of inference?
Speaking of prior knowledge can be misleading, that is why you often see people speaking rather about prior beliefs. You do not need to have any prior knowledge to set up a prior. If you needed one, how would Longley-Cook manage with his problem?
Here is an example from the 1950s when Longley-Cook, an actuary at an insurance company, was asked to price the risk for a mid-air collision of two planes, an event which as far as he knew hadn't happened before. The civilian airline industry was still very young, but rapidly growing and all Longely-Cook knew was that there were no collisions in the previous 5 years.
Lack of data about mid-air collisions was not a problem to assign some prior to it that lead to pretty accurate conclusions as described by Markus Gesmann. This is extreme example of insufficient data and no prior knowledge, but in most real life situations you would have some out-of-data beliefs about your problem, that can be translated to priors.
There is a common misconception about priors that they need to be somehow "correct", or "unique". In fact, you can purposefully use "incorrect" priors to validate different beliefs against your data. Such approach is described by Spiegelhalter (2004) who describes how a "community" of priors (e.g. "skeptical", or "optimistic") can be used in decision-making scenario. In this case it is not even prior beliefs that are used to form priors, but rather prior hypotheses.
Since when using Bayesian approach, you include both the prior and data into your model, information from both sources will be combined. The more informative is your prior comparing to data, the more influence it would have, the more informative is your data, the less influence would your prior have.
Eventually, "all models are wrong, but some are useful". Priors describe beliefs that you incorporate in your model, they do not have to be correct. It is enough if they are helpful for your problem, as we are dealing only with approximations of reality that are described by your models. Yes, they are subjective. As you already noticed, if we needed prior knowledge for them, we would end up in a vicious circle. Their beauty is that they can be formed even when confronted with shortage of data, so to overcome it.
Spiegelhalter, D. J. (2004). Incorporating Bayesian ideas into health-care evaluation. Statistical Science, 156-174.
I think you're making the mistake of applying something like the frequentist concept of probability to the foundations of the subjective definition. All that a prior is in the subjective framework is a quantification of a current belief, before updating it. By definition, you don't need anything concrete to arrive at that belief and it doesn't need to be valid, you just need to have it and to quantify it.
A prior can be informative or uninformative and it can be strong or weak. The point of those scales is that you don't have any implicit assumptions about the validity of your prior knowledge, you have explicit ones, and sometimes that can be "I have no information." Or it can be "I am not confident in the information I have." The point is, there is no requirement that prior knowledge is "valid". And that assumption is the only reason your scenario seems paradoxical.
By the way, if you like thinking about the philosophy of probability, you should read The Emergence of Probability by Ian Hacking and its sequel, The Taming of Chance. The first book especially was really illuminating in how the concept of probability came to have dual and seemingly incompatible definitions. As a teaser: did you know that until fairly recently, calling something "probable" meant that it was "approvable", i.e. that it was "approved by the authorities" or that it was a generally well respected opinion. It had nothing whatsoever to do with any concept of likelihood.