Is it cheating to use least squares to obtain a prior for Bayesian regression? Say that I have the model $Y=AX + E$ where $Y,X \in \mathbb{R}^{n\times m}$ and that $A$ is an unknown $n \times n$ matrix and E is a matrix of observation noise. When I place an uninformed prior on the parameters in $A$, NUTS sampler (specifically in PyMC) tends to throw an error that the gradients vanish. When I placed an informed prior (for example, a solution given by least squares), I get no such issue. So I am wondering if this style of prior is considered "cheating" in any way?
 A: Note that "a solution given by least squares" in itself is not a probability distribution and can't therefore be used as prior. I assume you mean to center the prior around the LS estimator (maybe using more information connected to it such as a variance estimator for setting further properties of the prior).
The aim of a Bayesian analysis is to make posterior probability statements about the parameters or future observations. These statements are meant to properly reflect existing uncertainty. Information to be used in the prior should be prior information, information available before observing the data. Bayes' Theorem will then combine the prior information with the information in the data, and the new information in the data will normally reduce the uncertainty that was present in the prior, as with "prior information plus data" we know more than from the prior, or from the data alone.
If you use the data themselves for setting the prior (which is what you propose), the data will be used twice; information in the prior that is in fact from the data will be treated as if it were prior information before using the data, and then, computing the posterior from prior+data, the data will reduce the prior uncertainty. Except that the "prior uncertainty" here isn't additional information but just information in the data themselves; and then, computing the posterior, the data will be used to "reduce its own uncertainty", which isn't possible.
So using the data themselves to set the prior will normally lead to an underestimation of uncertainty or "overconfidence" (too low variation in the posterior), as the information in the data is used twice instead of using additional information from other sources, which is what the prior is meant to encode.
