How do Sparsity Priors help for Identifiability? Let's say we have a Factor Analysis model with a latent variable $\mathbf{z}_t \in \mathbb{R}^k$:
$$x_t = A z_t + \epsilon_t, \qquad \epsilon_t \sim \mathcal{N}(0, \Sigma)$$
Let $A \in \mathbb{R}^{g\times k}$, and $\Sigma \in \mathbb{R}^{g\times g}$ and $\Sigma$ is diagonal.
Many sources say that the model is not identifiable. If I understand correctly, identifiability is studied by showing the uniqueness of the covariance matrix $\Lambda = A A^T + 
\Sigma$.
One issue is rotation: given an orthogonal rotation matrix $Q$, it holds that: $\Lambda = (A Q^T) (Q A^T) + \Sigma = A A^T + \Sigma$.
This problem can be tackled by e.g. assuming that $A$ is a lower-triangular (LT) matrix.
Instead of using LT matrices, some people use a sparsity-inducing spike-and-slab or horseshoe prior on all elements of $A$. It's clear to me that I have to impose a constraint on $A$ to make the solution unique, but I don't understand why placing sparsity inducing priors is one solution, because it is just a prior which is not enforced in the posterior solution. In Baysian inference, I place a normal prior on $z$ and a horseshoe prior on $A$:
$$z_t \sim \mathcal{N}(0, 1)$$
and
$$A_{gk} \sim \mathcal{N}(0, \tau^2 \cdot \lambda_{gk}^2)$$
How does this render the problem identifiable?
 A: This answer is not a proof that those priors will render the problem identifiable but an intuitive explanation that it should work.
As you have pointed out, the problem with ordinary factor analysis (FA) is that there are infinitely many different linear maps $A: z\to x$ which lead to the same likelihood of the data in this FA model. Thus, the idea is to weigh all those $A$ differently by giving them different prior probabilities.
The original FA could be thought of as all $A$ having the same weight (an improper prior), and by giving them different priors (weights) the above tie is broken and, hopefully, there is only one $A$ left that gives the largest likelihood to the data $\{x_t\}$. I am not sure what you refer to by "it is just a prior which is not enforced in the posterior solution", but the prior does change the posterior.
You have shown that, if $A$ is a solution to the original FA, i.e. an MLE solution (frequentist) or a MAP for a constant improper prior (Bayesian), then all $A^\prime = AQ$, with $Q$ any orthonormal map, are also MLEs (MAPs). I.e., they all have the same a posteriori density. Thus, intuitively, the plan is to choose a prior that is not constant on this set of matrices $A^\prime$, which also makes the posterior there nonconstant.
And while we are at it, we should also design the prior such that certain nice features of $A$, e.g. sparsity, are favored. In this case, it makes sense to use e.g. horseshoe priors.
In other words, the solution with horseshoe priors will be the sparsest matrix that is also "near" the set $\{A^\prime=AQ| Q\;\mbox{orthonormal}\}$.
Finally, note that restricting to LT matrices can be interpreted as a similar approach, where priors are just very strongly penalizing all matrices with nonzero values in the upper triangle.

Edit: Why did I write that "the solution with horseshoe priors will be the sparsest matrix that is also "near" the set $\{A^\prime=AQ| Q\;\mbox{orthonormal}\}$.", why only "near"?
Let's give this set a name, $S:=\{AQ|Q\in O(k)\}$, where $O(k)$ are the orthonormal matrices. For the improper prior $p^i$, the posterior $f^i$ has infinitely many MAPs, given by the set $S$. Note, that this doesn't mean that all the other matrices $A$ are impossible. They are very well possible, they are just not as likely. In fact, when computing the Bayesian predictive distribution, all $A$ are integrated over.
Next, we create a new posterior $f^n$ by multiplying the old posterior $f^i$ with a nonconstant prior $p^n$ (and normalizing), $f^n\sim f^i p^n$. Now it is not clear that the maximum of $f^n$ should also be a maximum of $f^i$, ie. be in $S$. In general, $f^i$ is preferring matrices in $S$, while $p^n$ is preferring sparse matrices, and thus the product $f^n$ will be a compromise which in general is not completely sparse (read: not a maximum of the horseshoe prior $p^n$) and not completely in $S$.
A: They don't --- adding a prior does not solve the identifiability problem
In this model you have an identifiable parameter $\Lambda = AA^\text{T} + \Sigma$ but the parameter $A$ is not identifiable.  The notion you are putting forward (which is unfortunately a popular belief) is that if you use a Bayesian model with a prior for $A$ then this will make up for the fact that $A$ is not identifiable.  The idea that this works comes from the fact that if you impose a prior then you will get a valid posterior for the unidentifiable parameter, which some analysts take to mean that identifiability is no longer a problem.  Unfortunately, it turns out that this posterior is not consistent, and it is almost entirely determined by the specified relationship between $\Lambda$ and $A$ in the prior.  (For a closely related discussion, see this related question.)
To see what happens when you impose a prior in this type of model, suppose you observe $n$ data values giving the sample vector $\mathbf{x}_n$.  The likelihood function here is fully determined by $\Lambda$, so the posterior for both parameters can be decomposed as:
$$\begin{align}
\pi(\Lambda,A|\mathbf{x}_n) 
&= p(A|\Lambda,\mathbf{x}_n) \ \pi(\Lambda|\mathbf{x}_n) \\[6pt]
&= p(A|\Lambda) \ \pi(\Lambda|\mathbf{x}_n). \\[6pt]
\end{align}$$
The second term in this product is the posterior for the identifiable parameter and the first is the conditional distribution of the unidentifiable parameter given the identifiable parameter.  The latter is fully determined by the prior distribution on the parameters; it is not influenced by the data.  So, you can see that the posterior you get for your non-identifiable parameter is determine in large part by the stipulated relationship between the parameters in the prior.
It gets worse.  Under appropriate conditions, as $n \rightarrow \infty$ the posterior for the identifiable parameter will converge towards a point-mass distribution on its true value $\Lambda_*$ (which we call posterior consistency).  We would hope that we would also have posterior consistency for the non-identifiable parameter of interest, but instead we get:
$$\begin{align}
\pi(\Lambda,A|\mathbf{x}_n) 
&\rightarrow p(A|\Lambda) \ \pi(\Lambda|\mathbf{x}_\infty) \\[6pt]
&\propto p(A|\Lambda = \Lambda_*). \\[6pt]
\end{align}$$
This shows us that the asymptotic posterior for the non-identifiable parameter is affected by the data only to the extent that it effectively allows us to condition on the true value of the identifiable parameter.  The uncertainty in the resulting asymptotic prior is fully determined by the specified relationship between $\Lambda$ and $A$ in the prior.  As you can see, even though we get a valid posterior from this type of analysis, we do not have posterior consistency in our inference and the resulting inference is determined primarily by our prior, rather than by the data.  This means that you can do a Bayesian analysis with a non-identifiable parameter, but it does not solve the identifiabilty problem and it does not let you get a sensible inference for non-identifiable parameters.
