The Gamma prior is certainly a good option as it is conjugate with the exponential likelihood. This means that the posterior will also be a Gamma distribution, and the update (prior $\rightarrow$ posterior) do not rely on any simulation.
Say you have a prior $\Gamma(a,b)$ whose density is:
$$
p(\theta ; a,b) \propto \theta^{a-1}e^{-\theta b}
$$
and the exponential likelihood based on observations $(X_1,\dots,X_n)$:
\begin{align*}
p(X \mid \theta) &= \prod \theta e^{-\theta X_i} \\
&= \theta^n e^{-\theta \sum X_i}
\end{align*}
What we are interested in is the posterior distribution of $\theta$, that is the distribution of $\theta$ after observing the data.
We get this posterior from the bayes theorem:
$$
p(\theta \mid X) = \frac{p(\theta) p(X \mid \theta)}{p(X)}
$$
which is sometimes written as $p(\theta \mid X) \propto p(\theta) p(X \mid \theta)$.
Here $p(\theta)$ is the prior distribution and $p(X \mid \theta)$ is the likelihood of the model.
Thus the posterior distribution of $\theta \mid X$ is proportional to the product of the prior and the likelihood (and I think this is what is meant by combine with the likelihood):
\begin{align*}
p(\theta \mid X) &\propto p(\theta ; a,b) p(X \mid \theta) \\
& \propto \theta^{a-1}e^{-\theta b} \theta^n e^{-\theta \sum X_i} \\
& \propto \theta^{a+n-1} e^{-\theta(b+\sum X_i)}
\end{align*}
The last line, seen as a function of $\theta$, is proportional to the density of a $\Gamma(a+n, b+ \sum X_i)$, which is the posterior distribution of $\theta$.
The posterior is completely determined an you can easily compute posterior mean/median or credible intervals without relying on any simulation.
You have to choose the parameter $(a,b)$ for the prior distribution of $\theta$. This should reflect you prior belief on the plausible values of $\theta$. This parameters $(a,b)$ are fixed, but can sometimes be considered themselves as random, but this is advanced bayesian statistics.
