Short version:
Poisson regression deals with some of the quirks of modeling count rates, including the requirement that rates be positive and the presence of zero count cases in the data, which will cause problems in ordinary least squares.
Longer version:
When you are dealing with count data, you will generally assume that the counts come in at some average rate (which is possibly affected by one or more independent variables -- a.k.a. "covariates"), but the actual counts that you see will vary around that average. This variation is assumed to be truly random, not predictable even in principle, so your model is going to try to predict that average, and in particular it's going to try to tease out the influence of your covariates on the average. With that in mind, let's think about how we might model the system.
The first thing we'll note is that although the counts are integers, the average need not be. It's perfectly fine to have an average rate of 3.5 counts per minute, even though we will observe an integer number of counts in each actual minute, so the discreteness isn't going to cause us any problems, which is good.
The second thing we have to consider is that the average rate must be positive, which is problematic for ordinary least squares (OLS). If the counts are low, and the effect of the covariates is strong, then we could easily end up with a model that predicts a negative or zero rate, which is not good. One way around this problem is, instead of having our model predict the average count rate, we could have it predict the log of the rate. The log of the rate can be any real number, so we don't have to worry about our model giving impossible results. So, you could do an OLS fit to the log of the counts and interpret the model predictions as the log of the rate.
Unfortunately, there's still a problem. If the rate is low, then you will occasionally get zero counts, and with the log of zero being $-\infty$, these cases are going to mess up your fit. You could try setting a small positive number, say $10^{-8}$ as a minimum, but if you do that then you'll find that your answer depends on the precise value that you select for that cutoff. Here's where Poisson regression comes to the rescue.
In nature, a lot of counting processes show a specific relationship between the average number of counts and the distribution of observed counts. That relationship is called the Poisson distribution, and if we call the average rate $\lambda$, then the Poisson distribution looks like this:
$$
P(y=k; \lambda) = \frac{\lambda^k e^{-\lambda}}{k!}.
$$
The important thing here is that P(y=0) doesn't blow up this expression; it gives a perfectly well-behaved number.
So, the name of the game is that we let $\lambda$ be a function of our coefficients:
$$
\log(\lambda) = \alpha + \beta_1 x_1 + \beta_2 x_2 + \ldots,
$$
and we solve for coefficients $\alpha$, $\beta_1$, etc. that maximize $P(y_i = k_i; \lambda_i)$ for all of our observations. This takes care of the requirement that $\lambda > 0$, deals with any zero-count observations, and as an added bonus accounts for the fact that for low rates the distribution of observed counts tends to be rather skewed (OLS is based on the normal distribution, which is symmetric). All of these are desirable, and that is why Poisson regression is preferred to OLS for count data.
These kinds of models, by the way, are called Generalized Linear Models (GLMs), and Poisson regression is just one possibility. For example, if we had started with the assumption that our observations were True/False observations of a binary experiment, and that our model had to produce probabilities bounded by 0 and 1, we would have arrived at a formula for logistic regression, a type of GLM for predicting probabilities of binary experiments. There are a whole slew of GLM formulations out there for different kinds of problems, with Poisson regression and logistic regression being the most common.