Suppose you have a model of the form:
$$X \beta= Y$$
where X is a normal 2-D matrix, for ease of visualisation.
Now, if the matrix $X$ is square and invertible, then getting $\beta$ is trivial:
$$\beta= X^{-1}Y$$
And that would be the end of it.
If this is not the case, to get $\beta$ you’ll have to find a way to “approximate” the result of an inverse matrix. $X^\dagger = (X'X)^{-1}X'$ is called the (left)-pseudoinverse, and it has some nice properties that make it useful for this application.
In particular, it is unique, and $XX^\dagger X=X$, so it kind of works like an inverse matrix would $(XX^{-1}X = XI = X)$. Also, for an invertible and square matrix (i.e. if the inverse matrix exists), it is equal to $X^{-1}$.