Why do we compute eigenpairs of $X^TX$ instead of $X$ in Principal Component Analysis? What is the intuition behind finding the eigen-vectors and eigen-values of $X^TX$ or $XX^T$ instead of the matrix $X$? One reason that I see immediately is when the matrix $X$ is non-square. I am searching for a more concrete reason. 
Also, what difference does it make when eigenpairs are computed for $XX^T$ instead of $X^TX$?
 A: Let's assume $X$ as $n\times m$, where $n$ is number of data samples and $m$ is number of features; and assume $X$ is mean-centered. You can't find an eigen decomposition for non-square matrices, as you also pointed out. But, of course, we don't take $X^TX$ just because it is square. It's the scatter matrix, i.e. proportional to the sample covariance. PCA's aim is to capture the maximum variance via designing new axes. So, what PCA does actually is a decomposition of this co-variance matrix as:
$$X^TX = \lambda_1v_1v_1^T+\lambda_1v_2v_2^T+\dots+\lambda_mv_mv_m^T$$
The eigenvectors are of unit length. But, the eigenvalues determine the contribution of them. So, if we wanted to use $k$ of the PCA axes, the scatter matrix would be approximated by the first $k$ summands. By choosing more and more PCs, you get close to the original covariance matrix. This is the idea of capturing the variance. 
If you operate on $XX^T$, you'd be treating your data samples as features, and your features as data samples. Your axes are your features.
