What is the 'same distribution' mean? Say if I have two random variable X and Y and they have the same distribution, what is that suppose to mean? Is that mean they have same mean and variance?
 A: It is more general than this. It means that $F_X(t)={\mathbb P}(X\leq t) = {\mathbb P}(Y\leq t) = F_Y(t)$, for all $t$. Then, in particular, if the mean and variance exist, then their values coincide for these variables.
The functions $F_X(t)={\mathbb P}(X\leq t)$ and $F_Y(t)= {\mathbb P}(Y\leq t)$ are termed the distribution functions of the variables $X$ and $Y$, respectively.  See
http://en.wikipedia.org/wiki/Random_variable#Equality_in_distribution
A: Strictly speaking, it means that the CDF is the same.
That is, the type of distribution, the mean, the variance, and all parameters are all the same, if they are well-defined.
For most of the commonly seen distributions, like normal distribution, if you can verify that type of distribution and all parameters are the same, then the distributions are the same.
However, be aware of that the mean and variance can be undefined; for example, see Cauchy distribution. In fact, the PDF and any other parameters can be undefinable (credit to Whuber).
However, their correlation can still be arbitrary (or undefined).
