Which of the 3 cases should my data matrix belong to ideally? I found this question, and while useful, I wanted to ask something more spcific:
I am trying to get a good handle/intuition for the two types of data dimensionalities (number of data samples, and the number of features of each sample), within the machine learning context. 
So, let us consider an input data matrix $X$, of size $m$ x $n$, ($m$ rows by $n$ columns), where $m$ is the number of features, and $n$ is the number of data samples we have. 
(For example, $n=10$ means we have, say, data on 10 people, while $m$ here would be the number of features we have per person, so if $m=3$, then we might have say, height, weight, and age. Thus, $X$ would be a $3$x$10$ matrix).
At any rate, for any data matrix $X$, we can have three possibilities: 


*

*Either there are more features than samples, so the matrix is long and thin. (ie, more rows than columns, $m$ >> $n$)

*Either there are more samples than features, so the matrix is short and fat. (ie, more columns than rows, $n$ >> $m$)

*Or lastly, we have a square matrix, where the number of features is equal to the number of samples. ($n$ = $m$)


My question is, how do either of those conditions on a data matrix $X$ affect whether or not we can properly learn/predict from our data, in the machine learning context? Is one of those conditions always bad? Is one always good? Is there an ideal we just aim to strive for?
Thanks.
 A: Data analysis in general is easiest when the sample size is much larger than the number of features. The larger the sample, the more opportunity you have to find a signal hidden in the noise. The smaller the number of features, the less data you need (and the less work you have to do) to find features that are related to the outcome variable and determine the nature of any such relationship.
The fact that a large number of features relative to the sample size makes data analysis hard is called the curse of dimensionality.
A: First, the way you have described $m$ and $n$ in your question is opposite of convention: we usually talk of $n$ observations (rows) with $m$ features (columns). To attempt to avoid further confusion, let's follow the convention of Introduction to Statistical Learning and call the samples/observations in the rows $n$ and the features in the columns $p$.
Second, when there are as many features as there are observations ($n = p$), the matrix $X$ is positive semidefinite, and when there are fewer observations than features ($p > n$), the matrix is called singular or degenerate and cannot be inverted.
Among other problems with this type of situation are the facts that (1) the least squares regression coefficient(s) $\beta = (X^T X)^{-1} X^T Y$ does not have a unique solution because $(X^T X)$ is not invertible, and (2) a solution can be found with zero variance, i.e. a perfect fit.
The short answer to your question is that more observations than features is always desirable, at least in an idealized world. In the cases where there are more features than observations, particular care must be taken in building and interpreting the machine learning model employed.
I recommend reading section 4 of chapter 6 of the above referenced book as a good introduction to the topic in more space than is appropriate for this context.
