Overfitting when dealing with nearly the same number of features and observations When dealing with nearly same number of features and observations, one of the most common problem is overfitting.  For my project I used 2 class LDA on a 1400 * 1000 dataset and to avoid overfitting, I used K-fold (k=10) cross-validation.  I set 10% of my observation vector aside each time as a test dataset and trained the remaining data and repeated the whole procedure 10 times. Unfortunately I am not certain if the overfitting problem was resolved by using this method.  So I have the following questions:


*

*How can I measure if overfitting happened or not?  

*After using some additional techniques for overcoming / avoid this problem (such as cross-validation,  regularization, early stopping, etc.), how can I know how much these extra techniques helped me avoid overfitting?  

*When dealing with nearly same number of features and observations, what is the best extra techniques one can use to prevent overfitting?  

*In my case, does increasing K, the number of folds, help me prevent overfitting?


Note:i don't know if this going to help or not , but my work is a supervised learning problem.
 A: 
1.how measure if overfitting happened or not?

You get a hint a model probably is overfitted when the performance on test that is unreasonably low compared to performance on train data or even compared to no-information model, but keep in mind algorithms are always expected to perform better on train data.

2.after using some additional techniques for overcoming/avoid this problem (such as cross-validation, regularization, early stopping, ...) how should i know how much these extra method help me to avoid the Overfitting problem?

The nearer your test performance gets to train performance the least overfitting there is. Caution is warranted because you may be leaving overfitting just to enter underfitting, i.e. train and test performance are reasonably similar yet both are bad. 

3.when dealing with nearly same number of feature and observation, what is the best extra method one can use for preventing overfitting?

Quite difficult to answer that without being based on opinion. Have you tried to diminish a bit the number of features, like eliminating linear combinations or features that have near zero variance (this is part of the model optimization and so should be done inside the cross-validation)? Also, embedded regularization methods like lasso are worth a check (I see you mentioned it). Search-type feature selection methods might actually make overfit worse, i.e. the feature selection itself might be overfitted to training data.

4.and last but not least, in my case dose increasing K number help me prevent Overfitting?

The choice of $K$ must take into account the bias-variance tradeoff. A good read about it is Chapter 3 with emphasis on Sections 3.3-5 of Kohavi, R. (1995). Wrappers for performance enhancement and oblivious decision graphs (Doctoral dissertation, stanford university). The point is large optimistic bias leads to overfitting. Increasing $K$ reduces the bias, but might increase variance to the point of uselessness. Repeated cross-validation can be used to reduce variance, but repeating it too much leads to underestimation of the variance. Too small $K$, like $2$-fold CV also has large variance. $10$-fold is usually considered a good compromise.

This answer brings a heuristic to estimate overfitting, but I never tried it so can't really comment on it.
