Random Forest - How to handle overfitting I have a computer science background but am trying to teach myself data science by solving problems on the internet. 
I have been working on this problem for the last couple of weeks (approx 900 rows and 10 features). I was initially using logistic regression but now I have switched to random forests. When I run my random forest model on my training data I get really high values for auc (> 99%). However when I run the same model on the test data the results are not so good (Accuracy of approx 77%). This leads me to believe that I am over fitting the training data. 
What are the best practices regarding preventing over fitting in random forests? 
I am using r and rstudio as my development environment. I am using the randomForest package and have accepted defaults for all parameters
 A: To avoid over-fitting in random forest, the main thing you need to do is optimize a tuning parameter that governs the number of features that are randomly chosen to grow each tree from the bootstrapped data. Typically, you do this via $k$-fold cross-validation, where $k \in \{5, 10\}$, and choose the tuning parameter that minimizes test sample prediction error. In addition, growing a larger forest will improve predictive accuracy, although there are usually diminishing returns once you get up to several hundreds of trees.
A: How are you getting that 99% AUC on your training data? Be aware that there's a difference between
predict(model)

and
predict(model, newdata=train)

when getting predictions for the training dataset. The first option gets the out-of-bag predictions from the random forest. This is generally what you want, when comparing predicted values to actuals on the training data.
The second treats your training data as if it was a new dataset, and runs the observations down each tree. This will result in an artificially close correlation between the predictions and the actuals, since the RF algorithm generally doesn't prune the individual trees, relying instead on the ensemble of trees to control overfitting. So don't do this if you want to get predictions on the training data.
A: Try to tune max_depth parameter in ranges of [5, 15] but not more than this because if you take large depth there is a high chance of overfitting.
A: For decision trees there are two ways of handling overfitting: (a) don't grow the trees to their entirety (b) prune 
The same applies to a forest of trees - don't grow them too much and prune. 
I don't use randomForest much, but to my knowledge, there are several parameters that you can use to tune your forests: 


*

*nodesize - minimum size of terminal nodes 

*maxnodes - maximum number of terminal nodes 

*mtry - number of variables used to build each tree (thanks @user777)

