I am conducting a classification problem. Usually AUC is used to check the effectiveness of a model when performing logistic regression model. However, if I fit the same data set by using Random forest and logistic regression models, which measures should be used to check the accuracy of the two models? Are there any measures like "BIC" can be used to compare the two models? Thanks.
Those are two separate issues, AUC can be used to compare arbitrarily many algorithms. AUC is also not more linked to logistic regression than to random forests or any other algorithm. Just test both algorithms on the same data-set splits (hold-out, cross validation, bootstrap or whatever it is you do) and compare the numbers.
AUC has other drawbacks like being limited to binary classification and being less intuitive than say accuracy, but that's data-set related and thus also independent of the number and type of algorithms to compare.
There are many measures you can use, AUC is one of them.
You can also use other measures like the brier score and logarithmic loss which look how well the probability estimations are done. Or of course the missclassification case, which has the drawback that a specific threshold has to be set. I always check these three measures together with the AUC. In R you can easily calculate them with the mlr package: http://mlr-org.github.io/mlr-tutorial/devel/html/measures/index.html