I am trying to perform one-class SVM for novelty detection, so I use cross-validation to find the best $\nu$ parameter, but I found that usually I got a big parameter $\nu$, resulting in a big false positive rate.
How to set and find the $\nu$ parameter for a one-class SVM?
How do you measure your performance via cross validation? Do you have label data? If so, why don't you try to do 2-class classification?! According to Roemer Vlasveld's blog, parameter nu characterizes the solution: 1. it sets an upper bound on the fraction of outliers (training examples regarded out-of-class) 2. it is a lower bound on the number of training examples used as Support Vector.
If you set nu to be large it is like saying you are expecting a lot of outliers. This is why you have many false positives.
One class svm are hard. It is not only the $\nu$ parameter, but also the parameters in the kernel, if you are using anything else than a linear kernel. So, the only way I know is if you have the true class for both your class of interest and the "other" class. Then you can ajust both $\gamma$ (in the RBF kernel) and $\nu$ using some metric of classifier success, either the accuracy or the just the accuracy of your class of interest (false positive rate or flase negative rate, specificity or sensitivity - it depends on whether you call your class of interest as positive or negative).
In my limited experience in this, the one class SVM is almost always worse than using a standard classifier if you look at accuracy (I dont remember if I tested for false positive rate). So, if you have both the interesting and the uninteresting classes, you should try the standard classifier as proposed by @Hanan Shteingart. The problem is that if you dont have enough or representative examples of the uninteresting class. In this case you should go into the direction of open set recognition. I know at least this paper http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6365193 (sorry behind pay wall). This paper http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1512050 may also be relevant (given the title) but I have not read it.
