The C parameter is used for multiple classification. The nu parameter is used in one-class classification, so as long as you optimize nu you are fine. nu is the upper bound in the fraction of training points. Please check this link
Depending on the kernel function you choose, you will have to optimize gamma too. The hyperplane varies depending on both parameters. If you choose the linear kernel, then there is no need to optimize gamma, as it uses the default value=0 where the gamma value is equal to 1/n (n is the number of features).
To tune your parameters, you can use the tune.svm function:
tuned <- tune.svm(x=yourFeatures, y=yourLabelY, data = yourData,
nu = 0.001:0.5,
gamma = 10^(-2:0),
type='one-classification'
);
Then you can use the best values to design your one-class svm model:
model <-svm(x=yourFeatures, y=yourLabelY,type='one-classification',
nu = tuned$best.parameters$nu,
gamma = tuned$best.parameters$gamma
);
This means your nu value will be the best value between 1% and 50%, meaning you will allow up to 50% of outliers (but you want the best value).