I ran a PLS-DA model with 10-fold cross-validation to classify data in 2 groups (using the Caret package in R). The predicted probabilities are close to 0.5 (the highest propbability is just 0.7). The area under de ROC curve is 0.65. My feeling is that the model does not fit closely to the data as I read that the probabilities represent the degree of separability of two classes.
My questions are:
how should the probabilities be interpreted?
are probabilities correlated to the accuracy of the model? For example, if probabilities are all close to 0.5, will the accuracy necessarily be low? Or can we have a model with good accuracy but probabilities close to 0.5?
when comparing results of calibration and cross-validation, probabilities are in the same range, but the accuracy is higher for calibration compared to cross-validation. Is there a way to interpret that?