How to make a confusion matrix when testing a model on data with only positive cases? I have recently encountered a problem with the function confusionMatrix in R from the caret package regarding a machine learning procedure. In detail, I have trained a binary classifier with two levels ("Cancer" & "Normal") and I'm trying to test it in independent datasets. However, one of my test datasets had represented only one of the levels regarding the response variable - that is only cancer samples. So,
final.classes.xgb.gse <- predict(xgbTune, newdata = test.set.gse)

head(final.classes.xgb.gse)
[1] Cancer Cancer Cancer Cancer Cancer Cancer
Levels: Cancer Normal

table(final.classes.xgb.gse)
final.classes.xgb.gse18088
Cancer Normal 
    51      2 

and my "true labels"-reference for this dataset factor variable:
head(test.labels.gse)
[1] Cancer Cancer Cancer Cancer Cancer Cancer
Levels: Cancer

But when I type:
confuse.mat.gse18088 <- confusionMatrix(data=final.classes.xgb.gse, test.labels.gse)
**Error in confusionMatrix.default(data = final.classes.xgb.gse, test.labels.gse) : 
  the data cannot have more levels than the reference**

Is there a way to solve this ? As this test dataset has only samples belonging to only one class of the two?
 A: The final.classes.xgb.gse18088 variable has two levels, i.e. Cancer and Normal, but test.labels.gse  has only one level, Cancer.  confusionMatrix is complaining. 
 about that.  
These are both factors and a factor can have levels that are not present in the data so just add the Normal level to the levels of test.labels.gse:
levels(test.labels.gse) <- c("Cancer", "Normal")

Now it should produce output.
For example,
library(caret)

# test data
final <- factor(c("Cancer", "Cancer", "Cancer", "Cancer", "Normal"))
levels(final)
## [1] "Cancer" "Normal"

test <- factor(c("Cancer", "Cancer", "Cancer", "Cancer", "Cancer"))
levels(test)    
## [1] "Cancer"

# gives expected error
confusionMatrix(final, test)
## Error in confusionMatrix.default(final, test) : 
##   the data cannot have more levels than the reference

# now it produces output
levels(test) <- c("Cancer", "Normal")
confusionMatrix(final, test)

giving:
Confusion Matrix and Statistics

          Reference
Prediction Cancer Normal
    Cancer      4      0
    Normal      1      0

               Accuracy : 0.8             
                 95% CI : (0.2836, 0.9949)
    No Information Rate : 1               
    P-Value [Acc > NIR] : 1               

                  Kappa : 0               
 Mcnemar's Test P-Value : 1               

            Sensitivity : 0.8             
            Specificity :  NA             
         Pos Pred Value :  NA             
         Neg Pred Value :  NA             
             Prevalence : 1.0             
         Detection Rate : 0.8             
   Detection Prevalence : 0.8             
      Balanced Accuracy :  NA             

       'Positive' Class : Cancer       

A: Do a table on your predicted variable. I'm going to bet that it only has one level. If so, this means that your model is predicting the same class for all cases in your dataset. Can't be sure why without more info, but it may be that you have terrible separation or severe class imbalance. 
