How to identify distinct classes for classification problems? I'm working with a dataset in which we've taken audio recordings of coral reef habitat from 3 different types: healthy, degraded and restored. From each recordings I have 13 different continous features (acoustic indixes that infer something about the recording).
I have contructed a model using disciminant analysis trained on the healthy and degraded habitat recordings that is able to differentiate these two habitats with 92% accuracy. I have then run this on the restored recordings and it has classified these in the pattern we expect; being that sites on which restoration started 6 months ago are mostly classified as degraded, and sites restored 5 years ago are mostly classified as healthy.
Other sceptical ecologists will certainly argue that we cant be certain that the 5 year old restored sites sound like healthy habitat, as restored sites may be entirely distinct and the model isnt able to reveal that, just that they are more similar to the healthy than degraded site but still highly distinct.
Is there an approach one can use to search to see if the new data you're running such a model on is itself different? My best guess is k-means clustering may be useful if I set K = 2 in one trial, and then k = 3 to see if the restored sites cluster with healthy/degraded or cluster seperately? Would this work, is there a better approach to explore this kind of problem?
 A: I'd start with visualisation. Assuming that your dataset is not super large, there are a few options. You can do a pairs plot. With 13 features this can probably just about be done if you have a big screen. Alternatives are principal components or a discriminant coordinate plot (the latter shows the projection of the data according to which the classes are optimally separated; actually for two classes there is only one informative dimension, however there are possibilities to do a meaningful 2-d plot of this kind, see Hennig (2004)
https://www.jstor.org/stable/27594085?seq=1#metadata_info_tab_contents
with code available in R-package fpc). I'd compute these on the training data, coloring the two classes differently, and then project the new data on the plots, using suitable different color(s). Then you get an impression if the new data are systematically different from the old ones.
A parallel coordinates plot may also show you something.
Here's some demonstration with the Iris data:
library(fpc)
library(mvtnorm)
data(iris)
irisx <- iris[,1:4]

# I generate some artifical new data to be compared with Iris classes
newdata <- rbind(rmvnorm(50,c(5,3.5,1.5,0.5),sigma=0.2*diag(4)),
             rmvnorm(50,c(7,3,4,2),sigma=0.2*diag(4)))

# Pairs plots
classvec <- c(as.integer(iris[,5]),rep(4,100))
pairs(rbind(as.matrix(irisx),newdata),col=classvec)

# Principal components
pcairis <- prcomp(irisx,center=FALSE)
# Projection of new points:
projpoints <- newdata %*% pcairis$rotation

plot(pcairis$x,col=iris[,5])
points(projpoints,col=4,pch="x")

# Discriminant coordinates; for only two classes I'd probably use 
# method="bc" 
discrcoordiris <- discrproj(irisx,iris[,5])
# Projection:
projpointsd <- newdata %*% discrcoordiris$units

plot(discrcoordiris$proj,col=iris[,5])
points(projpointsd,col=4,pch="x")

Better use discrproj than plotcluster, because plotcluster only produces the plot but doesn't give you the projection "units" for projecting new points.
I'm not quite sure about what you want to achieve with the clustering, so I don't comment on that.
