I have data where the response is multivariate and proportional (rows [observations] sum to 1). I am modelling this response using a Dirichlet regression via the DirichletReg R package where the predictors are the first $m$ principal components of a larger set of variables.
I could use AIC/BIC to select the model size (the value for $m$), but as this is a "prediction" exercise, I'd like to be able to compute some form of error for each of the $m$ models. If this were a simple linear regression I'd use the mean squared error or its square root.
In my data I have 4 proportions measured for each response and I can derive the predicted proportions for each training sample given the fitted model. What would a suitable measure of "error" be for a multivariate proportional response?
The response variable is the proportion of each of 4 vegetation types around a lake, for a set of $n$ lakes. The predictors are the first $1, ..., m$ principal components of a data set of counts of pollen grains on $p$ species. The aim is to predict the proportions of the 4 vegetation types for samples where only the pollen counts are know (like the past 10,000 years). I'd like to compute an "error" statistic based on the lack of fit to the response given the model. Then I would use a $k$-fold CV to select the "best" model for predicting the proportions of the vegetation types.