Comparing different model with the same output I work on developing 3 different models for predicting land surface temperatures in an area (validation is done with images taken by a thermal drone). The models are:


*

*A thermodynamic mechanistic model, which uses no statistics (written in Python)

*A statistical model with multilevel linear regression (written in R)

*A neural-network model in Keras+Tensorflow (in Python) 


The different models use slightly different inputs (especially the 1st, which requires extreme meteorological measures) and operate in different ways, but produce the (presumably) same result.
All 3 models' output is the same: an array/matrix which can be heatmapped to represent surface temperatures in an area, so a per-sample error mean and s.d. can be calculated for all 3 models, and for every single observation.
While I did run a simple pairwise t-test on the 3 groups of error means (a group for each model), comparing means doesn't represent the predictions well enough, because the outputs are matrices, so I'd like to perhaps include a comparison of the spatial variance. I briefly thought about AIC but am a bit stumped as per how to approach building the criterion on my own.
 A: If you're comparing your models to some observed reference, then it's a problem of measuring similarities of the two matrices. 
A matrix that assigns a scalar value to every single point in some area is, in other words, a really, really simple vector fields. What do you know, calculating differences between vector fields has been something centuries of physicists and engineers had dealt with on a regular basis.
The simplest similarity metric here would be just the sum - not the mean - of errors, absolute or squared (because otherwise you're just measuring systematic bias of each method without accounting for random errors) across every single entry of the array. Fairly obviously, the model with the smallest total sum of mispredictions predicts the best.
Since every item of your array is spatially connected to the adjacent ones, you may - and seem to - want to go easy on models which simply fat-fingered which coordinate to put some temperature in, in which case you can zoom out a bit and smudge the adjacent entries together by averaging the values in a 3x3 kernel centered on your current entries in both matrices and compare these. Same principle. 
You could use a weighted combination of the two, and/or you could go to higher levels, but beyond a certain point you'd be basically building a convolutional NN just to compare models, which would be quite silly.
