Should we use measured vs. modelled or modelled vs. measured?

Question

As a non-statistician, I am interested in understanding whether model evaluation should be conducted with the measured values on the x-axis and modelled (or predicted) values on the y-axis, or vice versa.

For context, I am teaching a university course on environmental modelling. One of the practicals utilises measured data on river water quality (e.g. measured NO2 concentrations). Based on the characteristics of the river catchment (i.e. land use, geology, soil types, topography) and using testing-training datasets, students are asked to create models to predict river water quality (i.e. modelled NO2 concentrations). EDIT: the measured data are first split into training (70%) and testing (30%) subsets. The models are tuned using the training subsets, and then evaluation is with the testing subsets.

They students are then asked to evaluate the models in a variety of ways, one of which is simply plotting measured vs. modelled data, and comparing the results of linear regression to 1:1. I have included some sample data and code below to illustrate, the results of which are shown here:

EDIT The aim here is to enable students to gain experience of model evaluation, specially thinking about missing variables as well as different types of error (i.e. systematic-random, under- or over-prediction).

This course has run for a number of years but I only recently began to contribute to it, and in checking over the method, I came across the papers by Piñeiro et al. (2008) and Pauwels et al. (2019). The former is highly cited (>700 times) and recommends the following: Observed (in the y-axis) vs. predicted (in the x-axis) (OP) regressions should be used ...

Pauwels et al. (2019) argue that this approach is incorrect and is an artefact of the experiment setup and that instead, assessing models in a scatter plot with the observations in abscissa (X-axis) and the corresponding simulations in ordinate (Y-axis) will lead to the correct conclusions regarding the model performance. This paper has only been cited once since publication.

While I have some understanding of statistics, I am not an expert statistician and would like to know if there is a general rule for evaluating measured-modelled data on the X-Y axes.

## Example Code

import numpy as np
import matplotlib.pyplot as plt

# Input data
Measured = [0.110, 0.150, 0.070, 0.070, 0.040, 0.020, 0.010, 0.020, 0.010, 0.290, 0.040, 0.020, 0.020, 0.010, 0.010, 0.010, 0.160, 0.030, 0.030, 0.080]
Modelled = [0.06, 0.064, 0.078, 0.068, 0.079, 0.019, 0.009, 0.014, 0.016, 0.087, 0.074, 0.034, 0.018, 0.009, 0.023, 0.027, 0.053, 0.047, 0.019, 0.058]

# Set up plot
fig, (ax1, ax2) = plt.subplots(1, 2)

# Plot Measured (X) vs. Modelled (Y), with 1:1 line
ax1.plot(Measured,Modelled, 'ro')
ax1.set_xlabel("Measured values")
ax1.set_ylabel("Modelled values")
ax1.axline([0, 0], slope=1)

# Plot Modelled (X) vs. Measured (Y), with 1:1 line
ax2.plot(Modelled,Measured, 'ro')
ax2.set_xlabel("Modelled values")
ax2.set_ylabel("Measured values")
ax2.axline([0, 0], slope=1)

#obtain m1 (slope) and b1 (intercept) for Measured vs. Modelled
m1, b1 = np.polyfit(Measured, Modelled, 1)
#obtain m2 (slope) and b2 (intercept) for Modelled vs. Measured
m2, b2 = np.polyfit(Modelled, Measured, 1)

#add linear regression line to scatterplot 
ax1.plot(Measured, m1*np.array(Measured)+b1, color='orange')
ax2.plot(Modelled, m2*np.array(Modelled)+b2, color='orange')

# using padding
fig.tight_layout(pad=1.0)
plt.show()

## **EDIT**

# Full input data, including predictors, split by test-training
group = ["Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Testing","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training","Training"]
measured = [0.11,0.15,0.07,0.07,0.04,0.02,0.01,0.02,0.01,0.29,0.04,0.02,0.02,0.01,0.01,0.01,0.16,0.03,0.03,0.08,0.08,0.02,0.07,0.05,0.06,0.03,0.09,0.07,0.03,0.04,0.07,0.17,0.03,0.01,0.05,0.01,0.01,0.04,0.01,0.01,0.01,0.14,0.21,0.04,0.02,0.02,0.03,0.02,0.01,0.02,0.01,0.01,0.01,0.02,0.01,0.01,0.01,0.02,0.06,0.06,0.02,0.02,0.06,0.06,0.07,0.07,0.07,0.13,0.06,0.05]
predictor_1 = [1.231824555,0.615234151,1.165186392,0.525859101,0.735572055,7.67780908,8.958903593,8.900461024,7.84647476,2.655325443,2.792221068,5.769693553,8.185012004,8.787211459,5.962420116,5.297960321,1.479561438,5.255290898,7.885828985,0.738002703,0.483714239,1.928896113,0.514826816,1.629628135,1.067315635,0.487807436,0.961284723,0.639645339,2.083082213,1.816748332,1.14119713,0.798762487,7.004250655,12.22349063,8.000742572,9.017686869,11.75561384,3.944032552,9.432875808,9.136357158,8.595211952,3.552441934,1.014219737,3.166220771,5.648604781,5.078663497,4.869929446,7.760899685,8.529723683,6.461895963,6.707680902,5.852565035,5.046068182,7.239011985,7.770876954,7.852262871,7.432389029,0.907447671,0.654238481,1.823821963,9.255090086,0.885662936,1.296456938,0.741698994,2.262652753,1.785146869,2.334035797,0.852653873,1.615497552,2.260345848]
predictor_2 =[20.94380929,22.28754677,38.81228455,25.88769612,37.95297372,5.306122449,0.71199097,6.019852706,3.647733194,53.617942,41.58515404,13.3125,6.267806268,0.436681223,2.730375427,4.155021427,15.40168664,24.30683482,6.862044317,16.93954187,49.46439824,35.71915474,51.2345679,19.19191919,43.77224199,26.84396699,20.8882198,10.34992607,29.30439073,27.18836218,45.91439689,59.22795797,1.314405889,0.245813489,7.367230134,0.675378065,0.442993705,43.89086595,6.382160707,10.60371517,2.124943,47.69738512,73.13639511,34.74692202,5.964254636,20.48104956,12.4137931,11.84210526,1.167031364,26.07170694,0.805951643,0,3.050847458,8.497854077,3.602941176,2.43567753,10.25430681,34.79212254,67.48427673,8.855545749,2.836335761,16.58502449,15.38725154,7.392759355,34.95531281,22.39939256,3.680501175,9.542991065,6.595855307,11.16022099]

Answer 1

Whether a value is observed or modelled is irrelevant. What matters is whether or not a value has an error or a random distribution that you want to study.

Two common cases

It is common to consider a conditional distribution of some variable based one or more other variables. Then we have the value for which we want to determine the (conditional) distribution on the y-axis and the value on which we condition on the x-axis.

• Physics experiments are a typical example. In those cases an experiment is often performed by changing/controlling some variable and this variable is considered an 'independent' variable which has little measurement error. Then the y-axis represents an observed variable that is the 'dependent' variable. The relevant question in such experiments is to determine $Y|X$ the distribution of $Y$ given $X$. And that is exactly what ordinary least squares regression does.

• In observational studies, as often seen in the field of economy, nutrition and health, or sociology, there are no 'independent' variables. The experimenter has no control over variables and is just observing patterns.

  In that case there is no natural variable to be placed on the x-axis. Still, one might be interested in a particular direction of patterns. For instance because of some application or goal for which one performs the study.

  An example could be a doctor that wants to predict some risk based on a set of variables. The risk is modelled as function of some variable, so risk is on the y-axis and the variable on the x-axis (but that doesn't mean that there is a causal relationship in that direction, it is just that the doctor wants to know the statistical model, and not the causal model).

Comments on your case

For your case it is not directly obvious what should go on the x-axis.

I personally prefer to have the modelled value on the x-axis, but that is because I am often dealing with physics experiments where the modelled variable is a well controlled variable and the observed variable contains the error. Plotting the modelled variable on the x-axis can also be a way to decrease the dimensionality.

If you have multiple variables that you control in an experiment then it can be difficult to plot the output/observed value as function of all those variables (because it is multidimensional). But instead of all those variables you can replace them by the modelled variable. This might be your case as you could view your data as the 3d plot below

This does not always need to be the situation. The modelled variable can depend on controlled variables, but there might be a degree of error in the actual values and the values that were set. (e.g. you might control some concentration of a solution, but due to experimental variations the concentration has error)

Relating to regression

• You migh want to know just the statistical relationship. How well does my model describe observations. What is the error that we make with a particular model? For such questions you want to characterise the distribution of the observations as function of the modelled values.
• You might want to use regression as a goodness of fit test. For this case you want to use a correct representation of the error distribution. You might not just have statistical variation in the observations but also in the modelled values. This is a more nuanced situation than either regressing observed vs modelled or regressing modelled bs observed. In this case you want to express both errors together and use something like Deming regression.

Another note on your plot. Possibly you might want to change the scales of the axes such that the line observation=modelled has a 45 degree angle. The use of 45 degree angles makes it easier to compare differences (see also this question and answer about slopegraphs)

Related questions

Inverse Regression vs Reverse Regression

Effect of switching response and explanatory variable in simple linear regression

If in this problem I regress $x$ on $y$ instead than $y$ on $x$, do I need to use an error-in-variables model?