Comparing survey data to passively measured data Basic Situation:


*

*I have quite a large sample of around 10k people.

*In that sample I have two values for each person: a passively measured value of an amount of time, as well as their estimation of said amount of time (via survey).

*So we basically have a paired sample situation for metric variables.

*I need to get to some kind of conclusion as to how similar the two variables (measured vs. survey) are distributed.

*(I can work with either R or SPSS)


Complicating Circumstances:


*

*the survey data is biased compared to the measured data. The estimations are on average quite a lot higher than what was measured.

*This was known/assumed (through different sources) before the data was gathered and is generally accepted, as the analysis is not about the central tendencies or variation of the two values, but their (dimensionless) distribution patterns.

*Because of this, it was never intended to compare the absolute values.

*As a result, the analysis should simply be concerned with their z-scores.


My Questions:


*

*Would a 'goodness of fit' test between the two distributions of the z-scores be a suitable way to go? (Kolmogorov-Smirnov comes to mind)

*Are there any further implications that come up because of the standardization to z-scores? Am I missing anything here?

 A: It is possible that standard scores of the two measurements of time will be sufficiently similar in shape to 'pass' a Kolmogorov-Smirnov test (that is, not reject $H_0$ that the two distributions are the same.
However, it seems you are really hoping that the two scores will be correlated. I suggest you look at Spearman correlation if (more usual)
Pearson correlation does not measure what you want.
Of course, I have no clue what values or distributions your data may take, but here are data that may illustrate what I have in mind.
I will use 1000 subjects instead of 10,000. The purpose of this is to
show how you might look at your data, not to try to predict what you will find. (Note: In R the ks.test is limited to a sample of 5000.)
set.seed(2019)  # for reproducibility
x = runif(10^3, 30, 120)        # uniformly dist'd actual times
y = 20 + x^1.5 + rnorm(10^3, 0, 15)  # perceived times
x.s = (x - mean(x))/sd(x)       # standardized ...
y.s = (y - mean(y))/sd(y)       # ... times
ks.test(x.s, y.s)               

The two versions of time (standardized) barely pass the K-S test.
        Two-sample Kolmogorov-Smirnov test

data:  x.s and y.s
D = 0.058, p-value = 0.06919
alternative hypothesis: two-sided

cor(x.s, y.s);  cor(x.s, y.s, meth="spear")
[1] 0.9957331    # Pearson correlation
[1] 0.9987191    # Slightly larger Spearman corr

par(mfrow=c(1,3))
  hist(x.s, prob=T, col="skyblue2")
  hist(y.s, prob=T, col="skyblue2")
  plot(x.s, y.s, pch=".")
par(mfrow=c(1,1))

Histograms look rather different in spite of passing K-S test.
The scatterplot shows higher correlation than is likely realistic
for your actual data.

Addendum on 'distance' between ECDFs, per Comments.
par(mfrow=c(1,3))
 plot(ecdf(x.s), col="blue"); plot(ecdf(y.s), col="orange")
 plot(ecdf(x.s), col="blue", main="Overlay")
  lines(ecdf(y.s), col="orange")
par(mfrow=c(1,1))
ks.test(x.s, y.s)$stat
    D 
0.058


You can read about the two-sample K-S statistic
on Wikipedia.
The statistic D in R is essentially the largest vertical
distance between the two ECDFs. [If this is potentially publishable work, reporting P-values as you mentioned may get you caught up in the current
controversy about the meaning of P-values. Useful discussion is possible, but IMHO total ignorance
of statistical inference has been no bar to entering the current debate.]
