Comparing simulated categorical data against empirical categorical data I wondered if anyone could give me some advice on this problem?
I need to compare one sequence of (mutually exclusive) states against another (of the same length) to say if they match or not.
For example:
a,a,a,a,b,b,c,c,c,b,c,a,a,d,d,d,d
a,a,a,a,a,a,a,c,c,c,c,c,b,b,b,d,d
Each sequence is actually about 3600 items long and there are 17 symbols (states).
The data are categorical labels, with no order in the symbols, so I can’t use a Kolmogorov-Smirnov (K-S) test.
(The data are not numeric, I can’t form a cumulative mass distribution.)
A simple term-by-term comparison, using scores of match=1, nomatch=0, could give a relative measure, but how then to assess significance?  Chi-squared test?
The sequence is actually a time-series, so it would be nice to somehow use the information contained in the order?
Is there a statistical procedure to do this comparison?
Alternatively, I could, with significant loss of information, summarize each sequence by calculating the frequencies of each of the (in reality 17) states:
For example, assuming 4 states: seq#, a#, b#, c#, d#
seq1, 6, 3, 4, 4
seq2, 7, 3, 5, 2
This data can be plotted on a bar chart – but not a histogram.
Again, I can’t use a K-S test, because the 4 states are categorical labels, not numeric, let alone continuous.
The states are mutually exclusive.
I believe a Chi-squared test would give an overall measure of the match and whether it was significant?
What I really won’t though is to compare the raw sequences.
I have over 200 pairs of sequences to compare.
Thanks!
Further details:
Apologies - I tried to reduce the amount of verbage!
As part of my PhD research I observed classroom lessons and collected data on who did what and when.
Then I developed an agent-based simulation of the lessons.
Now I need to compare the simulation output to the empirical data in order to calibrate and eventually validate the simulation.
The sequences are sequences of activity states, every second for about an hour, for each student and for the teacher.
I am trying to get my head around how to say that a simulated student is close enough to the empirical student, in terms of their behaviour, their sequence of activity states.
I have calculated simple comparisons of various summary performance indicator means (e.g. time in productive states), but the results of two completely different lessons can give almost the same aggregated value.
Greater resolution is needed:  hence my interest in comparing sequences of activity states.
 A: Your intuition is correct that you should compare the raw sequences rather than e.g. comparing the counts of the number of times each state appears, because aaabbb is likely to be quite different behavior from bbaaab.
However, significance isn't helpful because you want to examine the degree of association between the simulation and the data. Very weak association could be significant, and very strong association could be non-significant, depending on your sample size and the power of the test you use.

A simple term-by-term comparison, using scores of match=1, nomatch=0, could give a relative measure

This is a good way to start. The greater the proportion of matches, the greater the association. It's wise to compare this proportion to the proportion of times that the real student was in his most common state, in order to get a sense of what proportions are actually good. 97% accuracy might sound good, but if the real student is in a single state 98% of the time, it's really bad—your simulation is worse than a trivial model that predicts the most common state every time.
