1
$\begingroup$

I wanted to observe the performance improvements in older people playing online cognitive games. I had 33 participants that were playing 35 online games freely and independently during 10 months. Some participants played more than others. Every time they were playing a specific game a record was created. By the end of the study, I got 28.000 observations. I clustered the data by game, so I´m presenting the number of people that played a game, the number of times each people played a game, their minimum and maximum scores and the improvement in scores.

By doing that I got 677 observations in total. In order to have independent observation (as the same people were playing the different games) I ran a regression model for each game including improvement in scores as depend variable and Number of times played; Age, Education, positive attitude towards technology and initial cognitive state as independent variables.

I have received some questions about that is not valid to have regression models having 33 participants. Even though I argued I have several observations for each subject.

An In the second place some questions about interdependence between observations, that I think is solve by analyzing the results for each game independently.

Please let me know if you know some references to support my claims and if you have any suggestion for my analysis.

$\endgroup$
1
$\begingroup$

From your question it is not entirely clear what your research question is. You say that your dependent variable is the improvement in scores. This implies that you are interested only in relative differences when people play the same game multiple times and which factors influence this improvement.

In sum, your plan is actually to model first differences, but not difference the independent variables (the latter would be the first-difference estimator, or the fixed effects estimator). This could be a valid analysis, provided you have data for users each playing a game multiple times, i.e. at least two times and if more than that you have to specify how you define the score improvement in that case. (Something to realise though is that just modelling the actual scores could also be interesting, in which case you would enter longitudinal/panel data modelling.)

The next question is what model you should come up with and how to estimate it. You suggest to model each game separately. This is possible, but probably not very efficient as there may be high correlations between games. Therefore it is probably better to pool your data. For this purpose it is then useful to transform the game scores to some comparable standard if they are not, e.g. scores between 0 and 100.

The following points should be kept in mind:

  • There are probably correlations in the score improvements between games for the same people, so you should consider modelling this;
  • There are probably correlations in the score improvements between people for the same games, so you should consider modelling this;
  • You consolidated the time dimension by computing aggregate scores, so you don't have to consider modelling time.

To model such relations (which if done well may provide unbiased estimates) you can look into the many types of models available. For example, you could just add dummy variables for games, because different games probably have a specific fixed effect (e.g. some games are easier or more difficult and have a different base-score, thus aggregating different user's scores for games). Or you could make a full-fledged panel data model/mixed model using both fixed as well as random effects for games and/or people.

This requires some experimenting on your behalf and understanding the different models. What is possible also depends on the type of regressors you have. For example, you cannot add game-specific independent variables and at the same time add game fixed effects, as the fixed effect subsumes the variables.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ No problem! Good luck with your model! $\endgroup$ – Nick Jun 20 '17 at 21:45
1
$\begingroup$

The observations within an individual are not independent of each other. Additionally, observations within a game are also not independent of each other. This may cause you to violate the assumptions of multiple regression, and your standard errors might be too small. If this is the case, you could be reporting statistical significance where there is none.

One thing you could do would be to conduct a mixed-effects regression. This can account for the clustering of observations within individuals and games. This document by John Fox is a simple introduction to mixed models.

In your study, you have longitudinal data. You can create a time measure of time elapsed since the start of the study - there are many ways one could go about this. If you were to do this in R, the simplest model might look something like this, were you to use the lme4 package:

lmer(score ~ number_times_played + age + education + attitude + (1 | game) + (1 | person))

You would obtain the effect of number_times_played, age, education, attitude, and you would be accounting for the non-independence of observations within persons and games. Doug Bates, the author of the package, has a freely available book on using the package. I believe you can also run such models in SPSS. You can certainly run them in SAS.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.