5
$\begingroup$

I've been trying to analyze the possible association between the number of questions done while studying and the average result of users after several testing sessions. The average growth variable is a mean taken after multiple tests. I don't necessarily expect the amount of questions done to have significant impact on the results, contrarily to intuition. enter image description here

Given the Central Limit Theorem, this distribution of means is already expected to be approximately normal. It seems to "converge" towards the the mean. I could try to look into checking for a "ceiling" to how many questions one should answer after the gains start to be marginal, but I can't say that's a fair angle, since there's no trend after more questions are done, it is not upwards nor downwards and can go both ways.

Another feeling I get, is that "heavy users", people with dozens and dozens of questions answered, tend to fall closer to the mean and not get extreme results.

In short, it feels like I could ignore the $X$ variable and $Y$ would still behave the same (it sort of does when I plot the histogram of $Y$ alone. There doesn't seem to be a direct relationship, and if there is, it's not linear, so looking at correlations seem pointless. How to model this relationship? Are there any caveats I'm not seeing?

EDIT: added the density plot to hopefully help with diagnostics, excluding the possibility of trends being concealed by overplotting.

Improve this question
$\endgroup$
4
  • $\begingroup$ How many observations? There seems to be a lot of overplotting ... $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 2 at 0:07
  • $\begingroup$ They are plenty, around 200k. Is overplotting avoidable? The range of possible values for $Y$ is rather limited [-1,+1], always in increments of .1, so indeed a lot of repetition is going on. After you comment, I tried playing with the transparency setting of the plot, as well as with plot size, trying to make the regions more distinguishable, but couldn't really improve the solution. $\endgroup$
    – Pedro Alonso
    Commented Apr 2 at 0:30
  • 1
    $\begingroup$ You could try some other technique, like 2D density plot. Here are some resorces to deal with overplotting r-graphics.org/recipe-scatter-overplot, stackoverflow.com/questions/7714677/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 2 at 0:37
  • $\begingroup$ edited to add a density plot to help whoever comes across this post. $\endgroup$
    – Pedro Alonso
    Commented Apr 2 at 19:15

1 Answer 1

Reset to default
1
$\begingroup$

Here's a partial answer that is mostly aiming to help you ensure you are identifying the relationship of interest. Some things I would think about:

How could the timing of test questions be hiding or revealing the relationship? You are interested in within-person growth, so articulating the timing of your causal variable and explicitly modeling that should be important. Does everyone take the first test with zero study questions? If yes, and if $person_i$ can only take study questions between $attempt_{i,0}$ and $attempt_{i,1}$, then your approach of averaging all $attempt_{i,1+n}$ seems reasonable to reveal the relationship. If study questions can be taken at any time before the first attempt and between subsequent attempts, you could be underfitting.

For example, if study questions can be done at any time in relation to attempts, and if your data for study questions contains this information, you could be more specific. Imagine a person who takes the pre-test, barely studies, takes a second test with almost no improvement, does many study questions, takes a third test with a great deal of improvement. You would attribute only moderate growth for this person with many questions (due to the averaging of $attempt_{i,1}$ and $attempt_{i,2}$), because most of their questions could not affect $attempt_{i,1}$ at all.

Basically, if you have data on the timing of study questions it should be modeled explicitly. If you know that study questions can only be taken between the pre-test and the first attempt, your strategy is probably good (but if so, why are there multiple attempts and why would we expect continued growth?). You might consider separate plots for each differenced $attempt_{i,t}$ and $attempt_{i,t+1}$ pair with the study questions between each attempt on the x-axis (perhaps also controlling for study questions prior to $attempt_{i,t}$).

If you know that they can take study questions any time but do not have access to data on the timing of questions, you might have to accept you are missing out on important factors, in my opinion.

Also, why is the range of values different in the density plot?

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.