I have ~650 scores on various measures of academic performance in a subject. One novel aspect of the subject was that a study website was conducted on which students could answer practice questions.
First assignment score (out of 100, max 96)
Second assignment score (out of 100, max 97)
Final exam score (out of 100, max 95)
Website performance points (max 1617)
Website questions answered (max 183)
Here are the Pearson correlation coefficients:
The assignments and exam were compulsory, while students were merely encouraged to use the website. About 50% of students did not use the website, perhaps because they preferred to study some other way, or perhaps because they didn’t study at all. Either way, they scored 0 on both Website performance points and Website questions answered.
The study website related to material highly relevant to the exam, and only weakly relevant to the assignments. Thus I regarded it as a good sign that the website measures correlate more closely with exam performance than with assignment performance.
The correlation between website performance points and website questions answered is extremely high. This is because of the way website performance points were scored. Students received 10 points for each correct answer, and -2.5 points for each incorrect answer, and could answer as many questions as they liked. Thus students who answered a lot of website questions tended to achieve a lot of website performance points, even if they were not particularly accurate.
How can I best assess the efficacy of the website, given the data I have? Ultimately I am wondering whether the website helps students do better on the exam, or whether it merely attracts students who would have done well on the exam anyway.
What additional data would help me better assess the efficacy of the website? You can assume that it will never be possible to randomly assign some students to receive the study website while others do not. However, I could access other data, for instance the exam scores students got on related subjects.
What should I do with the ~300 people who didn't use the website? Should they be removed from the analysis?