1
$\begingroup$

I have ~650 scores on various measures of academic performance in a subject. One novel aspect of the subject was that a study website was conducted on which students could answer practice questions.

First assignment score (out of 100, max 96)
Second assignment score (out of 100, max 97)
Final exam score (out of 100, max 95)
Website performance points (max 1617)
Website questions answered (max 183)

Here are the Pearson correlation coefficients:

Correlation coefficients

The assignments and exam were compulsory, while students were merely encouraged to use the website. About 50% of students did not use the website, perhaps because they preferred to study some other way, or perhaps because they didn’t study at all. Either way, they scored 0 on both Website performance points and Website questions answered.

The study website related to material highly relevant to the exam, and only weakly relevant to the assignments. Thus I regarded it as a good sign that the website measures correlate more closely with exam performance than with assignment performance.

The correlation between website performance points and website questions answered is extremely high. This is because of the way website performance points were scored. Students received 10 points for each correct answer, and -2.5 points for each incorrect answer, and could answer as many questions as they liked. Thus students who answered a lot of website questions tended to achieve a lot of website performance points, even if they were not particularly accurate.

How can I best assess the efficacy of the website, given the data I have? Ultimately I am wondering whether the website helps students do better on the exam, or whether it merely attracts students who would have done well on the exam anyway.

What additional data would help me better assess the efficacy of the website? You can assume that it will never be possible to randomly assign some students to receive the study website while others do not. However, I could access other data, for instance the exam scores students got on related subjects.

What should I do with the ~300 people who didn't use the website? Should they be removed from the analysis?

$\endgroup$
1
$\begingroup$

One possibility is to add an extra variable that indicates whether an individual used the website (no = 0, yes = 1). Then it's straightforward to set up a regression model for analyzing the relation of Final Exam Score to the other variables. Given the high relation between numbers of web questions answered and the web performance score, you might want to remove one of these from your analysis to avoid a possible multicollinearity problem.

A standard linear regression of exam_score against the 4 predictors--assignment1, assignment2, webNoYes, and (say) questionsAnswered--then includes information from all cases. Regression coefficients for the first 3 predictors are straightforward to interpret statistically, although I see no way to determine from these data whether the coefficient for webNoYes represents self-selection by those who would have done well anyway or a real advantage of choosing to use the web site at all.

For questionsAnswered, this will be zero for all those who didn't use the web site. The regression coefficient for questionsAnswered can be thought of as the extra increase in exam_score per web question answered, given that the individual chose to use the web exercises in the first place. If you choose to use as a predictor the web-question performance instead of the number of questions answered, a similar interpretation would hold. My first guess is that the number of questions answered may be a better measure of effort expended on the web site, while the web-question performance might be more related to other measures of overall student aptitude.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.