2
$\begingroup$

What I have to do?

Two courses (A and B) students CGPA and percentage of class attendance is given. I have to compare CGPA of two groups (group is based on attendance - high vs low) regarding students of each course. Some of the students of course A are not enrolled in the course B, that is, all of the students of course A and course B are not same.

So, What creates the problem?

Since all the students of two courses are not same, that is, CGPA data set is not same in each of the courses, will I have to correct p values for each comparison test of CGPA?

Update - to clear the problem

What is the nature of my data?

I am using actual data, that is, no data is collected from the students. Each course attendance is independent of each other. e.g. Attending course A class lectures does not ensure attendance of same student in course B, even if that student is enrolled in both of the courses. Moreover, I do not expect that course difficulty will have impact on class attendance.

Which type of statistical test I am using and how?

I am using standard T test. Reason of using T Test is, I assume normally distribution using Using Central Limit Theorem. When two groups (high and low) of data was found not having equal variances, I used Welch's T Test as suggested by this.

For each course, I used T test dividing the students into two groups - how and low attendant. Top one third percentile was considered as high attendant and bottom one third percentile was considered as low attendant as done this research in dividing Facebook users into two categories - high and low.

What is my hypothesis?

CGPA of the high and low attendant of classes differs significantly regardless the course.

Improve this question
$\endgroup$
3
  • $\begingroup$ It seems you're following extant papers without really considering if their choices were valid. The first thing you should do is plot CGPA against attendance. Also, if students can be enrolled in both courses attendance is most definitely not independent. I suggest that you make the plot I recommend for each class and post it in a new question asking about analysis. Further, you should explain whether you care about finding differences between the classes or whether they're being used as independent measures of the effect. $\endgroup$
    – John
    Oct 13, 2019 at 19:18
  • $\begingroup$ @John, thanks for your comment. can you please let me know, how attendance is not independent since attending one course does not ensure attendance of another course. Moreover, since we trying out the difference of CGPA between two groups, why does the question of dependence and independence of attendance come? Obviously, one can not ensure CGPA is dependant on attendance. $\endgroup$
    – Md. Sabbir Ahmed
    Oct 14, 2019 at 1:25
  • $\begingroup$ Often classes are missed for a reason and the reason will cross courses. Illness in one class will be correlated with illness in others among the students enrolled in both. $\endgroup$
    – John
    Oct 15, 2019 at 1:54

2 Answers 2

Reset to default
2
$\begingroup$

I like to think about what would happen to my conclusions if I added a bunch of additional variables that were just random noise (but had names that fit the problem) and how my conclusion may change if I do or don't adjust for multiple comparisons.

If you will declare "Success" if you get any significant p-values then not adjusting will mean that you will probably declare "Success" based on one of the noise variables, but adjusting for multiple comparisons will decrease this likelihood.

On the other hand, if you will discuss the significance of a specific variable whether or not other variables are significant or not, then adding noise variables and adjusting for multiple comparisons will tend to mask information about the variable of interest, so it would be better to not correct in this case.

Another option instead of correcting p-values is to switch to a Bayesian analysis, more specifically a hierarchical model. Frequentist methods correct for multiple comparisons by making confidence intervals wider or p-values larger to correct for the fact that out of many comparisons a few are likely to be far from the truth by chance (but makes no adjustment to the point estimates). Bayesian hierarchical models instead shrink the point estimates (and corresponding intervals/posteriors) towards a common estimate, which I think gives better inference.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your answer. You have set some options with additional information. Which one do you prefer to do in this case? Specifically, I am asking should I correct p values or not? $\endgroup$
    – Md. Sabbir Ahmed
    Oct 11, 2019 at 16:32
  • 1
    $\begingroup$ @SabbirAhmed, It is hard to know exactly what to tell you based on a short description. You should probably consult with a statistician that can take the time to understand your data and the question(s) trying to be answered. From your description I think you probably do not want to correct the p-value. But note that "Some guy on the internet said ..." should carry a lot less weight than "My research suggests ..." or "My consultant that I spent time explaining this to says ...". $\endgroup$
    – Greg Snow
    Oct 11, 2019 at 16:47
  • $\begingroup$ I have updated my question. If possible, please read. $\endgroup$
    – Md. Sabbir Ahmed
    Oct 11, 2019 at 21:23
  • $\begingroup$ @SabbirAhmed, your expanded explanation makes this sound more like Analysis of Variance is more appropriate than multiple t-tests. If you are interested in differences between the courses then this would be a 2-way ANOVA, if you just want to adjust for the different courses, but the specific courses are not of interest, then a Randomized Block ANOVA with course as the blocking factor and attendance as the fixed effect of interest. $\endgroup$
    – Greg Snow
    Oct 14, 2019 at 15:29
1
$\begingroup$

I think we may need to learn a bit more about your experiment and hypothesis in order to be able to advise whether an adjustment for multiple comparisons is necessary.

It sounds like: 1) You are interested in comparing the cGPA between high- and low-attendance classes. 2) Your data are made up of cGPAs observed among students in two different classes. 3) Some students were in both classes.

There are a lot of potential issues here, so I'll try my best to walk you through the considerations you need to take.

It sounds as though you are considering an ANOVA test to detect a significant difference between the mean cGPA of high-attendance classes and low-attendance classes. Note that ANOVA is called an omnibus test, which means that it will tell you if one or more of your group means are different from the others, but it won't tell you which one. If you performed this test, found a significant difference, and then ran one or more pairwise tests to determine which group is different from the others, you would be expected to perform a multiple-comparisons adjustment. However, there is no "penalty" for running a test on data in which more than one group is represented--you won't even be thrown into "statistics jail".

However, if the group differences are important to your research question or hypothesis, you may find that a simple approach is not sufficient. I.e., do you expect that Class A and Class B will affect the difference between high-and low-attendance classes. If so, your question starts becoming a multivariate one, for which it is likely easier to simply fit a model that can account for the multiple variables.

I highly suggest that you seek out the assistance of a statistician, as I do not have a complete picture of your data and hypotheses, and cannot confidently advise you. For example, If students can appear more than once in your dataset (i.e., were in class A and class B), the assumption of independence common to many statistical tests may be violated and a univariate test such as ANOVA would not be appropriate.

In order to answer your question fully, I would need to know more specifically: "What is the nature of your data?" "What is your hypothesis?" "what statistical tests do you plan to perform?" "

Improve this answer
$\endgroup$
2
  • $\begingroup$ @ G. Vece Thanks for your interest to help me. I am trying to clarify the question as much as possible. I am updating my question. $\endgroup$
    – Md. Sabbir Ahmed
    Oct 11, 2019 at 20:48
  • $\begingroup$ Updated, please check $\endgroup$
    – Md. Sabbir Ahmed
    Oct 11, 2019 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.