1
$\begingroup$

I have a dataset of ~800 observations. The observations are tracked according to two variables: a year and a value. The years are all between 1980 and today. I've divided the observations into two groups: Group1 contains all the observations from before 2000, and Group2 has all the observations from (and including) 2000 to today.

For each year, I've calculated the average of the value I'm tracking. There are roughly the same number of observations in each year. I created Group1_means, which contains 20 observations (each of them the mean of one year in Group1), and Group2_means, which contains about 20 observations (each of them the mean of one year in Group2). I want compare the collection of yearly means from Group1_means to the collection of yearly means from Group2_means. From some simple analysis and graphing, it's easy to observe that while the means of Group1_means (i.e., the mean of the means) and Group2_means are similar, Group1 has greater variability around its mean than Group2. In other words, the volatility associated with the value has decreased over time, even if the mean hasn't.

I'm trying to determine how "statistically significant" this difference in variability is. I've run a Levene Test to evaluate the difference between the variances of Group1_means and Group2_means (again, each group here consisting of the average value from each year within each group, meaning that each group has ~20 values in it, not 400). The Levene test shows a p-value of .12.

So here, finally, is my question. The work I'm doing is in the social science realm, for general public consumption (meaning not an audience of social scientists). Of course, if this were something like a medical trial, a p-value of .12 would be unacceptable. But for my work, being able to say something like "There's a 12% chance that the difference in variability we're seeing between the two groups is attributable to nothing more than randomness" seems acceptable to me, as long as I disclose it. Meaning that it doesn't "invalidate" my analysis of the decline in variability from Group 1 to Group2, as much as it makes it less reliable than if we saw a p-value of, say, the traditional (and yes, I know, controversial) 0.05. Right?

Also, as long as I'm asking questions, does anyone see any problems with my methodology here? Specifically: 1) taking the average value for each year in the two groups, 2) calculating the variance of the yearly averages in Group1_means and Group2_means, and 3) using a Levene Test to assess the difference between the Group1_means variance and the Group2_means variance?

$\endgroup$

1 Answer 1

1
$\begingroup$

You know your comment to be misleading.

It might be reasonable to disclose the p-value as you have if you include a comment like, “…where professional scientists typically consider 5% to be the highest acceptable value to declare a difference to be real.” By just saying 12%, you are leading people to believe that this is a small value, when people who have studied the underlying mathematics typically do not.

A lay audience is relying on you to do most of the hard for work them. Part of doing that hard work involves interpreting and explaining the results, not leaving them to use their limited technical knowledge to do the work you’re supposed to be doing.

$\endgroup$
2
  • $\begingroup$ I’m not so sold on the methodology, given the time series nature of the data, but that is a separate issue. $\endgroup$
    – Dave
    Jan 4, 2023 at 22:01
  • $\begingroup$ Thanks @Dave. I agree with you. I was just being casual / informal in the language I used in my question. I would explain the p-value with more detail in the actual piece, along the lines you've described. But I'm curious: could you expand a little on your concerns about the methodology? Big thanks again. $\endgroup$ Jan 5, 2023 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.