0
$\begingroup$

I have calculated the RM-function from the MANOVA.RM package in R and now need to calculate the statistical power of the test and the effect size for its Wald-type-statistic, including the bootstrapped version.

"The RM function calculates the WTS and ATS in a repeated measures design with an arbitrary number of crossed whole-plot (between-subject) and sub-plot (within-subject) factors. The resampling methods provided are a permutation procedure, a parametric bootstrap approach and a wild bootstrap using Rademacher weights. The permutation procedure provides no valid approach for the ATS and is thus not implemented." https://cran.r-project.org/web/packages/MANOVA.RM/vignettes/Introduction_to_MANOVA.RM.html

The RM function has only one dependent variable (it is not a MANOVA). In my 2x2 model design, I combine a within- and a between-factor. I will also need to interpret the wild bootstrap. I used the RM function because my data are non-parametric and my sample size is only n=21 in total. Generally, I work with R and SPSS.

I would greatly appreciate your help!

Edit:

  1. Why do I calculate the power of the test? Before conducting my calculations and choosing the design, I had calculated the necessary sample size to conduct repeated measures ANOVAs with a given power (0.80), using G*Power. But as my study is part of a yet unfinished project, the total sample size is not as large as the needed one. And as my data (or model) are non-parametric, I am not calculating ANOVAs in the first place. That is why I now want to a posteriori calculate the power that the tests achieved.

  2. Why do I calculate the effect size? I would like to have further information on the importance/size of the effects I found in my analyses. Is there a way of calculating that for a Wald-type statistic?

  3. What hypotheses am I testing and what data do I have? I have three hypotheses and calculate an RM function for each of them. My hypothesis are: Do children with social anxiety disorder experience lower 1)self-focused attention 2)anticipatory rumination 3)post-event processing, if during a social performance task they get a) help from their parents or b) self-efficacy is induced? So I have 2 binary independent variables: support(yes/no); type of support (parental support/self-efficacy). The dependent variable is continuous (mean of questionnaire items). My data is not normally distributed nor following any other specific distribution. However, there are no outliers and the variances were homogenous (Levene-Test, Box-Test).

As an alternative to the ANOVAs I had planned, I also tried calculating Generalized Estimating Equations (geeglm function from the geepack package in R). "The geeglm function fits generalized estimating equations using the 'geese.fit' function of the 'geepack' package for doing the actual computations. geeglm has a syntax similar to glm and returns an object similar to a glm object. An important feature of geeglm is that an anova method exists for these models." https://cran.r-project.org/web//packages/geepack/geepack.pdf

It shows me very similar although not the same Wald-type statistics and significances. But I read, that to conduct robust and reliable Generalized Estimating Equations, large sample sizes n>100 are needed.

$\endgroup$
4
  • $\begingroup$ I have found this question on how to calculate the effect size for a Wald test: stats.stackexchange.com/questions/336176/…. I am doubting if the answer is correct and I struggle to find a source confirming this. $\endgroup$
    – Luise
    Commented Apr 13 at 18:27
  • $\begingroup$ Welcome to Cross Validated! Please edit the question to explain why you need to calculate power and this type of effect size. First, power calculations are important for study design but generally aren't appropriate once a model has been fit. Second, as a comment on your linked page says, Wald tests don't usually have a meaningful "effect size" analogous to that of an individual predictor in a model ... $\endgroup$
    – EdM
    Commented Apr 13 at 20:19
  • $\begingroup$ Third, please include more information about your data and the hypothesis you're testing. Remember that it's your model that might be non-parametric, not your data. I'm not sure that the RM function is a good choice here. With more information you might get a suggestion for a better approach. $\endgroup$
    – EdM
    Commented Apr 13 at 20:22
  • $\begingroup$ Thank you for the warm welcome and your suggestions, I appreciate it a lot! I have edited my question based on your comments :) $\endgroup$
    – Luise
    Commented Apr 14 at 13:20

1 Answer 1

1
$\begingroup$

First, as this page and its many links explain, the time to do power calculations is before you do the study. A power estimate is just your chance of finding a "significant" result if the true effect has a certain size. Once you've done the study you either found the "significant" result or you didn't. If your initial power estimate says that you haven't yet gotten the necessary sample size, focus on getting the necessary sample size.

You are certainly welcome to use the results of this study to help design a new study. If you do that, you should consider basing your design on a type of model that might better match your data; see below for a suggestion.

Second, the word "effect size" can take several meanings. See this Wikipedia entry, for example. It refers to some type of change in outcome associated with a change in a predictor. Sometimes it's a difference in outcome divided by a measure of dispersion of the predictor in the underlying population, like the standard deviation used in Cohen's d', providing a "standardized" effect size. Ideally, that type of "effect size" gets a more precise estimate as you increase the sample size, but the "effect size" is considered to be representative of the underlying population rather than a function of sample size per se.

That's not what you get with a Wald test. A Wald test on a single coefficient* is the ratio of a coefficient estimate to the standard error of the estimate. That standard error tends to decrease as the sample size increases. A Wald test provides information about the precision of the estimate, but doesn't take into account anything like the standard deviation of a predictor variable in the underlying population.

In your situation with binary predictors, however, you can use a simple "unstandardized" type of effect size: the difference in outcome when a predictor changes from one level to the other, given the level of the other predictor. That's presumably what you and your audience care the most about.

Third, your data aren't strictly continuous if they are based on responses to questionnaire items that have upper and lower limits. You then have a fixed number of ordered outcome values, which often are best handled by ordinal regression. That avoids the usual worries about normality and so forth, and can be considered to be an extension of some well-known non-parametric tests. It can be used with repeated-measures designs.

Sometimes, however, the outcomes in a study using questionnaire responses as outcomes are well enough behaved that the assumptions underlying ANOVA are good enough. It's the normality of the residuals around model predictions that matters, not the normality of the observed outcomes themselves. If the residual variances are homogeneous, as you suggest, you might be OK with a simple repeated-measures ANOVA model. You also might consider a multivariate repeated measures model that handles your three outcomes (and their likely correlations) together rather than separately.


*There also are Wald tests on entire models or on sets of coefficients, but the basic idea is the same.

$\endgroup$
3
  • $\begingroup$ Thank you very much for your response, this is very helpful. Do you know any source I can cite stating that, even when the normality assumption is not met, if the residual variances are homogeneous, you might be OK with a simple repeated-measures ANOVA model ? $\endgroup$
    – Luise
    Commented Apr 14 at 17:51
  • $\begingroup$ @Luise Assumption 4 on this page about repeated-measures ANOVA in SPSS might be a source. What you need for reliable p-values is normality of the coefficient-estimate sampling distribution, which is assured by normality of error term but can occur under weaker conditions. See the central limit theorem. It might help to read this page about whether testing for normality even makes sense. $\endgroup$
    – EdM
    Commented Apr 15 at 21:00
  • $\begingroup$ @Luise as Frank Harrell says in an answer on this page, you could be better off with a semi-parametric ordinal regression model. It's designed for situations like yours, with a set of ordered outcomes having limits and where you can't be sure that each step between outcome levels is really the same "size" in some sense. It's a generalization of some standard non-parametric tests. Even under the normality assumption it can be almost as powerful as a parametric model but it makes no assumptions about residual distributions. $\endgroup$
    – EdM
    Commented Apr 15 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.