I would like to perform a study with physicians of three levels of experience. They shall perform a task under three different conditions. For each condition, the task is performed multiple times (e.g. 70 times). While performing the task, numeric metrics are recorded for each task. This results in multiple data points per measurement (=condition) and in three measurements per physician. I would like to find out how the recorded metrics differ between levels of experience and between types of condition. When calculating the statistical power I cannot find an option that takes into account the multiple data points per measurement. My problem is, that I only have limited number of participants (2 for each condition), and the required effect size is 0.3. This is why I try to compensate for the low count of participants by increasing the number of data points per condition. Is there any solution to this? Or do I need another statistical test (e.g. complex mixed-effects model to compute the statistical power a priori?). Thank you in advance.
4
-
$\begingroup$ I really struggle to understand what exactly is going on in your experiment. Could you maybe please try to clarify a bit all the varying elements, and the total number of datapoints you expect to have? $\endgroup$– Guillaume DehaeneCommented Jul 5 at 13:49
-
$\begingroup$ Sure, sorry for the confusion: The participants consist of 6 physicans (2x low experience, 2x medium experience, 2x high experience). They are asked to write a medical history (task) by using his/her own words (condition 1), by using template 1 (condition 2) and by using template 2 (condition 3). For each condition, the physcian is asked to write for example 20 medical histories. During the writing, certain values are recorded (e.g. word count). $\endgroup$– HiddigeiCommented Jul 5 at 14:17
-
$\begingroup$ This results in three groups (for each experience level) and three conditions (report mode). For each combination of participant and report mode, there are multiple data objects (multiple word counts, as they are asked to write multiple medical histories). I hope this clarifies my question :) $\endgroup$– HiddigeiCommented Jul 5 at 14:19
-
$\begingroup$ All in all, I will have 20 data points per physician per condition = 20*6*3 = 360 $\endgroup$– HiddigeiCommented Jul 5 at 14:22
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Maybe not a complete answer, but a question, a comment, and some suggestions.
- Question: in order for your physicians to capture the medical history, they need some subjects (patients, real or simulated). How many different subjects will there be? 360? (every capture is of a different patient's history). Or only 60? (the 6 physicians all capture the same patient's history; i.e. the data is "paired", but different patients are used for each condition). Or only 20? (the same patients are used for all conditions and all physicians; which would be problematic...). That changes how one might analyze the data.
- Comment; rather than asking "how should I analyze the data?", you should start by telling us what you are investigating? What question are you trying to get an answer to. Are you trying to show that physicians with different levels of experience capture medical history differently? Then why use 3 conditions? That becomes a confounding variable which does not help answer the question. So you are probably trying to see if using some form of template "helps, or improves" (leaving the definition of helping/improving up to you). Then why use 3 different levels of experience? To control for possibly another confounding variable? (i.e. the amount of "help/improvement " provided may depend on the level of experience. OK, this would make good sense, except for the sample size. With only 2 per level of experience, you have little hope of convincing anyone that the level of experience has an influence, or does not. The CI's on the effect of experience will be too large to reach a credible conclusion (particularly with the small expected effect).
- Suggestion 1; I would forget about the level of experience, if all you can do is 6 physicians. Instead I would block against that confounding variable, i.e. selecting 6 physicians with the same level of experience. But maybe that is too late (i.e. you already have the data, or are in the process of collecting it).
- Suggestion 2: I would then suggest that you ignore the level of experience, and treat all 6 physicians as a single random sample (6 is not large but much better than 2; you can think of it as clustered sampling). Then you basically have a repeated measured 1-way ANOVA (1 DV: template: 3 levels. And 6*20=120 replicates per level. Quite a good sample size). You can also think of it as a DOE (Design of Experiment) (same math under the hood as repeated measures 1-way ANOVA). It will tell you if, on average, the template has an effect...
- Suggestion 3: you could instead treat the data as a 2-way repeated measures ANOVA (i.e. a 2-factors full factorial DOE: 3 levels of template, 3 levels of experience, 40 replicates by combination of levels). Unlikely to find some significant effect due to the small sample size, but you may get lucky?
