Power analysis for observational study of secondary data I am involved in a project that is attempting to measure the impacts of person-level, county-level and medical service area-level predictors on individual probability of returning to a hospital within a year window. 
I see this as a 3 level random effect logistic regression model.  There are lots of predictors, probably 10 at the person level, 10 at the county and probably 5 at the hospital level. There are nearly 300,000 individuals nested withing 3000 counties and 154 hospitals.  I would think that I should have good statistical power to detect differences in my outcome.  My question, is: since the sample is already determined and I'm not collecting data (this is all secondary data), is a power analysis even appropriate?  If it is appropriate, but my PI can't specify the effect size they are looking for, then what?
I would appreciate any input the list may have on this problem.
Best,
Corey
 A: Power analysis is typically undertaken a priori to conducting an experiment, in which the researcher, given a pre-specified effect size, identifies a sample size that sufficiently controls type II error. So two things are immediately different in observational designs, 1) you can't change the sample size , 2) the effects of interest are likely to have non-zero correlations with one another (whereas in experiments they should be zero by design.) 
So even with your very large individual sample, I would not take for granted the power of tests for the direct effects of the higher levels, especially given ecological correlations between aggregate units tend to be very high. Also I don't know the distribution of the outcome, and if re-admittances are rare one needs (much) larger sample sizes than in typical experiments with continuous outcomes. Further, cross-level interactions may potentially have little power as well. 
As far as motivation for conducting power analysis in observational settings (to be clear power analysis before the study is undertaken), one may wish before hand to check whether the study is sufficiently powered, or whether more data should be collected before the study is undertaken. Also one may wish to determine if certain exploratory analysis (such as identifying interaction effects) are worth undertaking. 
I admit, this conflicts with the notion that one should have an identified model before hand (e.g. if you need to include an interaction effect for other effects to be properly identified it needs to be included regardless of its power). But, my  response would be such interaction effects are very rarely specified at the onset as necessary components in the model, and are typically added in a model building strategy. Such as that advocated by Raudenbush and Bryk starting with empty models and iteratively adding higher level variance components and interactions (as esimation problems frequently occur with complicated models, especially if a higher level variance component is zero).
For estimating power, it is unlikely any standard equation will capture the complexity of the situation. And so one frequently  turns to simulation. One then generates a fake set of data that approximates what the true data will look like, then generates an outcome distributed according to pre-specified effect sizes, then estimates the model and calculates the number of true and false negatives. Even if your client can't put a number on a specific effect size, you can graph the expected power of a test given a range of effect sizes. 
As Michael already stated, it will likely be a chore to specify the variability in the data (i.e. the variances and the co-variances of the independent variables). It will likely take custom coding to simulate the nested structure of the data while at least approximately constraining the variances and covariances of the independent variables. Quickly reviewing the multi-level modelling books on my shelf suggests most power analysis were considered in the context of multi-level experiments, so you may be foraying into new territory. 
I will refer to some other literature in hopes others have provided more directly actionable advice.


*

*Andrew Gelman and Jennfier Hill in their book, Data Analysis Using Regression and Multilevel/Hierarchical Models has a chapter on power analysis, and gives a brief example in R code. They also give a few references at the end of the chapter on the topic.

*Tom Snijders webpage has a windows program that calculates power for fixed effects in two-level models, PINT. He also appears to have publications of potential interest. I found this via the chapter in Joop Hox's multi-level book on power analysis.

*The G*Power 3 software tool doesn't appear to have any modules for multi-level analysis, but they do have a wealth of references to other literature on power analysis. 



As a note on post hoc power analysis, I agree it is typically considered inappropriate (see this response by Freya Harrison on our site). Although I admit I recently conducted one in a publication at the request of a reviewer. Although to be fair, in retrospect I wish I had conducted a-priori power analysis, as the question of power in and of itself is interesting, even though it was an observational, quasi-experimental study. 
A: Power analysis depends on the distribution of the test statistic under alternatives when the critical value has been specified to control the type one error.  All this assumes some form of random sampling. In an observational study where is the random sampling for a target population?  I first question whether hypothesis testing makes sense in this situation and it surprises me that no one has mentioned this.
If we were in a situation where the assumptions about sampling at random are reasonable you need some idea of variability.  Sample size problems almost always involve guess work on variability and so several scenarios should be looked at.  Even if the investigator has trouble determining effect size or clinically important difference the statistician canlook at how the results change over a resonable range of values.  But in my experience a clinically important difference that is worth detecting may be the easiest thing to determine. 
