# "Examining the data" before analysis: I still don't know what I'm looking for

I am a recent graduate of a biostatistics program working as an analyst for health-related studies. We were taught to always "examine the data" before conducting an analysis by generating summary statistics to determine if the analysis you plan to run is appropriate for the data you have, has bias, could be done in a more valid way, etc. I still don't understand... What exactly am I looking for that can improve my analysis methods?

I understand that each method of analysis has a set of assumptions, and these assumptions must be met for the method to produce valid results. But it seems like there's SO much more to be learned from descriptive statistics tables than I can see. I can ensure that the data meets these assumptions, and still someone far more senior than me will say something like "Ah yes, but there isn't enough variability between these two categories of this variable and so the estimates for these categories won't be reliable."

Say, for example, I plan on running a Cox proportional hazards model. I have generated a summary table which shows the frequencies and means of my covariates for event=0 and event=1 of my outcome. This type of descriptive statistics table is a staple in pretty much every peer-reviewed observational study ever. What is the point of this table? What information can I glean from this table that will affect how I will run my Cox model?

I have tried searching online and in books for what feels like days for even one resource that can help me understand what to look for in the data that can help me improve my analysis methodology, but the only resources I could find were not for people actually doing the analysis (not technical enough).

I feel as if I'm going insane. What have I missed?!

• For a somewhat different view than the one you were taught, you might find it useful to read Gelman & Loken on "the garden of forking paths". stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf though you may need to read some of the references to get the full context. Commented Jul 5, 2018 at 7:32

It is useful to think of data analysis as being split into two parts, called exploratory analysis and confirmatory analysis. These two aspects of analysis complement each other, and they are both important (though the former does not always get enough credit), but you have to be very careful when you combine them. As a general rule, it is bad to use the same data for both, and it is bad to give yourself too much flexibility at the modelling stage. Here is some general advice.

Exploratory data analysis: In exploratory data analysis we don't yet have a specific research question (i.e., not specific enough to test yet) and we want to look at the data to figure out what kind of patterns are there that would indicate useful hypotheses to test (using different data). In this phase we rely primarily on graphs of the data. If we fit any formal models, these are used merely to obtain other useful metrics to graph (e.g., residuals from a particular kind of analysis), and not to undertake formal testing of hypotheses. Descriptive statistics are less helpful than graphs, and so it is not surprising that when you look at the former, you don't see anything useful. Instead you should try making some scatter-plots, correlation plots, density plots, violin plots, time-series plots, etc., based on how your data is structured. This should give you a better sense of what is going on in the data, and allow you to see patterns indicating research hypotheses to test (with different data).

For the particular case you mention, you have a summary table of frequencies. This is effectively a matrix of count values, and you can represent this graphically in a matrix plot, showing the size of each frequency (e.g., as shades of a colour, or as circles of different sizes, etc.). You can also make a similar plot showing relative frequencies, scaled by the row or column totals. This should allow you to visualise the extent to which the absolute/relative frequencies change in shape as you move across the rows or columns. Since pictures are more rapidly interpreted than tables of numbers, it is likely that this will capture your data more clearly than a table of frequency values.

Confirmatory data analysis: The other part of data analysis is confirmatory data analysis, where we already have hypotheses that are specific enough to test, and we know (within a reasonably narrow scope) how to test them without further inspection of the data. At this point we model the data and use the model to test our hypotheses. We still look at diagnostic plots to make sure our model fits the data, and we tinker with our model if it doesn't fit, but at this stage, we know the hypotheses we are testing and we know the model form we want to use, within a narrow amount of "wiggle room".

Danger! Danger! Now, as anyone in applied statistics can tell you, there is enormous danger in testing hypotheses using the same data that you looked at in the exploratory phase to formulate your hypotheses. If you undertake exploratory analysis, see patterns, formulate hypotheses based on these patterns, and then test those hypotheses using the same data, you are biasing your analysis enormously in favour of positive results in whatever tests you conduct (i.e., you are going to get lots of false positives). This kind of post hoc analysis leads to such invalid statistical practices as data dredging, p-hacking, etc., but it also sometimes manifests in more subtle ways. In fact, the same phenomenon occurs to a lesser degree whenever we have a wide discretion in our modelling at the confirmatory analysis stage. So any statistician reading your question is going to wince a little bit when you say that you have been taught to examine the data before your analysis, without any further qualification.

Be careful to split exploratory and confirmatory analysis: Aside from overt data dredging, there are many subtle and unintentional ways that you can bias your analysis if you have too much flexibility at the confirmatory analysis stage. The only way to learn about this is to read widely about it, and there are many excellent discussions on this topic, particularly by the excellent statistician Andrew Gelman. Gelman and Loken (2013) discuss the "garden of forking paths" that confronts researchers who have excessively wide scope to vary their models in the confirmatory stage, and Gelman and Guerts (2017) discuss the application of these issues to the replicability crisis in social science research. There is also a useful discussion of researcher degrees-of-freedom by Jeff Leeks here. These are good resources to get you starting in understanding this issue.

• Exploratory data analysis was named by John W. Tukey, although I doubt that either he or anyone else would deny that it has longer roots. But he and many others underlined how it could help you spot important problems you would miss otherwise (and some important possibilities too, such as the scope for a helpful transformation). It has become mandatory to stress how far such exploration may compromise inference, but I think we need a pendulum swing back to the scope for learning from the data what to do and what not to do. P-values are often dubious any way (dependence, sampling bias, etc.). Commented Jan 17, 2023 at 16:45
• Besides, whatever moralising methodologists say, most researchers are going to throw out what didn't work, publish what did (or seemed to) and not explain or discuss their choices in ways that are embarrassing. Commented Jan 17, 2023 at 16:48
• Here's a poser: suppose I skip exploratory analysis (i.e. don't peek at the data) because previous experience inclines me to certain choices, e.g. to work on logarithmic scale. Does that make my analysis in this project beyond reproach? The discussion seems to rest on an idea that projects start at defined points, but much research is one long irregular flow. Commented Jan 17, 2023 at 16:51

The way to avoid the "garden of forking paths" is to keep the exploration of the predictors separate from the exploration of the outcomes. Frank Harrell has a helpful illustration of how to do this in Chapter 8 of Regression Modeling Strategies, "Case Study in Data Reduction."

At first, you evaluate the outcomes only to evaluate their nature and the number of coefficients that you might reasonably be able to estimate without overfitting. In your Cox survival model example, that would be recognizing that the outcomes are times to events and that you are limited to something on the order of 1 coefficient per 15 events in your data set. Resist the temptation to do anything else with the outcomes at this point.

Then evaluate the predictors, independent of their associations with outcome, to see how well you can reduce the number of coefficients that you have to estimate. For one, predictors with narrow distributions are unlikely to be useful, as you note, and probably can be omitted without much danger.

In addition, multiple redundant predictors might usefully be combined into single or smaller numbers of predictors. In clinical studies, predictors are frequently correlated and are thus providing very similar information about the clinical situation that is associated with outcome. Categorical predictors might be clustered, based on their similarities in distributions. Principal component analysis (without peeking at the outcomes) could reduce multiple continuous predictors usefully into just a few principal components. Harrell illustrates additional approaches in that chapter, and applies the principles in examples of several types of models in other chapters.

The "descriptive statistics table" of clinical values has a different point. That's mainly to assure those reading the report that the characteristics of the individuals in the study are reasonable representative of a broader population. As such tables are "a staple in pretty much every peer-reviewed observational study ever," there are lots of opportunities for comparison. It's thus possible to get a sense of any unusual characteristics of the study sample that might limit generalization of your results.