How to display a large number of model results from groups of data?

Question

Suppose I have $n$ data sets where each variable has the same variable types, but possibly a different number of data points. For instance, age, sex, and height, for $n$ different counties. After performing analysis (linear, logistic, Poisson, etc..) on each data set (country), how do I display information for all the results? Do I show a histogram of the regression coefficients, do I give the number of significant results? What do statistical packages do? (Does R have a library for this?) I'm currently not sure how to search for an r package that does this.

The purpose of the information display is to summarize the results of all the different models, where I asume that a model is a combination of method and data set. What I don't understand is how to compare or summarize, let's say $n$ p-values, where each p-value is computed with a different data set. Is there an intelligent way summarize all the different p-values, regression coefficients, AIC, BIC, ...?

An example taken from the above would be: After $n$ models have been computed (1 for each country), we gather 2 $\times$ $n$ $\times$ $m$ p-values and regression coefficients where $m$ is the number of variables used in each of the $n$ models. Is there an informative method for describing or summarizing my all my results. The brut force method would, in my option, be to display each model summary (p-values, coefficients,etc), but this could be a list that is too large. Getting an understanding of the state of all countries after analysis, might be difficult with the brut force method. How would someone go about getting a simple view of the state of all countries?

Answer 1

I would display the coefficients from the regression models and their confidence intervals graphically in a plot modelled after the forest plots traditionally used in meta-analysis. This Wikipedia article has some more information on forest plots. These are also sometimes called effect plots. You could even consider synthesising your results using meta-analysis, software is widely available for this.

Answer 2

You are referring to a task known as model selection (check here for more info: https://en.wikipedia.org/wiki/Model_selection).

Typically, once a model has been fit on your data the p-value can be used as a goodness-of-fit metric. Subsequently, you have to employ some decision rule to select the best fitting model.

As discussed in 1 and [2], p-values are uniformly-distributed random variables. Getting a low p-value may mean that probably your null hypothesis does not hold (with your null hypothesis being that the actual data were generated from your constructed model). However this is not absolute. In addition, getting a high p-value does not automatically mean that your hypothesis should be accepted. Instead, it is best to rely on resampling techniques to obtain a distribution of p-values and compare that to a uniform distribution. This will give you a better idea for model selection.

Having obtained p-value distirbutions for all candidate models, you can then plot their ecdfs on a single plot, such as below:

Here I compare the p-value distribution generated by 6 methods against the uniform distribution cdf (referred to as 'training').

1 Murdoch, D, Tsai, Y, and Adcock, J (2008). P-Values are Random Variables. The American Statistician, 62, 242-245.

[2] Why are p-values uniformly distributed under the null hypothesis?