5
$\begingroup$

Suppose I have $n$ data sets where each variable has the same variable types, but possibly a different number of data points. For instance, age, sex, and height, for $n$ different counties. After performing analysis (linear, logistic, Poisson, etc..) on each data set (country), how do I display information for all the results? Do I show a histogram of the regression coefficients, do I give the number of significant results? What do statistical packages do? (Does R have a library for this?) I'm currently not sure how to search for an r package that does this.

The purpose of the information display is to summarize the results of all the different models, where I asume that a model is a combination of method and data set. What I don't understand is how to compare or summarize, let's say $n$ p-values, where each p-value is computed with a different data set. Is there an intelligent way summarize all the different p-values, regression coefficients, AIC, BIC, ...?

An example taken from the above would be: After $n$ models have been computed (1 for each country), we gather 2 $\times$ $n$ $\times$ $m$ p-values and regression coefficients where $m$ is the number of variables used in each of the $n$ models. Is there an informative method for describing or summarizing my all my results. The brut force method would, in my option, be to display each model summary (p-values, coefficients,etc), but this could be a list that is too large. Getting an understanding of the state of all countries after analysis, might be difficult with the brut force method. How would someone go about getting a simple view of the state of all countries?

$\endgroup$
2
  • 4
    $\begingroup$ Welcome to our site! To help us focus our replies, could you please tell us the purpose of the information display? It would also help to indicate you have spent at least a little time researching this problem: the capabilities of statistical packages are well-advertised and easy to discover, so your last question is not really appropriate here (and, given R's extensive capabilities, is too broad). $\endgroup$
    – whuber
    Jun 15, 2012 at 12:46
  • $\begingroup$ I feel like this question would be better received on Stackoverflow. Having said that, check out the broom package in R. It gathers regression output from grouped data and puts it in a easy to interpret format. It should let you for example, see all the AIC values for each group of data so that you can put in a bar chart if you like. $\endgroup$
    – Alex
    Jul 27, 2016 at 6:39

2 Answers 2

1
$\begingroup$

I would display the coefficients from the regression models and their confidence intervals graphically in a plot modelled after the forest plots traditionally used in meta-analysis. This Wikipedia article has some more information on forest plots. These are also sometimes called effect plots. You could even consider synthesising your results using meta-analysis, software is widely available for this.

$\endgroup$
0
$\begingroup$

You are referring to a task known as model selection (check here for more info: https://en.wikipedia.org/wiki/Model_selection).

Typically, once a model has been fit on your data the p-value can be used as a goodness-of-fit metric. Subsequently, you have to employ some decision rule to select the best fitting model.

As discussed in 1 and [2], p-values are uniformly-distributed random variables. Getting a low p-value may mean that probably your null hypothesis does not hold (with your null hypothesis being that the actual data were generated from your constructed model). However this is not absolute. In addition, getting a high p-value does not automatically mean that your hypothesis should be accepted. Instead, it is best to rely on resampling techniques to obtain a distribution of p-values and compare that to a uniform distribution. This will give you a better idea for model selection.

Having obtained p-value distirbutions for all candidate models, you can then plot their ecdfs on a single plot, such as below:

enter image description here

Here I compare the p-value distribution generated by 6 methods against the uniform distribution cdf (referred to as 'training').

1 Murdoch, D, Tsai, Y, and Adcock, J (2008). P-Values are Random Variables. The American Statistician, 62, 242-245.

[2] Why are p-values uniformly distributed under the null hypothesis?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.