2
$\begingroup$

I have one variable that attains a certain value (between 0 and 1), about 1000 observations and each belongs to one (and only one) of fifteen groups. Now I would like to reduce the number of groups to two or three by combining them. It is important that all the data within a given group in the former division will stay together after the pooling.

The problem is, that the whole dataset has too large variance, so I would prefer several groups that are more focused, less varied.

Example: Say you have hundreds of data on incomes of people from different EU countries. From this, you estimate average salary for each country (just an example). You would like to now create a group of "rich countries" and "poor countries" — but where do you draw the division line? Or two division lines in case of three groups (adding, say, "average countries").

Since yet again I don't know the proper statistical approach, I at least give you what my idea was:

For the reduction to two, instead of trying all possible divisions, I had the idea of sorting the groups by their sample means (from $g_1$ to $g_{15}$ where means satisfy $m_1 \leq m_2 \leq \dots \leq m_{15}$) and creating pool combinations — first combination would be {[$g_1$]; [$g_2, ..., g_{15}$]}, second would be {[$g_1, g_2$]; [$g_3, ..., g_{15}]$} and so on -- this notation means that in the first case the first pool are only values from $g_1$ and the second pool are all other values. The second case the first pool are values from $g_1$ and $g_2$, the rest is in the other pool.

I would then compute the p-value for sample mean equivalence test of these pool combinations. The lowest p-value of these 14 combination tests would mark the "most different" division into pools.

The same could be applied to the three groups, though there would be a higher number of pool combinations and the "diversity test" would have to be constructed quite differently.

Feel free to ignore my scribbling... since I do not know statistics very well, I at least tried an approach that seemed most logical.

Thanks again!

Improve this question
$\endgroup$
7
  • $\begingroup$ So you have 1000 observations, each of which is now categorised into into one of 15 groups, which you want to shrink to 2-3. What is your objective in shrinking to a predetermined number of groups (what will you do with the results/conclude from the grouping), and how many variables have you measured that you can group on? $\endgroup$
    – Michelle
    Commented Feb 1, 2012 at 5:03
  • $\begingroup$ It is primarily for simplicity reasons, I have trouble orienting in 15 groups, would rather have less of them so that I can observe other variables (this variable is only one dimension of a dataset). The conclusion would be like in the example -- different pools in terms of "performance" -- in this case the size of the variable in question. $\endgroup$
    – Ondrej
    Commented Feb 1, 2012 at 7:54
  • $\begingroup$ It would help to know what your overall intent is. If you wish to do a multivariate analysis later, this could be time wasted. $\endgroup$
    – Michelle
    Commented Feb 1, 2012 at 7:57
  • $\begingroup$ OK, so there are basically two intentions: First is pooling of high and low performance countries/regions just out of curiosity. The second is that I will perform a regression later and would like to know if these "high performing" countries are statistically different in some other aspect than the low performing. I know I could just use the this variable in a regression instead of this pooling, but I feel like this approach is more "visible" and comprehensible in the output. I guess there will be a problem accuracy as this way some information will be lost. Hope that makes sense. $\endgroup$
    – Ondrej
    Commented Feb 1, 2012 at 10:06
  • $\begingroup$ To sum up, instead of running $y = x_1 + x_2 + performance$ ($performance$ being the exact value of the observed variable), I would like to run $y = x_1 + x_2 + highperforming$, where $highperforming$ would be a dummy. I guess I should be working on something more meaningful :-) $\endgroup$
    – Ondrej
    Commented Feb 1, 2012 at 10:06

1 Answer 1

Reset to default
1
$\begingroup$

I would try to do a hierarchical cluster analysis on your outcome variable(s) using your original data (rather than the country averages). Take a look at the dendrogram of the results and see if any reasonable clusters of people emerge. Look and see if they have countries in common.

Another approach, if you have enough data, would be to calculate the mean for your outcome, its variance and maybe some other higher moments (to get at issues of distribution) within each country, and then do a cluster analysis on those moments to group the countries.

In the end, I think you will have to use domain knowledge to make sure that your grouping looks reasonable.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I agree with @Dimitriy, especially about the necessity of domain-specific knowledge as opposed to some statistical criterion for clustering. The clustering is probably going to be somewhat arbitrary. There are few rules to go by. The very choice of hierarchical cluster analysis vs. some other type, and the options one uses within that method, are going to be somewhat arbitrary. And to tell you the truth, I wasn't entirely convinced by your rationale for avoiding regression. If you explore you'll find a number of ways to make regression results visually interesting. $\endgroup$
    – rolando2
    Commented Mar 3, 2012 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.