TL;DR: Is there an existing k-means clustering algorithm that can have different weights for the (minimized) in-group distance measure and the (maximized) between-group measure? Or, better yet, can one use slightly different subsets of the data for these two distance measures and allow the algorithm to learn the weights rather than hard-coding them, but leaving out some variables in the between-group measures?

The problem: Nearly three years after the 2016 U.S. presidential election, there is still no wide consensus on what combination of demographic groups best characterizes the coalitions of more-Republican vs. more-Democratic voters in that contest. It's common knowledge that exit polls have all sorts of problems, and we're further misdirected by dated, anecdotal cultural tropes.

The goal: Develop an empirical, probabilistic portrait of the coalitions that turned out to support the current president or his opponent without relying on dated, predetermined demographic blocs, like "suburban white women" or "educated urbanites." My aim is NOT to predict the 2020 election, but rather to develop an statistical portrait of the different groups that made up the base supporters of each party last time around, and then look at how they've grown or contracted since. Naturally we don't know who within a county actually voted, but differences in the overall makeup of the eligible population ought to still be illuminating.

The data: For each of the ~3,140 counties, we have both the 2016 election results--and 2012, if the delta turns out to be more interesting--and a reasonably current portrait of the voting-eligible population. I don't like counties any better than the rest of us, but they are the most granular unit for which election data and Census data can be merged while preserving a reasonable sample size of different slices of the population. Of course, as we all know, there are significantly fewer Democratic-leaning counties, which have drastically higher populations.

What I've tried (in R): I started by running a bunch of regression models. Next, I tried classifying the counties into between 3 and 9 categories based on percentage support for D or R, and then running a variety of classification models as well as separate PCAs for the different groups. It was a noble effort and vaguely illuminating, but I'm most interested in discrete coalitions within the umbrella populations of people who significantly support one party over the other. So I'm pretty sure clustering is the way to go.

Enter k-means: First, I tried running separate clusters for D and R counties, which was far too crude since it didn't account for the difference between a 0.1 percentage-point victory and a 25-pp victory for either camp.

Next, I tried just including the election returns alongside the demographic percentages and weighting the election returns highly. Besides the fact that it seems like a terrible idea to combine dependent and independent variables, this doesn't work because the algo was including political support in maximizing the between-group distances.

The goal is not to divide the population into different degrees of partisanship. The ideal output would be a certain number of demographic blocs that have high in-group similarity in both political leaning and demographics, but are merely as distinct as possible only at the demographic level between groups. It's absolutely fine if, say, a 9-cluster model produces three groups that have nearly identical Democratic leaning but maximally different combinations of demographics (say, age, race+ethnicity, education, and so forth), and 6 groups that likewise are all equally Republican but also well-clustered demographically.

So my main question is whether a weighted k-means can have different weights for the distance measure inwardly and outwardly, in which case the outward weight of political outcome would be 0.

Even better, if it's possible to use a different subset of variables for homogeneity and heterogeneity, I would be delighted if I could find a model that learned the weights, such that I didn't have to run thousands of different combinations of three or four demographic measures with different manual weights in order to figure out the ideal way of segmenting the population. For example, we know what percentage of people are married, but we don't know, with any precision, how proportionally more or less important this is than their age, or whether it's worth factoring in at all. It's also quite possible that education is not nearly as predictive as is widely assumed unless combined with some other variable. It'd prefer not to rely on getting lucky!

  • $\begingroup$ Given that your data is pretty crude - and you only look at a single election in retrospective -, I doubt you will get any better results than the current bloc system. Which to my understanding is applied at a finer granularity than counties... K-means is not magic. It minimizes the summit squared differences, which IMHO may be the wrong measure to optimize for your task. It's not a magic function that solves all your problems. $\endgroup$ Sep 7 '19 at 7:25
  • 1
    $\begingroup$ Interesting problem statement. We can definitely choose different weights for inter cluster and intra cluster metrics. Do share the results $\endgroup$ Sep 7 '19 at 8:32
  • $\begingroup$ Is there a name for that model, @AnantGupta? Or a phrase I can Google to get me there? Thx! $\endgroup$ Sep 7 '19 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.