I have been searching for several days for a method that fits this description, though cannot find one. I'm pretty sure it must exist.

The problem (short version):

I'd like to run something like a CART, though instead of making splits to improve information/purity, it would make splits to maximise the average of the absolute value of the correlation between two variables in the daughter nodes. I'm pretty sure this is something that people do in cluster analysis, though I'm not acquainted very well with those methods.

Importantly, I'd need this method to be able to output a leaf/cluster membership at the observation level.

Any pointers would be appreciated.

The problem (long version):

We've run a randomised control trial with outcomes $Y$ and treatment $T$ (for the moment, you can consider these as being continuous). We also have surveyed each of the people on $K$ characteristics which are in matrix $X$.

Ideally, I'd like to know everyone's individual treatment effect, which of course is impossible. If we had that, we could prioritise treatment to those who are likely to benefit most. Heterogeneous treatment effect models aren't great, and the (best of class) methods I've been trying do not cross-validate well.

It may be a good second best solution to prioritise clusters of people who have the largest (expected) treatment effects. One way of clustering observations that are likely to have similar treatment effects would be to cluster them based on the correlation between $T$ and $Y$. Another method would be to cluster based on measured treatment effect in the daughter nodes, though then you'd need to specify some cost function to differentiate between cases of higher treatment effects with wide confidence intervals and lower TEs with tighter precision.

This may be misguided, and I'm very open to suggestions.


1 Answer 1


One solution is to use principal component regression. The principal component decomposition of X should give you the directions along which the data vary the most. The regression step is saying what proportion of the variance in Y is being explained ONLY along these directions. There are several packages which do this sort of regression.

Could you please be more specific about T...is it a number? is it a vector? is it a part of X? Another detail which will be very useful in such problems is the dimensions of X and Y

  • $\begingroup$ Hi Sid - thanks for your response. T here is a treatment vector. It could be binary, or a continuous dose. Likewise, Y could be a binary or continuous outcome variable. X has K columns (it does not include T) and N rows. Y is N*1, as is T. I'm not precisely sure how doing a PCA regression would help me. Perhaps I didn't phrase the question well: Ideally, I'd like to rank-order my data based on who has the greatest treatment effect (an unobserved variable). Ranking on sub-group correlations between treatment and outcome seems a crude (but possible) second-best. $\endgroup$
    – Jim
    Commented Jul 24, 2014 at 4:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.