Dimensionality reduction technique similar to LDA when class labels are probabilistic

Given discrete class labels, say True and False, LDA (linear discriminant analysis) can be used to perform discriminant dimensionality reduction and attempt to find a subspace that best separates the two classes.

What if we are given, instead of the labels themselves, confidences of the labels? i.e. [0.2, 0.7, 0.3, 0.9, 0.1] instead of [F, T, F, T, F]. Is there a dimensionality reduction technique that works with this kind of data?

• Interesting question, +1. I guess you could try using a weighted LDA, weighing each sample point by the label confidence (e.g. $w = |0.5-p|/0.5$). – amoeba Jan 6 '15 at 1:01
• Yeah that was kind of like what I was thinking, I had an idea for a modified LDA but I'd be very surprised to find there's not a technique for this already! – Andrew Latham Jan 6 '15 at 1:10
• LDA can be seen as a particular case of canonical correlation analysis which is for numeric data (with single dependent variable, like yours, it is even plain linear regression!). So, why not simply logit- or probit- transform your "confidences" and do the regression? – ttnphns Jan 6 '15 at 8:47
• @ttnphns: This is an interesting idea! But note that the OP does not have a single DV, he has as a single IV (class labels) and probably multivariate data points. So LDA is equivalent to CCA between this multivariate data and two-dimensional (or one-dimensional? -- not sure) dummy-coded class labels. Your suggestion to use "T class probability" $p$ (perhaps logit-transformed) instead of dummy class labels is very interesting. CCA will indeed find a 1-dimensional subspace of DVs that is best correlated with the class probability. – amoeba Jan 6 '15 at 11:07
• @amoeba, thank you. The class label nominal variable in LDA is usually considered as the "DV" and the quantitative variables-to-reduce as "IVs". But for CCA, the two sets are symmetric, you may call any of the two as "DVs", as you wish. – ttnphns Jan 6 '15 at 11:20

I liked @ttnphns's suggestion (that he made in the comments above) so much, that could not resist from trying it out.

As @ttnphns said, LDA is equivalent to canonical correlation analysis (CCA) between your multivariate data and the set of dummy variables coding class labels. In case of only two groups, there is only one non-redundant dummy variable that takes e.g. value of $0$ for one class and value of $1$ for the second class. Running CCA between this dummy variable and the multivariate data will yield the canonical axis (maximizing correlation with the dummy variable) identical to the LDA axis (maximizing between-class/within-class variance ratio).

If we now replace this dummy variable by your class probability variable that takes values from $[0, 1]$, we can still run CCA analysis and use the first canonical axis. It will not be equal to any LDA axis anymore, because now that there are no clear groups LDA does not make sense.

Here is a figure illustrating it:

Blue dots are one class, red dots are another class, size of the dots represents the probabilistic label (smallest dots for $p\approx 0.5$, largest dots for $p\approx 0$ and $p\approx 1$). Solid line is the LDA axis. Dashed line is the CCA axis, computed using probabilistic labels. It goes diagonally because if you look very carefully you will notice that the probabilistic labels were assigned such that they increase along the diagonal.

Note that in this very simple case of only two classes and consequently only a one-dimensional label variable, CCA and LDA are equivalent to good old multiple regression (regressing label variable $l\in \mathbb R$ on $\mathbf x \in \mathbb R^2$ results in a two-dimensional $\beta$ coefficient that defines an axis in $\mathbb R^2$). I demonstrate this equivalence numerically in my code below.

MATLAB code

%// generate data
n = 100;
x = randn([n*2 2]);
x(1:n, :) = bsxfun(@times, x(1:n, :), [1 4]);
x(1:n, :) = bsxfun(@plus,  x(1:n, :), [1 1]);
x(n+1:end, :) = bsxfun(@times, x(n+1:end, :), [1 4]);
x(n+1:end, :) = bsxfun(@plus,  x(n+1:end, :), [10 10]);
x = bsxfun(@plus, x, -mean(x));

labels = [ones(n,1); 2*ones(n,1)];      %// group labels (ones and twos)

prob_labels = (labels-mean(labels) * x * [1 1]'); %// probabilistic labels (from 0 to 1)
prob_labels = prob_labels / max(abs(prob_labels)) / 2 + 0.5;

%// LDA via within- and between-class covariance matrices (my own function)
axisLDA = mylda(x, labels);

%// CCA with probabilistic labels
[axisCCAprob, ~] = canoncorr(x, prob_labels);
axisCCAprob = axisCCAprob / norm(axisCCAprob);

%// plot
axesScale = 15;
dotScale = 200;
figure
hold on
axis([-20 20 -20 20])
axis square
scatter(x(1:n,1), x(1:n,2), abs(prob_labels(1:n)-0.5)*dotScale, 'b')
scatter(x(n+1:end,1), x(n+1:end,2), abs(prob_labels(n+1:end,:)-0.5)*dotScale, 'r')
plot(axisLDA(1)*axesScale*[-1 1], axisLDA(2)*axesScale*[-1 1], 'k')
plot(axisCCAprob(1)*axesScale*[-1 1], axisCCAprob(2)*axesScale*[-1 1], 'k--')


We can now check that different approaches mentioned above give identical results:

%// check that different approaches give identical results
display(['LDA, take 1: ' num2str(axisLDA')])

[axisCCA, ~] = canoncorr(x, labels); %// LDA via canonical correlation with dummies
axisCCA = axisCCA / norm(axisCCA);
display(['LDA, take 2: ' num2str(axisCCA')])

axisRegression = regress(labels, x); %// LDA via regression
axisRegression = axisRegression / norm(axisRegression);
display(['LDA, take 3: ' num2str(axisRegression')])

display(['CCA with probabilistic labels, take 1: ' num2str(axisCCAprob')])

axisRegressionProb = regress(prob_labels, x); %// CCA via regression
axisRegressionProb = axisRegressionProb / norm(axisRegressionProb);
display(['CCA with probabilistic labels, take 2: ' num2str(axisRegressionProb')])


Running this code produces the following output:

LDA, take 1: 0.99735    0.072685
LDA, take 2: 0.99735    0.072685
LDA, take 3: 0.99735    0.072685
CCA with probabilistic labels, take 1: 0.68477     0.72876
CCA with probabilistic labels, take 2: -0.68477    -0.72876

• Nicely done. I'll come back later to check it in SPSS, for myself. – ttnphns Jan 6 '15 at 16:02
• Nice, thanks for running with this problem, I'll take a look in the next few days! – Andrew Latham Jan 10 '15 at 6:00
• Looks great, do you think there's a technique for finding more than just the one dimension? I know that's beyond the scope of the question but I'm curious. – Andrew Latham Jan 12 '15 at 2:54
• @Andrew, remember that standard LDA with only two classes also finds only one discriminant direction/axis (in general with $k$ classes LDA results in $k-1$ directions). The same remains true here for probabilistic labels, but this is a general LDA feature... – amoeba Jan 12 '15 at 10:11
• I know, that is what I meant about it being beyond the scope of the question since that would improve on LDA, but I wonder if it's possible? – Andrew Latham Jan 12 '15 at 15:11

Unless I completely misunderstood what you're trying to achieve, I would suggest to take a look at some probabilistic methods of dimensionality reduction, such as probabilistic PCA (PPCA) and probabilistic non-linear PCA with Gaussian process latent variable models. Finally, this excellent tutorial on dimensionality reduction by Christopher Burges (Microsoft Research) is rather comprehensive and contains description of PPCA. I hope that this is relevant and will be useful.

UPDATE. Upon your mentioning, I became curious about weighted LDA approach and decided to read a bit on it. That resulted in finding the following papers on this topic, which I thought might be of your interest (based on your comment): first, second, third, fourth and fifth.

• Can you comment on how PPCA is related to the problem of the OP? I don't see it. OP wants to find subspace that is best discriminating between classes. PCA/PPCA do not take class labels into account at all, neither usual nor probabilistic. That's why the question was about LDA and not about PCA (and rightly so). -1 until you [edit to] explain. Your update about weighted LDA is interesting, but could you maybe provide at least a brief description of weighted LDA based on these papers and also give proper references (as links unfortunately may rot)? – amoeba Jan 6 '15 at 11:11
• @amoeba: Thank you for your comment. Discriminating between classes is exactly how I understood the OP's task. Will do my best to comment on and/or update my answer later. It could take some time, though, as with this material I'm pushing myself somewhat beyond my current level of statistical knowledge. And that is a good thing - exactly what I need to learn something new. – Aleksandr Blekh Jan 6 '15 at 23:39
• Yes, curious about this, I looked into PPCA as well since the name made it sound like it might serve my purposes but I didn't get that from it. – Andrew Latham Jan 10 '15 at 6:00
• @AndrewLatham: Will comment or update at my earliest convenience, hopefully, this weekend. – Aleksandr Blekh Jan 10 '15 at 6:13