5
$\begingroup$

I want to know if it is possible to construct a problem with following properties:
$M_1$ is $n \times p$ matrix of $n$ observations from Class A
$M_2$ is $n \times p$ matrix of $n$ observations from Class B (I am keeping $n$ same for simplicity but it is not required)
Below function returns a vector of zeros:

D=function(X,Y)
{
    m1=colMeans(X)
    m2=colMeans(Y)
    s1=apply(X,2,sd)
    s2=apply(Y,2,sd)
    return(abs((m1-m2)/(s1+s2)))
}

when called with $X=M_1$ and $Y=M_2$.
However, when logistic regression is run on the data set, it comes up with a classifier that can accurately classify between the two classes. I think its not possible, but wanted to ask. The motivation behind the question is that I was working on a real problem where $D$ values were pretty small (less than 0.1 on average) but the LR classifier was able to score AUC of 0.66 on the training data set.

EDIT: I am posting a follow up question after the answer. Based on the answer, I feel its reasonable to hypothesize that performance of LR should be positively correlated with $D$ - indeed I use $D$ values to decide which features I want to input to LR (feature selection). But recently I came across a problem where this hypothesis seemed to be violated. I cannot post the data, but here are the two cases:

In one case (case 1) my feature vector had $D$ scores given by:

$D = (0.1290, 0.07961, 0.06397, 0.07427, 0.04373, 0.06814)$

as can be seen the values are pretty small but when I ran LR I got AUC of 0.66 and here is the output of LR:

Call:
glm(formula = class ~ ., family = "binomial", data = df)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.6911  -0.3394  -0.2880  -0.2376   2.9908  

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -3.671e+00  3.412e-01 -10.758  < 2e-16 ***
V98          1.671e-04  2.751e-05   6.074 1.25e-09 ***
V99         -5.708e-04  1.296e-04  -4.403 1.07e-05 ***
V100         4.075e-04  8.066e-05   5.051 4.39e-07 ***
V101        -1.084e-03  2.441e-04  -4.442 8.92e-06 ***
V102         6.915e-03  1.510e-03   4.580 4.64e-06 ***
V103        -2.220e-02  5.890e-03  -3.770 0.000163 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1929.8  on 5039  degrees of freedom
Residual deviance: 1852.2  on 5033  degrees of freedom
AIC: 1866.2

Number of Fisher Scoring iterations: 7

Now compare above to case 2 where I have feature vectors with following values of $D$:

$D=(0.0350, 0.1545, 0.0942, 0.0182, 0.2346, 0.3499)$

on average above $D$ values are almost 2x higher than in case 1 but the LR classifier fell flat on its face. Here is its output:

Call:
glm(formula = class ~ ., family = "binomial", data = df)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.7968  -0.3597  -0.2762  -0.1380   2.8111  

Coefficients:
                 Estimate Std. Error z value Pr(>|z|)   
(Intercept)       7.56148    9.79303   0.772  0.44004   
energy           -0.09437    0.08758  -1.078  0.28123   
entropy           1.19914    2.67011   0.449  0.65336   
correlation     173.27758   75.08438   2.308  0.02101 * 
sd.energy         0.22770    0.16946   1.344  0.17906   
sd.entropy      -15.64633    9.30878  -1.681  0.09280 . 
sd.correlation -287.35575  104.54289  -2.749  0.00598 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 192.98  on 503  degrees of freedom
Residual deviance: 173.42  on 497  degrees of freedom
AIC: 187.42

Number of Fisher Scoring iterations: 8

> auc
[1] 0.4585417

So my dilemma is how can above be explained? How come features with low $D$ are able to give better classification than features with higher $D$? and thus if $D$ is not an indicator of a good feature then what metric could be used to determine what features to feed into a LR?

$\endgroup$
3
$\begingroup$

Your intuition is correct: such an example is impossible.

To see why not, consider both $M_1$ and $M_2$ as collections of $p$-vectors. Because the predicted value of any vector in a logistic regression is a linear function, perfect prediction means there exists a codimension-$1$ affine hyperspace that separates all the points in $M_1$ from those in $M_2$. That implies their centroids cannot coincide, QED.

Figure

In this figure $p=2$ and the groups have sizes $30$ (red circles) and $10$ (blue triangles). Their centroids are shown as corresponding filled graphics. Perfect separation occurs, as shown by the gray dotted line. Since the centroids must lie on opposite sides of this line, they cannot coincide.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.