2
$\begingroup$

Let's say I have a large collection of emails and I want to classify in two different class: Non-Spam, Spam. Assume this classification process is very costly, but I have access to an approximate classification procedure that outputs a value $\kappa$ in the range $[0;1]$. In general the method behaves as follows:

  • If the email $i$ is spammy $\rightarrow \kappa_i < 0.5$
  • If the email $i$ is non-spam $\rightarrow \kappa_i \ge 0.5$

Now what I want is some statistics assurance that the approximate metric is actually a good approximation for the original classification problem. What kind of method should I use?

Initially I was thinking to take a random sample from the initial dataset, consisting of 100 emails. The point is that the entire dataset is skewed towards spammy email:

0,1,0.508
1,0,0.580
2,1,0.684
3,0,0.717
4,1,0.575
5,0,0.427
6,1,0.791
7,0,0.694
8,1,0.752
9,1,0.538
10,1,0.174
11,1,0.021
12,1,0.795
13,1,0.412
14,0,0.668
15,1,0.560
16,1,0.714
17,0,0.425
18,1,0.367
19,0,0.538
20,1,0.104
21,1,0.555
22,0,0.512
23,0,0.722
24,0,0.300
25,0,0.803
26,1,0.517
27,1,0.649
28,1,0.125
29,0,0.220
30,1,0.711
31,1,0.253
32,1,0.229
33,1,0.516
34,1,0.460
35,1,0.046
36,1,0.453
37,1,0.343
38,0,0.453
39,1,0.549
40,1,0.654
41,1,0.516
42,0,0.725
43,0,0.735
44,1,0.606
45,0,0.795
46,1,0.444
47,1,0.141
48,1,0.724
49,0,0.635
50,1,0.614
51,1,0.506
52,1,0.572
53,1,0.591
54,1,0.169
55,0,0.521
56,1,0.909
57,1,0.489
58,0,0.694
59,1,0.121
60,1,0.648
61,1,0.415
62,1,0.446
63,1,0.509
64,0,0.501
65,0,0.187
66,1,0.537
67,1,0.196
68,0,0.565
69,1,0.373
70,1,0.153
71,1,0.490
72,1,0.350
73,1,0.317
74,1,0.772
75,1,0.726
76,1,0.374
77,1,0.483
78,1,0.348
79,1,0.743
80,0,0.897
81,1,0.463
82,1,0.015
83,1,0.798
84,1,0.539
85,1,0.456
86,1,0.507
87,1,0.028
88,1,0.397
89,1,0.565
90,1,0.245
91,1,0.501
92,1,0.748
93,1,0.475
94,0,0.764
95,1,0.679
96,1,0.347
97,1,0.646
98,1,0.206
99,1,0.406

Any pointer to page/tool with a practical example will be greatly appreciated since I am relatively new to this field.

Update

I have tried all the methods you have suggested but I am still confused. I firstly used R to cross tabulate the dataset, and then used Chi-Square test. I found out that there is a relationship between the two variables, namely the approximate function and real function.

I also tried to fit a logistic regression model (and a SVM linear classifier), registering an F1 score of 0.8. But this is somehow wrong, since the dataset is really skewed as I previously pointed out and the system always returns the label 1 independently from the input.

Therefore I tried to use an Anomaly Detection algorithm (one class SVM with RFB kernel) that seems to be more applicable here, but I don't think this is a good way to prove that the approximate method resembles the original one with high accuracy. What do you think should I use to prove my point?

In order to avoid the problem with skewed label distribution, I thought that may be it's better to create a new training dataset composed as follows:

  • 10 emails with $\kappa$ between $[0;0.10]$
  • 10 emails with $\kappa$ between $[0.10;0.20]$
  • ...

Do you think this approach can improve the situation?

$\endgroup$
  • $\begingroup$ Look at Anomaly detection technique. $\endgroup$ – user36710 Dec 30 '13 at 11:06
2
$\begingroup$

At least two options:

1) Use the categorization you list and then crosstabulate the original method with the approximate one

2) Keep $\epsilon_i$ as is and then do logistic regression with the true value (original classification) as the dependent variable and $\epsilon$ as the indepnedent variable.

(As an aside, $\epsilon$ is usually used for the error term. You might want to call it something else if you write this up - just to avoid confusion)

$\endgroup$
  • $\begingroup$ Thanks for the reply. I have updated the question accordingly and added a sample dataset that I am using. Anyway could you please provide provide me some links with an example or some documentation to better understand your suggestions? $\endgroup$ – nopper Dec 25 '13 at 9:49
  • $\begingroup$ I never did cross-tabulation, but I studied logistic regression in a ML course. I understand the fact that fitting a logistic regression model actually proves the dependence between the two variables, but I wanted to have some sort of measure that tells me how much the two variables are correlated. I need to justify the use of this approximate method in a paper. Are you suggesting me to just show a F1 score and coefficients ($w_0 \ldots w_n$)? $\endgroup$ – nopper Dec 25 '13 at 9:52
  • $\begingroup$ Crosstabulation is very simple. PROC FREQ in SAS or table in R. You don't need to test a hypothesis, just show that the table shows good agreement. I don't understand what you are asking in your last sentence; the logistic regression would have only one coefficient. $\endgroup$ – Peter Flom - Reinstate Monica Dec 25 '13 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.