What is “positive”?

Question

I'm deeply confused by some concepts.

We often hear the term true/false positive/negative.

While it is straightforward to tell if the result is true or false, I find it confusing to tell if it is positive or negative.

In hypothesis testing:

Let's say we have a new kind of medicine A. The 2-by-2 table looks like this:

                      | takes effect on patients |  no effects
-------------------------------------------------------------------
A is effective        |        #1                |        #2
-------------------------------------------------------------------
A is not effective    |        #3                |        #4
-------------------------------------------------------------------

I think everyone agrees that #1 and #4 are true and #2 and #3 are false.

But what about positive/negative?

I heard two different opinions:

1. It depends on your hypothesis. If your $H_0$ is "A is not effective", then #2 is negative. If your $H_0$ is "A is effective", then #2 is positive.

2. There is a "ground truth" positive, that is "A is effective". So #2 is negative and false, #3 is positive and false.

It seems opinion 2 is popular in medical science. However, we have other fields where the "ground truth" is not so clear.

Take binary classification as an example:

Binary classification

                 | predicted as 1 |  predicted as 0
-------------------------------------------------------------------
labeled 1        |        #1      |        #2
-------------------------------------------------------------------
labeled 0        |        #3      |        #4
-------------------------------------------------------------------

Here people tend to call #2 false negative. However, if we flip the label and the prediction, #2 would be called false positive.

So what do we mean when we say "positive"?

Answer 1

Hypothesis Testing - is drug effective or not?

In Hypothesis testing usually the objective is to establish whether a treatment is effective or not. This can mean a number of things such as, whether a drug cures a disease or not? Whether offering discounts improve conversion rates or not?

By convention Null Hypothesis $H_{0}$ assumes state of status-quo. It means we assume situation will not change, i.e. drug won't cure the disease and discount won't lead to increased conversion rate.

In drug testing, sometimes contingency table is used to analyze effectiveness of a drug. An example is below:

+--------------+-----------+----------+
| Table 1      | No-effect | Effect   |
+--------------+-----------+----------+
| No Drug      |  110      | 10       |
| Drug         |   20      | 1009     |
+--------------+-----------+----------+

$\chi^{2}$-test is common to analyze such tables and a high value of $\chi^{2}$ statistic means that you fail to assume status-quo and therefore believe that drug is effective.

Diagnosis - were you able to identify the disease or not?

True/False Positive/Negative will come into picture in example of diagnosis of a disease in population. Let's say 5% of population suffer from disease D and you have a method for diagnosing the disease. You test this method on 1000 people randomly selected from population (50 had the disease) and tabulate the results below:

+----------------+-----------------+-----------------+
|   Table 2      | Tested Negative | Tested Positive |
+----------------+-----------------+-----------------+
| Doesn't have D |  900            |   50            |
| Have D         |   15            |   35            |
+----------------+-----------------+-----------------+

Now lets observe the table carefully with the following definitions:
True Positive: People who have D and are tested Positive are called True Positives.
True Negative: People who don't have D and are tested Negative are called True Negatives.
False Negative: People who have D and are tested Negative are called False Negatives.
False Positive: People who don't have D and are tested Positive are called False Positives.

Here by convention we call positive as the one who has the disease or condition you are diagnosing.

Referring to Table 2,
True Positive = 35
True Negative = 900
False Negative = 15
False Positive = 50

Binary Classification

There are parallels to Diagnosis example above to what is done in Binary Classification where Tested Negative/Tested Positive become Predicted Class 0/Predicted Class 1.

As a general rule minority class is represented as the positive class (or 1). If your model identifies whether a visitor on the website is purchaser or not, purchaser is represented by 1 (or positive class) and non-purchaser as 0 (or negative) because purchasers are usually the minority class.