Questions tagged [bayes-optimal-classifier]
The bayes-optimal-classifier tag has no usage guidance.
33 questions
1
vote
1
answer
46
views
Is this the proof the the Bayes classifier is optimal?
In"Introduction to statistical learning" they say that the proof that the Bayes classifier is optimal is outside the scope of the book:
In "Elements of statistical learning" the ...
- 164
3
votes
1
answer
65
views
Modeling the probability distributions for the Bayes classifier
According to the Wikipedia, the Bayes classifier assumes knowledge of the distributions of $X | Y$, where $X$ and $Y$ are the random variables of the features and the classes, respectively. Let's ...
- 371
1
vote
0
answers
63
views
Is Fisher's discriminant analysis equivalent to the Bayes optimal LDA when the no. of classes is greater than two and covariances are all equal?
P.S. While I gave a brief background to make the question complete, informed readers can move to the questions 1 and 2 towards the end of this post, right after 'what are not clear to me are:'.
Fisher'...
- 761
5
votes
1
answer
1k
views
Should a language model like GPT-3 be directly used to perform classification?
The OpenAI API enables classification by sampling from GPT-3 given a prompt. Is estimating posterior probabilities a more statistically sound approach? Below is a specification of what "...
- 1,625
1
vote
0
answers
69
views
How do I minimize the cost from errors?
Bayes' Optimal Classifier is known to achieve the minimum error rate for a dataset $x_1, \ldots, x_n, x_i \in \mathbb{R}^d$. Suppose that each error had a cost associated with it. For example, in a ...
- 255
0
votes
1
answer
713
views
Use Naive Bayes to label unlabeled data
I have an Excel file that includes all product information (web scraped from Zalando) of 10k dresses. So for each dress/line I have multiple features available (brand, color, neckline, length...)
I ...
2
votes
0
answers
226
views
Derive the criterion for minimizing the expected loss when there is a general loss matrix and general prior probabilities for the classes
In the book "Pattern Recognition and Machine Learning" I am trying to do exercise 1.23 (p.63):
Derive the criterion for minimizing the expected loss when there is a general
loss matrix and ...
- 309
2
votes
0
answers
891
views
What is the relation between Linear Classifier and Linear Decission Boundary (or Non Linear Decision Boundary)?
As we know (Wikipedia Definition): Linear Classifier makes a classification decision based on the linear combination of the feature vectors.
Mathematically : $y = f(\sum w_i x_i)$
So , $f$ is our ...
1
vote
1
answer
713
views
How would you find a p threshold for a binary classification prediction? [duplicate]
Lets say that there's a binary classification problem where $X$ ∈ $R_p$ and $Y ∈ \{0,1\} $ and $Pr(Y = 1 | X = x) = p$ for $p$ in $[0,1]$. There is a loss function $L_{falseneg} > 0$ for false ...
- 11
3
votes
1
answer
122
views
This decision is the best we can do if we have no prior information about the probabilities of the two classes?
I am currently studying the textbook Learning with kernels: support vector machines, regularization, optimization and beyond by Schölkopf and Smola. Chapter 1.2 A Simple Pattern Recognition Algorithm ...
- 2,204
2
votes
1
answer
238
views
How is this a "Bayes classifier"?
I am currently studying the textbook Learning with kernels: support vector machines, regularization, optimization and beyond by Schölkopf and Smola. Chapter 1.2 A Simple Pattern Recognition Algorithm ...
- 2,204
0
votes
1
answer
74
views
How $\dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} = \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x})}}$?
Related: Empirical Risk Minimization: Rewriting the expected loss using Bayes' rule and the definition of expectation
I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai,...
- 2,204
6
votes
1
answer
495
views
Empirical Risk Minimization: Rewriting the expected loss using Bayes' rule and the definition of expectation
I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Chapter 2.2 Instance-Based Noninductive Transfer Learning says the following:
As mentioned ...
- 2,204
2
votes
2
answers
1k
views
Improve Adaboost that using weighted logistic regression instead of decision trees
I implemented Adaboost that using weighted logistic regression instead of decision trees and I managed to get to 0.5% error, I'm trying to improve it for days with no success and I know it possible to ...
- 21
2
votes
1
answer
161
views
Use different Naive Bayes classifiers to target different data
I am practicing using the Naive Bayes classifier to predict whether people get a stroke or not, but, I am confused with two classifiers. One is categorical Naive Bayes, another is Gaussian Naive Bayes....
- 207
3
votes
0
answers
273
views
How to proof that the bayes optimal classifier is optimal for a continuous domain
Exercise 3.7 from the book »Understanding Machine Learning: From Theory to Algorithms«, Shalev-Shwartz and Ben-David, states the following:
The Bayes optimal predictor: Show that for every ...
- 263
1
vote
1
answer
256
views
Bayes Optimal Classifier for multinomial classification
I understand the meaning and how to deduce a Bayes optimal classifier in binary classification, but I am not sure how to derive this in the context of multinomial classification. Do we use the naive ...
- 11
3
votes
1
answer
1k
views
LDA and Fisher LDA - are their weight vectors always equivalent?
Linear Discriminant Analysis (LDA) and Fisher Linear Discriminant Analysis (FLDA) both project high-dimensional observations to univariate classification scores using different rationals and ...
- 1,845
0
votes
0
answers
757
views
How to find the optimal classifier for a given loss function?
For a binary classification problem (labels being 0 and 1) and a classifier $g$ we consider the loss function $L(g)=P[Y\neq g(X)]$. It is known that the optimal classifier $g^*$ that minimizes $L$ is $...
- 281
1
vote
1
answer
47
views
Penalization term for unfairness
I am reading [1], where the researchers do a logistic regression, but add to the loss function the following penalization term for fairness
$ R^{AVD}_{FP}(\theta; S) = \left\lvert \dfrac{\sum\limits_{...
- 2,433
3
votes
0
answers
71
views
Question about using Bayesian rule as a classification for continuous data set
Please note that my question is not about coding.
I am now learning Bayesian classification and I think I understand it in a discrete case. I have trouble understanding it for multivariate continuous ...
- 316
2
votes
1
answer
876
views
Combining Classifiers with different Precision and Recall values
Suppose I have two binary classifiers, A and B. Both are trained on the same set of data, and produce predictions on a different (but same for both classifiers) set of data. The precision for A is ...
- 21
1
vote
0
answers
59
views
Can we increase the accuracy of a classifier using sketches?
I am using a sketch technique to improve the memory of a standard classifier (naive Bayes) with data streams. The sketch technique is composed of a sketch table (hash table) means the true values can ...
- 103
1
vote
1
answer
192
views
What is D in Optimal Bayes Classification (Machine Learning by Tom Mitchell Ed2)?
I'm reading chapter 6 from "Machine Learning" by Tom Mitchell, 2nd edition.
It seems like the author changes in each paragraph what "D" is without saying anything, but it becomes really confussing at ...
1
vote
0
answers
59
views
Bayes classifier in terms of generative and discriminant analysis approaches
For simplicity I will only discuss binary classification. If $p_k(x) = P(X \mid Y = k)$ for $k = 0,1$, then Bayes classifier $h$ minimizes the risk $P(h(x) \neq Y)$. It is well known that
$$h(x) = 1\{...
- 261
0
votes
0
answers
218
views
Is the Bayes optimal classifier well defined?
The Bayes optimal classifier (BOC) is defined as follows. When data $D$ is given, the classifier returns the value
$$\text{argmax}_{y\in Y} \sum_{h} P(y\mid h) P(h\mid D)\text{,}$$
where the $Y$ is a ...
- 833
1
vote
1
answer
1k
views
Finding the error probability of an optimal bayes classifier analytically
I have two classes $\omega_1,\omega_2$ with equal prior probability $P(\omega_1)=P(\omega_2)=0.5$.
And the points in 2D are distributed $\mathcal{N}(\mu_i,\Sigma), \mu_1=(0,0)^T, \mu_1=(4,4)^T, \Sigma=...
- 11
1
vote
1
answer
232
views
error probability of decision function
If I have a binary calssification task with prior probability $p(0) = 0.6$, and I make two decisions.
1) solely based on the prior probability i.e. I make prediction 0 60% of the time and prediction ...
- 297
11
votes
1
answer
7k
views
Bayes optimal classifier vs Likelihood Ratio
I am getting slightly confused by all the probabilistic classifiers.
The bayes optimal classifier is given as $ max (p(x|C)p(C)) $ and if all classes have equal prior then it reduces to $ max (p(x|C))...
- 665
1
vote
1
answer
846
views
Can the Bayes Optimal Predictor be generalized?
I'm reading Understanding Machine Learning by Shai and Shai. In it, the Bayes Optimal Predictor is defined as
$$f_{\mathcal{D}}(x) = \mathbb{1}[\mathbb{P}[y = 1 | x] \geq 1/2]$$
Where $\mathcal{D}$ ...
- 125
2
votes
1
answer
642
views
Why classifiers report the class with maximum posterior probability as the predicted class?
When we train a classifier to predict $y \in \{1, \dots, K\}$ given an input $x$, classification is done by reporting the class with the highest posterior probability as the prediction; that is:
$$
\...
- 2,261
7
votes
0
answers
3k
views
Help with a proof of Bayes classifier optimality
I have a class assignment to provide a proof that Bayes classifier for the two label version is optimal in that it's error rate is always ${\le}$ any other classifier.
I've never worked through a ...
- 695
12
votes
4
answers
21k
views
Calculating the error of Bayes classifier analytically
If two classes $w_1$ and $w_2$ have normal distribution with known parameters ($M_1$, $M_2$ as their means and $\Sigma_1$,$\Sigma_2$ are their covariances) how we can calculate error of the Bayes ...
- 1,003