Skip to main content

Questions tagged [bayes-optimal-classifier]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
3 votes
1 answer
51 views

Modeling the probability distributions for the Bayes classifier

According to the Wikipedia, the Bayes classifier assumes knowledge of the distributions of $X | Y$, where $X$ and $Y$ are the random variables of the features and the classes, respectively. Let's ...
synack's user avatar
  • 371
1 vote
0 answers
49 views

Is Fisher's discriminant analysis equivalent to the Bayes optimal LDA when the no. of classes is greater than two and covariances are all equal?

P.S. While I gave a brief background to make the question complete, informed readers can move to the questions 1 and 2 towards the end of this post, right after 'what are not clear to me are:'. Fisher'...
Mathmath's user avatar
  • 761
5 votes
1 answer
989 views

Should a language model like GPT-3 be directly used to perform classification?

The OpenAI API enables classification by sampling from GPT-3 given a prompt. Is estimating posterior probabilities a more statistically sound approach? Below is a specification of what "...
chicxulub's user avatar
  • 1,420
1 vote
0 answers
67 views

How do I minimize the cost from errors?

Bayes' Optimal Classifier is known to achieve the minimum error rate for a dataset $x_1, \ldots, x_n, x_i \in \mathbb{R}^d$. Suppose that each error had a cost associated with it. For example, in a ...
z611's user avatar
  • 255
0 votes
1 answer
642 views

Use Naive Bayes to label unlabeled data

I have an Excel file that includes all product information (web scraped from Zalando) of 10k dresses. So for each dress/line I have multiple features available (brand, color, neckline, length...) I ...
Mara Socquet's user avatar
2 votes
0 answers
204 views

Derive the criterion for minimizing the expected loss when there is a general loss matrix and general prior probabilities for the classes

In the book "Pattern Recognition and Machine Learning" I am trying to do exercise 1.23 (p.63): Derive the criterion for minimizing the expected loss when there is a general loss matrix and ...
Slim Shady's user avatar
2 votes
0 answers
792 views

What is the relation between Linear Classifier and Linear Decission Boundary (or Non Linear Decision Boundary)?

As we know (Wikipedia Definition): Linear Classifier makes a classification decision based on the linear combination of the feature vectors. Mathematically : $y = f(\sum w_i x_i)$ So , $f$ is our ...
Girish Kumar Chandora's user avatar
1 vote
1 answer
686 views

How would you find a p threshold for a binary classification prediction? [duplicate]

Lets say that there's a binary classification problem where $X$ ∈ $R_p$ and $Y ∈ \{0,1\} $ and $Pr(Y = 1 | X = x) = p$ for $p$ in $[0,1]$. There is a loss function $L_{falseneg} > 0$ for false ...
321ahno's user avatar
  • 11
3 votes
1 answer
122 views

This decision is the best we can do if we have no prior information about the probabilities of the two classes?

I am currently studying the textbook Learning with kernels: support vector machines, regularization, optimization and beyond by Schölkopf and Smola. Chapter 1.2 A Simple Pattern Recognition Algorithm ...
The Pointer's user avatar
  • 1,902
2 votes
1 answer
234 views

How is this a "Bayes classifier"?

I am currently studying the textbook Learning with kernels: support vector machines, regularization, optimization and beyond by Schölkopf and Smola. Chapter 1.2 A Simple Pattern Recognition Algorithm ...
The Pointer's user avatar
  • 1,902
0 votes
1 answer
72 views

How $\dfrac{P_t(\mathbf{\mathrm{x}}, y)}{P_s(\mathbf{\mathrm{x}}, y)} = \dfrac{P_t(\mathbf{\mathrm{x}})}{P_s(\mathbf{\mathrm{x})}}$?

Related: Empirical Risk Minimization: Rewriting the expected loss using Bayes' rule and the definition of expectation I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai,...
The Pointer's user avatar
  • 1,902
6 votes
1 answer
465 views

Empirical Risk Minimization: Rewriting the expected loss using Bayes' rule and the definition of expectation

I am currently studying Transfer Learning by Qiang Yang, Yu Zhang, Wenyuan Dai, and Sinno Jialin Pan. Chapter 2.2 Instance-Based Noninductive Transfer Learning says the following: As mentioned ...
The Pointer's user avatar
  • 1,902
2 votes
2 answers
1k views

Improve Adaboost that using weighted logistic regression instead of decision trees

I implemented Adaboost that using weighted logistic regression instead of decision trees and I managed to get to 0.5% error, I'm trying to improve it for days with no success and I know it possible to ...
Oran Sherf's user avatar
2 votes
1 answer
158 views

Use different Naive Bayes classifiers to target different data

I am practicing using the Naive Bayes classifier to predict whether people get a stroke or not, but, I am confused with two classifiers. One is categorical Naive Bayes, another is Gaussian Naive Bayes....
Woden's user avatar
  • 207
3 votes
0 answers
247 views

How to proof that the bayes optimal classifier is optimal for a continuous domain

Exercise 3.7 from the book »Understanding Machine Learning: From Theory to Algorithms«, Shalev-Shwartz and Ben-David, states the following: The Bayes optimal predictor: Show that for every ...
user3389669's user avatar
1 vote
1 answer
254 views

Bayes Optimal Classifier for multinomial classification

I understand the meaning and how to deduce a Bayes optimal classifier in binary classification, but I am not sure how to derive this in the context of multinomial classification. Do we use the naive ...
JacksBarrel's user avatar
3 votes
1 answer
1k views

LDA and Fisher LDA - are their weight vectors always equivalent?

Linear Discriminant Analysis (LDA) and Fisher Linear Discriminant Analysis (FLDA) both project high-dimensional observations to univariate classification scores using different rationals and ...
Trisoloriansunscreen's user avatar
0 votes
0 answers
734 views

How to find the optimal classifier for a given loss function?

For a binary classification problem (labels being 0 and 1) and a classifier $g$ we consider the loss function $L(g)=P[Y\neq g(X)]$. It is known that the optimal classifier $g^*$ that minimizes $L$ is $...
mlx's user avatar
  • 281
1 vote
1 answer
46 views

Penalization term for unfairness

I am reading [1], where the researchers do a logistic regression, but add to the loss function the following penalization term for fairness $ R^{AVD}_{FP}(\theta; S) = \left\lvert \dfrac{\sum\limits_{...
robertspierre's user avatar
3 votes
0 answers
71 views

Question about using Bayesian rule as a classification for continuous data set

Please note that my question is not about coding. I am now learning Bayesian classification and I think I understand it in a discrete case. I have trouble understanding it for multivariate continuous ...
Mary's user avatar
  • 316
2 votes
1 answer
833 views

Combining Classifiers with different Precision and Recall values

Suppose I have two binary classifiers, A and B. Both are trained on the same set of data, and produce predictions on a different (but same for both classifiers) set of data. The precision for A is ...
user8115948's user avatar
1 vote
0 answers
59 views

Can we increase the accuracy of a classifier using sketches?

I am using a sketch technique to improve the memory of a standard classifier (naive Bayes) with data streams. The sketch technique is composed of a sketch table (hash table) means the true values can ...
mbahri's user avatar
  • 103
1 vote
1 answer
188 views

What is D in Optimal Bayes Classification (Machine Learning by Tom Mitchell Ed2)?

I'm reading chapter 6 from "Machine Learning" by Tom Mitchell, 2nd edition. It seems like the author changes in each paragraph what "D" is without saying anything, but it becomes really confussing at ...
Cristian Carrasco's user avatar
1 vote
0 answers
58 views

Bayes classifier in terms of generative and discriminant analysis approaches

For simplicity I will only discuss binary classification. If $p_k(x) = P(X \mid Y = k)$ for $k = 0,1$, then Bayes classifier $h$ minimizes the risk $P(h(x) \neq Y)$. It is well known that $$h(x) = 1\{...
user369210's user avatar
0 votes
0 answers
214 views

Is the Bayes optimal classifier well defined?

The Bayes optimal classifier (BOC) is defined as follows. When data $D$ is given, the classifier returns the value $$\text{argmax}_{y\in Y} \sum_{h} P(y\mid h) P(h\mid D)\text{,}$$ where the $Y$ is a ...
Antoine's user avatar
  • 833
1 vote
1 answer
1k views

Finding the error probability of an optimal bayes classifier analytically

I have two classes $\omega_1,\omega_2$ with equal prior probability $P(\omega_1)=P(\omega_2)=0.5$. And the points in 2D are distributed $\mathcal{N}(\mu_i,\Sigma), \mu_1=(0,0)^T, \mu_1=(4,4)^T, \Sigma=...
user3208617's user avatar
1 vote
1 answer
229 views

error probability of decision function

If I have a binary calssification task with prior probability $p(0) = 0.6$, and I make two decisions. 1) solely based on the prior probability i.e. I make prediction 0 60% of the time and prediction ...
Shew's user avatar
  • 297
10 votes
1 answer
7k views

Bayes optimal classifier vs Likelihood Ratio

I am getting slightly confused by all the probabilistic classifiers. The bayes optimal classifier is given as $ max (p(x|C)p(C)) $ and if all classes have equal prior then it reduces to $ max (p(x|C))...
RuiQi's user avatar
  • 655
1 vote
1 answer
828 views

Can the Bayes Optimal Predictor be generalized?

I'm reading Understanding Machine Learning by Shai and Shai. In it, the Bayes Optimal Predictor is defined as $$f_{\mathcal{D}}(x) = \mathbb{1}[\mathbb{P}[y = 1 | x] \geq 1/2]$$ Where $\mathcal{D}$ ...
erip's user avatar
  • 125
2 votes
1 answer
631 views

Why classifiers report the class with maximum posterior probability as the predicted class?

When we train a classifier to predict $y \in \{1, \dots, K\}$ given an input $x$, classification is done by reporting the class with the highest posterior probability as the prediction; that is: $$ \...
Sobi's user avatar
  • 2,251
7 votes
0 answers
3k views

Help with a proof of Bayes classifier optimality

I have a class assignment to provide a proof that Bayes classifier for the two label version is optimal in that it's error rate is always ${\le}$ any other classifier. I've never worked through a ...
Zack Newsham's user avatar
12 votes
4 answers
21k views

Calculating the error of Bayes classifier analytically

If two classes $w_1$ and $w_2$ have normal distribution with known parameters ($M_1$, $M_2$ as their means and $\Sigma_1$,$\Sigma_2$ are their covariances) how we can calculate error of the Bayes ...
Isaac's user avatar
  • 1,003