2
I understand that 'Naive' Bayes refers to the approach where all the
features are assumed to be independent.
Features are assumed to be conditionally independent given class label. And, you can fit Gaussian (or any other) Naive Bayes using one feature, in which it's not forced to be Naive any more.
2
Let me copy and paste a warning message from sklearn permutation importance page
Warning: Features that are deemed of low importance for a bad model (low cross-validation score) could be very important for a good model. Therefore it is always important to evaluate the predictive power of a model using a held-out set (or better with cross-validation) prior ...
1
The discriminative value of a feature is based on its statistical distance between classes.
I have calculated the mean and variance for each feature and each class
Using your feature $i$ class $j$ estimated mean $\hat{\mu}_{i,j}$ and estimated variance $\hat{\sigma}_{i,j}^2$, one approach would be to compute the symmetric KL divergence for each feature for ...
1
It is just a linear function with parameters $\beta_1 = 1$ and $\beta_2 = -1$
$$
\beta_1 x_1 + \beta_2 x_2 = 1 \times x_1 + (-1) \times x_2 = x_1 - x_2
$$
Neural networks use such linear functions commonly. If you have a simple multilayer feed-forward network, just one of the layers would need to learn the parameters above to calculate the new feature to be ...
1
It depends.
Most likely the network is probably going to pickup some of the noise from the 'redundant' variables so you could see an increase in validation forecast accuracy and a decrease fit accuracy if you drop them. You could also see nothing much happen if the variables are truly redundant and the network isn't really doing much with them anyway.
So, ...
1
I did an adjustment to Shannon's formula, where $X$ is the random variable of study:
$$H(X)=-\sum_i p_i \times \log_2 pi \\ H(X)_{adj}= H(X) \times \frac{1}{\log_2n}$$
Where $n \in \mathbb{N} - \{1\}$ is the number of levels of $X$ and $p_i$ represents the probabilities of $X = x_i$, $\forall i =1,\dots,n$.
Please let me know if I'm making a mistake or if ...
Only top voted, non community-wiki answers of a minimum length are eligible
