New answers tagged classification
5
This is sometimes called "concept drift:" when you're developing the model and deploying it, the scores do a nice job of matching the true values, but over time, the scores and true values start to diverge.
Concept drift arises in lots of contexts that have a time component that is not explicitly modeled. For example, new malware will be invented ...
1
If you are using trees then the algorithm will select the bins for you, regardless of whether the variable is skewed or normal or whatever. There is no need for you to "pre-bin" and such an approach can only make the result worse.
If you are using some kind of regression (you mention logistic regression) then you can use a spline of a continuous ...
0
Unfortunately, there's no way to answer this other than saying that you choose the operating point that matches your needs -- I can't tell you what the trade-offs are, because I don't know the equities involved, and how I value the outcomes might differ from how you value them. You'll have to ask yourself how many FPRs are you willing to incur to achieve a ...
0
Yes, the typical cross validation assumes iid samples, so that it can freely split the data into training and validation. In case of dependency, such as the temporal dependency in time series datasets, modifications respecting this dependency should be done for the splitting of data. Otherwise, there will be data leakage. See the following for an example of ...
0
I have been looking into this a bit more, and it seems as though a Normal confidence interval plus a logit transformation does very well in modest sample sizes.
As earlier, define
$$\widehat{\mathrm{sens}}\sim N(\mathrm{sens}, \sigma^2)$$
and
$$\widehat{\mathrm{spec}}\sim N(\mathrm{spec}, \tau^2)$$
then for balanced accuracy
$$\widehat{\mathrm{bla}}\sim N\...
0
Adding to montols answer:
I think he is right on most points, except that, from my understanding, it is the learning rate 𝜖, not 𝜆, that controls for validity of the Taylor expansion(TE). This is because 𝜖 scales the final step size taken towards the TE-minimum and for small 𝜖 TE clearly becomes a better approximation. Moreover, since the Hessian is ...
1
For the machine learning classifiers that I know of (i.e. logistic
regression, neural networks, bayesian classifiers, etc.), I am only
familiar with these giving simple 1 or 0 classifications of some input
falling into a particular class.
This is not true. Most machine learning algorithms make predictions in some kind of score, that can be used for making ...
0
In my opinion, yes you can use the variables that are most significant in constructing the respective factor scores, especially if you data is data-centered (minus the respective mean).
The most obvious advantage is, unlike factor analysis produced constructs, your explanatory variable is readily understandable. However, the standard regression derived beta ...
1
In Logistic Regression, collinearity inflates the uncertainty in the learned estimates. Here is an example. I'll simulate 1000 datasets from the same phenomenon and fit a model. Then, I will plot the estimates of that model under the assumption that the features are independent and that they have correlation of 0.8
sim<-function(corr=F){
if(!...
0
Whether to bin or not to bin may be answered by the quote (due to George Box?):
All models are wrong, but some models are useful.
Broadly, models are created to either understand the data or to make predictions (and of course for both!).
In your situation I would carry out some experiments and test a range of bin sizes starting with a no bin model.
The "...
1
I'm going to assume that by multicollinearity you mean a perfect linear correlation between predictors, and the goal of the modelling task is prediction (not causal inference), because that's my first guess based on the wording of your question.
In that case, multicollinearity may not be a problem in decision tree or random forest, even if it is a problem in ...
0
I remember an example of a donut distribution (not a data distribution with a multi-modal radial component if that is what you are looking for) from a course that I took in probabilistic robotics which dealt with Bayes filters, particle filters, Kalman filters, and so on. It might be a bit to specific to this area.
The example is related to a task called ...
2
Assuming, there's no problematic bias in how teachers are assigned to students (ideally, you'd have [stratified] random allocation), one standard approach to this would be to fit a random effects model. Let's say that the score is a number on an ordinal scale (i.e. a finite number of discrete values e.g. A, B, C, D, E, F or 1,2,3,4,5,6 or 0,1,2,3,...,15 or ...
1
One approach that comes to mind is to segment the image into "road" and "not-road" (a nice introduction is given e.g. in lecture 7 of the fastai Deep Learning for coders course - a lot can also be found in relevant Kaggle competitions).
Note that the fastai library or its version 2 fastaiv2 (associated with the aforementioned fastai ...
1
Given that you're assuming the two tests are the same and independent Bayes formula becomes
As you thought, you'll need an estimate for P(I), which I'm sure there are a number of estimates floating around for COVID.
1
I know I'm late to the party, but: the theory behind the data imbalance problem has been beautifully worked out by Sugiyama (2000) and a huge number of highly cited papers following that, under the keyword "covariate shift adaptation". There is also a whole book devoted to this subject by Sugiyama / Kawanabe from 2012, called "Machine Learning ...
2
While I'm not at all convinced balanced accuracy is a useful summary, that's also not how you compute a confidence interval for it.
To a reasonable approximation, the estimated sensitivity and specificity will be Normally distributed around the true values.
If
$$\widehat{\mathrm{sens}}\sim N(\mathrm{sens}, \sigma^2)$$
and
$$\widehat{\mathrm{spec}}\sim N(\...
0
Sensitivity and specificity are two entirely different measures of the usefulness of a test. One is based on a (presumably small) population of subjects who have the disease or condition; the other is based on a (presumably much larger) population of subjects who don't.
I can see no valid rationale for averaging the two. As an example, suppose a test has ...
2
With respect to the question: "Is the notion of bias and variance relevant to a classifier?" an answer is not directly or accurately.
My rationalization is based on a statistic (the Gini coefficient) which is related to one of the more utilized metric in machine learning application, namely AUC, which stands for the area under the ROC curve.
Note, &...
0
I've done the experiment where I train the Random Forest with 1000 trees on 72 classification tasks from OpenML-CC18 benchmark. I got better results when tuning the number of trees with single tree precision than by using all 1000 trees.
By tuning I mean: train the Random Forest with maximum number of trees (in my case 1000) and then check the performance of ...
1
There are two things to separate here:
The metric
The threshold
You should choose the metric based on business goals. If you need a good balance between precision and recall, F1 is a good choice; though as I mention in my answer to this similar question I have found models that optimize logloss tend to be more robust when released into the wild.
For the ...
0
In such a case, you may have a biased estimator since data will be imbalanced as you mentioned. But this is ok as long as you are aware to the different mis-classification errors (and their costs) and their interpretations. All errors cost the same?
Yet, the question of whether to take care of the bias and how - depends on the interpretation of the final ...
0
Following this issue, looks like it is recommended to use the decision_function() in such a case.
For example:
X = np.array([[1, 2], [2, 1], [-2, 1], [-2, -1]])
y = np.array([2, 0, 1, 2])
multi_clf = OneVsOneClassifier(Perceptron(shuffle=False))
ovo_prediction = multi_clf.fit(X, y).predict(X)
ovo_decision = multi_clf.decision_function(X)
votes = np.round(...
0
Welcome Kipkogei Francis!
The prediction step should normally provide you with two columns each with one predicted value for each class.
If you provide a sample of your data and the code you have tried then it may be possible to solve your problem otherwise only general comments are possible.
4
The short answer appears to be Yes: there is some evidence that upsampling of the minority class and/or downsampling of the majority class in a training set can somewhat improve out-of-sample AUC (area under the ROC curve, a threshold-independent metric) even on the unaltered, unbalanced data distribution.
With that said, in most or all of the examples I've ...
1
In your example, SVM will try to divide your observations into two groups: cancerous and benign. Clustering looks at all of the features of the observations and tries to form clusters such that the points within each cluster are most similar to each other. It doesn't care about whether a tumor is cancerous or benign, just which tumors are similar to each ...
1
With supervised learning you point what exactly you want the algorithm to learn. If you had a medical data on patients diagnosed with cancer or not, clustering algorithm may find many different kinds of pairs of clusters, e.g. young vs old patients, men vs women, etc., and you have no guarantee that it would be about the medical condition. Of course, if ...
0
In a real production environment, if you want to predict ytest using Xtest, you won't have access to ytrain as you did in your experiment. The statistician's suggestion mimics reality better and is expected to provide more reliable predictive performance. With that said, is your approach not reliable? Not necessarily. I think the reliability depends on the ...
1
What you seem to be missing, is that after sampling the bootstrap sample, you use this data to train the model, then you make predictions using it, and calculate accuracy. Sampling the results calculated on all data would not work, because it does nothing to check what would be the results if you had different data to train your model.
See this and this ...
0
You need to decide two fundamentally different things when measuring the performance of a predictive model:
A plan how to get appropriate test cases, and
figures of merit that measure the performance properties you are interested in.
These two are independent of each other! In other words, "we need to measure TPR" does not tell you on what kind ...
1
There are a couple of query algorithms to do so, see e.g. QBC algorithm. The basic idea here is to train a set of diverse and reliable learners/models on the currently labeled dataset. Then you use these models to predict the class of each unlabelled datapoint. For a certain datapoint, If all the learners predict the same class, you conclude that you have ...
0
Yes, but be careful if you are planning to do a performance test (such as cross-validation).
In that case, you need to define the clusters using the training set and project the test features on the training clusters. For instance, if you discretize the training features using k-means, the test features can be assigned to the closest cluster (be careful ...
4
In a slightly different direction, one way to look at this is to consider more generally the continuous ranked probability score (CRPS), which is a proper scoring rule.
For a predicted CDF $F$ and an observation $y$, the CRPS is defined like this:
$$\text{CRPS}(F,y) = \int (F(z)-I(y\leq z))^2dz$$
Intuitively it is a measure of the distance between $F$ and a ...
9
Take a simple example where $p_i$ are known probabilities and $y_i$ are Bernoulli($p_i$).
What is $\hat y_i$? The best choice is obviously $\hat y_i=p_i$. Alternatively, we might take $\check y_i = 1$ if $p_i>0.5$ and $\check y_i=0$ if $p_i<0.5$.
Suppose $p_i>0.5$ (for simplicity).
The expected Brier loss of $\hat y_i$ is $(1-p_i)^2p_i+p_i^2(1-...
11
Let's first make sure we agree on definitions. Consider a binary random variable $Y \sim \text{Ber}(p)$, and consider a loss function $L(y_i|s)$, where $s$ is an estimate of $p$ given the data. In your examples, $s$ is a function of observed data $y_1,\dots,y_n$ with $s = \hat{p}$. The Brier score loss function is $L_b(y_i,s) = |y_i - s|^2$, and the absolute ...
0
@Accumulation made good points (+1) that by minimizing loss you are already incorporating feedback and that you can always use weighted ensemble of different classifiers.
Another thing that you could do is to use AdaBoost algorithm, where you train a model, then calculate errors, and train another model where misclassified samples are up-weighted during ...
1
is it different somehow in the multi-class classification? Or does it mean my model is basically guessing randomly?
You have to understand how you get the score of a random baseline.
Suppose you have a binary classification problem where classes are balanced, that is, the pmf of the labels in your dataset is $q_i=1/2$. If you do random guesses, you don't ...
1
This webpage answers your question, http://nicolas-hug.com/blog/pdps.
A minor issue is that I don't think the fast way always have the same result as the slow method (the definition). The fast way is more like averaging on the conditional distribution of the other covariates (not the strictly defined conditional distribution, but in the tree structure).
0
You don't need to do anything, at least as long as the variables as encoded as numerical values, i.e. binary variables need to be coded as 0 and 1, or -1 and +1, categorical variables need to be dummy encoded, for things like dates you need to come up with some meaningful coding (e.g. number of days since some fixed date), etc. Regression will handle ...
4
Yes, what you're describing is a model where the predicted probability of the positive class is obtained by passing a piecewise linear function of the input through the logistic sigmoid function. That is:
$$p(y=1 \mid x) = \frac{1}{1 + \exp(-\phi(x))}$$
where $y \in \{0,1\}$ is the class label, $x \in \mathcal{X}$ is the input, and $\phi: \mathcal{X} \to \...
1
I'm assuming that you have in mind that the number of "knots" (pieces of the piecewise linear function) are known, but their locations are not.
Here are two ideas.
Decision trees
Vanilla decision trees (trivially) form piecewise (axis-aligned) decision boundaries, but I don't think that's what you had in mind.
"Multivariate Decision Trees"...
0
After having read the paper again in-depth, I found the answers to my questions:
Answer to first question Yes, it's (much) better to use the 1. formula, that is, calculate the macro F1 as average over class-wise F1 scores. The other macro F1 formula (harmonic mean over class-wise precision and recall averages) can lead to overly-optimistic scores because it ...
1
This problem is sometimes called Open set recognition, or classification. There was a recent survey on open set recognition on Arxiv https://arxiv.org/pdf/1811.08581
The problem is also called zero shot learning https://en.wikipedia.org/wiki/Zero-shot_learning when the data points are images (I think)
0
I agree with you and I see no information contamination from the test into the training. You do not use any of the xtest sets into the training. The most one can claim is that probably the Xtest is not that different that Xtrain, and the same for the ytest and ytrain given the autocorrelations of naturally occurring time series. If you can use the ...
1
I can't see any reason why they wouldn't be comparable, but I don't know what I could cite to convince a skeptic of this. They're probabilities. As long as P(L present) + P(L not present) == 1 for all labels L (which they have to be to be probabilities), the probabilities of different labels should be comparable.
Nevertheless, there may be cases where, for ...
2
Yes, it all depends on the algorithm and the cost function. For example, logistic regression's output is between 0-1 and you can't calculate cross entropy loss with labels -1 and 1. Similar things happen in neural nets, you'd want to use tanh in the last layer (because it's output is between -1 and 1). But, you won't be able to use the cross entropy loss ...
1
I think you can use glmnet package, with options family='binomial' and adjusting ElasticNet coefficient alpha to balance between L1 and L2 regularisations. An example is here.
Also, wouldn't the training error for my models just be 0? As I am
training my model with this data, and then predicting it against the
same data? Thanks.
Not necessarily, and ...
0
I'm a complete noob and I'm not sure if my case translates to the case you're describing, but I'll have an attempt and accept criticism.
From my experience, the problem with F1-score is that it doesn't consider true-negatives. This means that in the case of heavily inbalanced datasets, the false-positives (when considering the minority class) will dominate, ...
2
Practically, I'm unsure why one would need to rely on AdaBoost if we already had a strong classifier. Tl;dr: I don't believe that having a weak learner is requirement for AdaBoost to work.
I can try to walk through some of the analysis. We'll deal with empirical error, and then generalization error.
Empirical Error (train)
We define a weak learner as any ...
Top 50 recent answers are included
