New answers tagged

1 vote

What distribution assumptions do Gupta, Podkopaev & Ramdas (NeuroIPS 2020) think could be made?

As far as I can tell, in a binary “classification” model, the outcome is conditionally Bernoulli, end of discussion. Thus, it does not seem like there are any distributional assumptions to make. The ...
Sextus Empiricus's user avatar
0 votes

Including Collider Variables in Prediction

Agree with @JulianSchuessler. You should consider causal inference if you will make inference (forgive the redundancy) on the learnt model (i.e. the variable associations). But for mere black box ...
Marco.B.M's user avatar
4 votes
Accepted

Two-sided KS-Test for Evaluating Prediction Model?

EDIT I earlier claimed that the Kolmogorov-Smirnov statistic yields a proper scoring rule. That does not appear to be the case, judging from Thorarinsdottir, 2012, "Proper scoring rules and ...
Stephan Kolassa's user avatar
1 vote

Scaling laws for neural network memorization

The best paper I have seen on the subject of scaling laws for neural network memorization was published at ICLR last year: Provable Memorization Capacity of Transformers. Since you "don't care ...
Bill Vander Lugt's user avatar
0 votes

Compare two datasets and whether they agree

You do not need to give up as Kamil Kaczmarek suggests, and you also do not need to shoehorn a transformed variable into a logistic regression as Peter Flom suggested (although to be fair to his ...
Galen's user avatar
  • 8,370
2 votes
Accepted

How to analyze uncorrelated data?

In some sense, you already have analyzed the data. After all, you determined that that there is minimal (Pearson) correlation between the target and the proposed features. That sure seems like ...
Dave's user avatar
  • 61.6k
0 votes

Problems Determining Optimal Cluster Number for Time-Series Data

You can use results from K-means to explore cluster validity using other methods, based on different distance methods. Firstly, I would recommend looking at the effect of various feature ...
wjktrs's user avatar
  • 815
0 votes

Compare two datasets and whether they agree

I am not so sanguine about assuming 1 to 3 = bad and 4 to 5 = good. But, if you do that, then you can do logistic regression with "marital status" as the dependent variable and "group&...
Peter Flom's user avatar
  • 118k
0 votes

Does a model learn the same attention scores when retrained?

The attention scores of a model directly depend on the weights of the model. Usually re-training a model (unless fixing the seed) will not lead to the same model weights, and for that reason, in many ...
Janosch's user avatar
  • 962
1 vote

Features available during training but not at prediction

Your question relates to a few different and loosely-related concepts, so I'll try to address each of them. (1) In short, SHAP value of a feature $x_j$ is the average absolute contribution of this ...
Spätzle's user avatar
  • 3,719
2 votes

Binary decision boundary requiring 2 hidden layers in neural network with limited neurons

Neural networks with one hidden layer are universal approximators (given some constraints), so you won't find such a class of functions. See Does the universal approximation theorem apply to ReLu?
Firebug's user avatar
  • 19k
1 vote

Scaling laws for neural network memorization

I've thought about this some more and I'm not sure if I can get you an exact "minimum number" or a tight scaling law, but I can get you some bounds if we make many simplifying assumptions. ...
chang_trenton's user avatar
0 votes

How to evaluate multi-class classifier on probability prediction task?

What I suspect you have read is that classification accuracy becomes problematic in the case of imbalance. There are valid arguments that classification accuracy is problematic in the balanced setting,...
Dave's user avatar
  • 61.6k
0 votes
Accepted

Calculating the True Error of a Distribution

You're missing chain rule step which is $D((a,0))=D(a)D(0|a)=\frac{1}{2}(\frac{1}{2} - \epsilon), D((b,1))=D(b)D(1|b)=\frac{1}{2}(\frac{1}{2} - \epsilon)$ So after straightforward substituting your ...
cinch's user avatar
  • 270
1 vote

Find event date given the probabilities of finding an event

If you are starting with probabilities of event, then I don't think you can find the event date. You can find the probability of event taking place on a specific date. You could model this as Poisson ...
Cryo's user avatar
  • 572
0 votes

Literature on discriminatory model VS discriminatory feature

Question is not totally clear, and you would benefit from giving more details. But try logistic regression, even if individual predictor does not discriminate between the two groups, there might be ...
kjetil b halvorsen's user avatar
0 votes

What is the relationship between estimation error, approximation error, bias, variance in machine learning?

The book, and the post above, are not quite correct. In general, bias is not approximation error, and variance is not estimation error. It is not even the case that one is a special case of the other,...
Gavin Brown's user avatar
3 votes

Is it really so bad to do SMOTE on the training set before crossvalidation?

I understand that doing this leads to data leakage, but if I get better performance on the test set does it really matter? YES, it matters! The problem with data leakage is that you can no longer ...
civilstat's user avatar
  • 3,755
4 votes

Is it really so bad to do SMOTE on the training set before crossvalidation?

Yes, it is bad, and it isn't just leakage (+1 to the previous answers). This is because while the synthetic data generated by SMOTE is asymptotically from the same distribution as the data, this is ...
Dikran Marsupial's user avatar
0 votes

Training with a target value as feature

The only thing that limits which features you should use for prediction are which features you actually expect to have when making a prediction. If there are scenarios where you will know y-hat and x ...
Nuclear Hoagie's user avatar
1 vote
Accepted

Maximum Likelihood - Information Matrix Identity Derivation

This is lot easier to show if you use the fact that the expectation of the score is always zero, so $$ \mathbb{E}(s(\lambda;\mathbf{X})^2)=\mathrm{Var}(s(\lambda;\mathbf{X})) $$ You then have $$ \...
Doctor Milt's user avatar
  • 2,817
1 vote

can i have multiple similar features derived from same properties in a dataset?

It depends on the nature of your data. If you have a massive sample size and your variables have low noise, adding more variables will make your model better. If you have a tiny sample size and your ...
pvelayudhan's user avatar
2 votes
Accepted

robust linear regression with interaction

Let's run through the model parameters and emmeans. Your model itself will predict the mean epigenetic age given a set of predictor variables. I'm not so sure a ...
PBulls's user avatar
  • 3,858
1 vote

Can any class of ML algorithms efficiently learn the modulo function (x mod y)?

i would say the answer is 'no', you need a different representation to generalise the mod function as opposed to just memorising the mod for a fixed range of numbers. simple solution (make input the ...
seanv507's user avatar
  • 6,683
1 vote

simple ANN as a set of linear transformations

I think your specific problem is: However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, without using an activation function If you write out the ...
pvelayudhan's user avatar
0 votes

simple ANN as a set of linear transformations

You cannot achieve this using multiple layers and more perceptrons without non-linear activation functions. As you noticed, if no activation is used, any dense network, no matter how many layers and ...
rapaio's user avatar
  • 6,964
2 votes

Which Forecast Evaluation Metric To Use?

The evaluation function and the scaling are distinct issues in my mind. To me, scaling to $0$-$100$ is straightforward: compare to a reasonable baseline model. This is what the usual $R^2$ does by ...
Dave's user avatar
  • 61.6k
0 votes

Random Forest Regressor gives negative test score in GridSearchCV

To answer your question, I consulted the following sources: sklearn.model_selection.GridSearchCV How does GridSearchCV compute training scores? sklearn.model_selection.train_test_split Firstly, it's ...
Carlos André Dos Santos Lima's user avatar
2 votes

Can any class of ML algorithms efficiently learn the modulo function (x mod y)?

The following idea can be salvaged from this paper "Grokking Modular Arithmetic" https://arxiv.org/pdf/2301.02679.pdf. We will construct a two-layer neural network which computes modular ...
dohmatob's user avatar
  • 548
5 votes

Comparison of roc-auc values ​through cross-validation for feature selection

I would suggest using penalized/regularised methods directly and avoid feature selection altogether. CV.SE has some excellent threads on the matter (e.g. Variable selection for predictive modeling ...
usεr11852's user avatar
  • 43.9k
6 votes

Comparison of roc-auc values ​through cross-validation for feature selection

The concordance probability (AUROC) is not sensitive enough for comparing models. Use a sensitive measure such as mean squared error, log-likelihood, AIC. Related information is here.
Frank Harrell's user avatar
3 votes
Accepted

Why does PyTorch Linear allow multiple output dimensions?

One use case I can think of is multivariate regression,as opposed to multvariable regression, the former being a model with more than 1 outcome variable, whereas the latter being a model with one ...
Robert Long's user avatar
  • 60.3k
0 votes

Why is ROC curve always increasing (or non-decreasing)?

Let's answer by explaining all the involved topics. What is a probability threshold? Assuming binary classification, your model should return the predicted probability of a given feature $\{x\}$ ...
Élio Pereira's user avatar
0 votes

How does whitening or decorrelation help machine learning models?

Inputing 2 (and more) correlated features (known to be multicollinear features) to the model you inflate variance of the model's output but in order to create good model you should strive to dicrease ...
JeeyCi's user avatar
  • 191
0 votes

ML Method for directional forecast

This is a binary outcome: either up or down. So many models can be used to do this, ranging from logistic regressions to random forests to neural networks. Perhaps consult the classification and ...
Dave's user avatar
  • 61.6k
0 votes

Why sequential forward doesn't select same feature as sequential backward?

The SBS and SFS algorithms do not give the same feature selection because they are greedy, which results in different choices depending on the "direction" (backwards of forward). The SFS ...
nchabrao's user avatar
2 votes
Accepted

Smoothness of a neural network (specifically second-order)

If we are talking about the usual fully connected / convolution layers combined with activations (i.e. not a complicated deep learning model with all sorts of layers), yes, they are second-order ...
gunes's user avatar
  • 57.1k
0 votes

How is the Representer theorem used in the derivation of the SVM dual form?

The representer theorem isn't needed for SVM. The solution form comes out on its own just by solving for the dual (i.e., introducing variables for the constraint, and through the KKT conditions).
MotiNK's user avatar
  • 1,499
6 votes

Why this error surface for this loss function has negative values>

However, mathematically, the loss function used by the author cannot generate negative values. Right? Correct. This loss function cannot be negative. As mentioned in comments one possibility is that ...
Robert Long's user avatar
  • 60.3k
2 votes

improving classification accuracy of the dataset as a whole by considering classifier distributions

If the problem is to estimate the relative class frequencies in a test set (or operational conditions) that may differ from those in the training set, then the best approach is to use a probabilistic ...
Dikran Marsupial's user avatar
1 vote

could someone please give an concrete example to illustrate what does Multiplicity mean in the context of Bag-of-words model?

Ignoring multiplicity (i.e., multiples instances of the same word) leads to a regular set: $$\text{go to the store to get the milk} \implies \{ \text{get}, \text{go}, \text{milk}, \text{store}, \text{...
Ben's user avatar
  • 124k
4 votes

Neural Network with 1 hidden layer with ReLU modeling capabilities

Neural networks with 1 hidden layer still have 1 output layer. So put a coefficient on each $g$ function, and then you’ll be able to fit decreasing segments.
Sycorax's user avatar
  • 90.7k
6 votes

Is F-score the same as accuracy when there are only two classes of equal size?

No. The F-score is a weighted average of precision (accuracy among predicted positives) and recall (accuracy among actual positives). Neither measure says anything at all about the group of true ...
Nuclear Hoagie's user avatar
5 votes

Is F-score the same as accuracy when there are only two classes of equal size?

NO, see the counterexample below Classify every instance the same way. You have a recall of $1$, a precision of $0.5$, and an accuracy of $0.5$. However, your $F_1$ score is: $$ F_1 = 2\dfrac{\text{...
Dave's user avatar
  • 61.6k
0 votes

Predict unseen samples with Graph Neural Networks

Your question seems theoretical, but I struggled to find the answer to the corresponding practical question of how to actually run inference for a single unseen node. It's well possible that many ...
user8948's user avatar
  • 302
0 votes

PCA: should standardization be applied on features or samples?

I can't figure out the reason why I should standardize with respect to the feature By no means a complete or authoritative answer... I just wanted to point out that PCA finds the axes that are ...
dariober's user avatar
  • 4,180
0 votes
Accepted

Does the Bias of the Model Only Depend on Model Class?

Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to ...
Richard Hardy's user avatar
0 votes

What is the role of temperature in Softmax?

Increasing T above 1 tends to spread the probability among the inputs, while decreasing T below 1 tends to concentrate all of the probability on the most likely class. In the limit that T reaches ...
Josiah Yoder's user avatar
0 votes

What is the best method to calculate confidence intervals on precision and recall which are not independent?

You need to account for the "repeated measurement" of a single user. You can use multiple group shuffle test/train splits where you account for the groups of data generated by a single user ...
Ggjj11's user avatar
  • 1,207

Top 50 recent answers are included