New answers tagged

0

It seems that it has been the standard to use batchnorm in CV tasks, and layernorm in NLP tasks. The original Attention is All you Need paper tested only NLP tasks, and thus used layernorm. It does seem that even with the rise of transformers in CV applications, layernorm is still the most standardly used, so I'm not completely certain as to the pros and ...


0

As noted in my comment above, I suspect that the reason why the first quote you've shared states that the OLS estimator is not the Best (i.e. minimum variance) Linear Unbiased Estimator - it is not 'BLUE' - is because it is in a context where the Gauss-Markov assumptions are not met. As you correctly state, the Gauss-Markov theorem shows that the OLS ...


0

If you're not doing hyperparameter optimization or model selection, you don't need to do both cross_val_score and train/test split. Depending on your size of the dataset, I'd advise doing cross validation (possibly >10 folds and maybe leave-one-out if model executes quickly) and obtaining a performance metric that way. It's more robust than just a train/...


2

$$\operatorname{var}(\sqrt{P(\lambda)})=E[P(\lambda)]-E[\sqrt{P(\lambda)}]^2=\lambda-E[\sqrt{P(\lambda)}]^2$$ The second term can be approximated better as follows, see this post: $$E[\sqrt{P(\lambda)}]\approx \sqrt{\lambda}-\frac{\lambda^{-1/2}}{8}+\frac{\lambda^{-3/2}}{16}+...$$ Which is why the expected value is also approximated by $\sqrt \lambda$. ...


2

Using the same or different splits amounts to using a different experimental design for your optimization: Using the same splits means that you are setting up the comparisons for your optimization in a paired fashion. Paired tests/comparisons typically have higher statistical power which you may use for your optimization decisions.


1

From a linear algebra perspective, we are dealing here with vector spaces. For example, $T : \mathbb{R}^4 \to \mathbb{R}^2$ with $T(x) = Ax$ (transformation matrix). The matrix $A$ has size $2 \times 4$. You enter a 4d coordinate and get a 2d coordinate out. Your input has four features and you transform it into two features. If you have more than one input ...


4

It'd actually be better to use the same folds while comparing different models, as you've done initially. If you input the pipeline object into the randomCV object, it should use the same folds. But, if you do the other way around, each run will change the folds as you said. Even in that case, you can fix the folds by fixing the cv argument in the pipeline ...


0

I think what you're seeing with the loss of 1e-7 is a textbook case of overfitting. It should be obvious why a larger weight decay value increases the model's training loss. The training and holdout loss having approximately the same value shows that you've found the "goldilocks zone" that balances bias and variance for the model -- this is what ...


2

Start with your initial expression for the log marginal likelihood: $$\log p(y|X) = \log \int_{f,u} p(y|f) p(f|u,X,Z) p(u|Z)$$ Multiply by $\frac{q(f,u)}{q(f,u)} = 1$: $$= \log \int \frac{q(f,u)}{q(f,u)}p(y|f) p(f|u,X,Z) p(u|Z)$$ The integral can be seen as an expectation w.r.t. $q(f,u)$: $$= \log E_{q(f,u)} \left[ \frac{p(y|f) p(f|u,X,Z) p(u|Z)}{q(f,u)} \...


7

These are great questions, and their answers can be found in the paper Why the logistic function? A tutorial discussion on probabilities and neural networks by Michael I. Jordan, 1995. I highly recommend you read it all. However, I will provide a short summary here. Note that some of what I write will not necessarily be found in the paper. In the context of ...


0

RNN is a time based neural network.. at the end of time steps ( length of the input) it forms a vector which represents a thought preserving sequence information across the time. Thinking of thought vector like some sort of figure or object might help, which gets it's proper shape (depending on the input sequence) through time steps depending on the inputs ...


0

I haven't done this exact problem, but I can take a stab in the absence of better answers... It looks like this paper is pretty close to your problem, and this one is a similar problem in a different industry. The first will give you lots of ideas about techniques, but has hundreds of thousands of text samples, plus a list of millions of products. They only ...


0

convert it to hours or minutes, then you can use it like that. You may need to scale it also depending on the model you use


0

With unbalanced data, ROC is not a good representation of your model. PR is generally better. With your extremely unbalanced data, you need to think carefully about what your quality goal actually is, then both measure it and optimise for it. I assume that you really dislike false negatives (otherwise the model that always answers "no" would be ...


1

There are several possible scenarios when one would think about calibrating probabilities: The model is misspecified or not optimally trained. That will be the case when non-linear relationships are modeled with a linear learner; or model is too rigid due to excessive regularization (model underfits); or to the contrary, the model is too flexible (overfit ...


0

As an example, RNN with encoder-decoder archs. are used in these cases. But it does not necessarily take text input. You need to convert your text to numeric, and an embedding layer learns cont. vector representations of the converted text. So yes, several ML models can achieve this particular goal, but I believe none of them processes raw text input. Data ...


1

Cross validate the different values for thresholds. But why not do something simpler first and just take the average of the two models?


1

The parameters in a ridge regression can be estimated using gradient descent. Gunes basically answered this above - but gradient descent is a way of estimating the parameters of a model. It's like ordinary least squares - except a lot more versatile and can be applied to a lot of different optimisation problems.


0

For datetime components, best trick is cyclical encoding. This lets the model learn that 1am and 11pm are similar (which isn't the case if you were to feed it raw variables of 01 and 23). Example code for month, or week of year: df['checkin_month_sin'] = np.sin((df["checkin_month"]-1)*(2.*np.pi/12)) df['checkin_month_cos'] = np.cos((df["...


1

TLDR; Contrary to the literature which all traces back to an arbitrary proposed definition, using a $\beta$ term like OP suggests is actually more intuitive than the $\beta^2$ term. A Person's answer does well to show why $\beta^{2}$ appears, given Van Rijsbergen's chosen way to define the relative importance of precision and recall. However, there is a ...


1

You should try to fit a Tweedie distribution, as mentionned by Tylerr, it is adapted to zero inflated count. You could find several examples on the web, like this one: https://towardsdatascience.com/insurance-risk-pricing-tweedie-approach-1d71207268fc


0

I didn't come across with it anywhere to be honest. But, if you apply the activation function to the input layer, it'll correspond to a feature transformation, e.g. it's like taking the cube of all your features and inputting them to the neural network. It may not ruin your data always but note that neural nets typically benefit from zero-mean standardised ...


2

See the description of probability in the documentation, and the note in the User Guide. The issue is that SVC is not probabilistic by nature, and setting probability=True just fits a (Platt) calibration model on top of the support vector model. The predict method still uses the raw support vector decision surface, while the predict_proba method uses the ...


2

Yes, it is possible. XOR problem is a simple example for this case. The dataset is $$C_1=(0,1),(1,0), C_2=(1,1),(0,0)$$ where $C_i$ is class $i$. In the root, the class distribution is $1/2-1/2$. Any split (e.g. $x\lessgtr 0.5$, $y\lessgtr0.5$, ...) will result in the same class distribution, so information gain is 0 or gini-index will not decrease. But, the ...


0

A tree's(Decision Tree/ RF) nodes are only split based on the Information Gain or Gini impurity. The number of features is not a parameter(more precisely, hyperparameter).


1

I don't think that link from scikit learn gives you want you want. That method only produces the quantiles, not any probabilities you see prices that extreme. If I've understood correctly, you want to know the probability that you see price that is over \$55 and under \$45 for something that is priced \$50. The method you linked would give you prices for ...


3

We move in the direction of the negative gradient, but the gradient is different, because in (full-batch) GD and in (batch) SGD the data are different! And that's the point: SGD adds randomness so that it can more easily escape local minima.


4

The core concept is that the gradient is a statistic, a piece of information estimated from a limited sample. The difference between GD and SGD is that if you repeated SGD twice for the same initial parameter values but use different batches, you're likely to get a different estimate of the gradient. This is because the SGD gradient is computed with respect ...


1

I'm guessing that by $j$ you mean the index of the batch, i.e. $j=1$ means 1st batch, right? What is happening is that each column $i$ gets normalized to zero mean and unit standard deviation and then shifted and scaled by $\beta$ and $\gamma$, accordingly. This means that since you have $D_{l-1}$ columns in $H_{l-1}$: $\mu, \sigma, \beta$ and $\gamma$ all ...


0

As quoted here: What measure of training error to report for Random Forests? Andy Liaw, who maintains the R-Package randomForest, states that Random Forests tend to nearly zero loss on trainingset and that this perfect prediction is "by design". So to tell wether a model is overfitting, one has to compare the accuracy from e.g. cross-validation ...


0

Both of them are state estimators, both of them involve the following steps: predict the state ahead ( this is the so-called prior) predict the covariance ahead compute Kalman gain update estimates with means (this is the filtering step) update error covariance the KF or linear Kalman filter is the optimal estimator when the estimations and/or the ...


2

You're inputting binary classification results (1 or 0) to auc and roc calculations. Instead, these functions require non-thresholded outputs. Because, roc curve is plotted by changing a threshold and applying it to the raw model outputs. So, you should input pred in neural net, Predict_logistic in logreg and for the decision tree you need to get probability ...


0

As you have some predictors for the sales variable, I would suggest you to analyze the correlation between different variables and sales. Use those variables as predictors while fitting regression model those have a better correlation. Also, try fitting autoregressive models available for time series analysis. Then you can choose the best model to forecast ...


0

sklearn defines score to be the $R^2$: $$R^2 = 1 - \frac{\text{RSS}}{\text{TSS}}$$ Where $\text{RSS} = n\text{MSE}$ and $\text{TSS} = (n-1)\operatorname{Var}(y)$, the sample variance of $y$. The RMSE on the other hand is: $$\text{RMSE} = \sqrt{\text{MSE}}$$ So we can relate one to the other: $$R^2 = 1 - \frac{\text{RMSE}^2}{\text{TSS}}$$ And $$\text{RMSE} = \...


1

Finding a non-linear separation curve is equivalent to projecting the dataset into a higher dimensional space with a transformation closely associated with this curve. In essence, that is trying to find a non-linear separation. Sometimes, we don't even map the data onto a higher dimensional space (like in RBF kernel) but we use its direct consequences, e.g. ...


1

Independence is something that's hard to formulate mathematically. For example, un-correlatedness is not sufficient for independence (that's why PCA does not perform ICA). Spatial ICA takes the following formulation: $$X = A S$$ $X_{\tau\times\upsilon}$ is you data matrix of observed states. $A_{\tau\times\phi}$ is the mixing matrix, which translates the ...


1

If you have a feature vector of size $p$ and use PCA selecting $p$ components, you will end up with a feature vector that has the same size as the original vector ($p$ variables), and that, globally, provide the same amount of information as the original vector. But individually, the variables would be different. From a start, your new $p$ variables after ...


0

I am not familiar with brain images analysis but I am still going to try to help you here. If each of your independent spatial components are denoted as long vectors $x_i$ $\forall i$, then each of your original brain images can be expressed as a linear combination of your independent spatial components. The fact that your spatial components $x_1$ and $x_2$ ...


0

This guy does an excellent job of working through the math and explanations from intuition and first principles. Take a peek. tl;dr Hinge stops penalizing errors after the result is "good enough," while cross entropy will penalize as long as the label and predicted distributions are not identical. The choice of cross-entropy entails that we aiming ...


0

I'll go through some of your questions: Why can we use this reconstruction loss to judge the pixel-wise differences between the input x and the decoded latent sample z? Implicitly, we assume that pixel values are Normally distributed with uniform diagonal covariance. I mean, x gets encoded and specifies some latent distribution over z's. Now we sample ...


1

Not necessarily. These parameters control how "complex" or "overfit" the model can be. If your model is too simple or too complex, misclassification will increase. There is some sweet spot of parameters for your data and problem, which you should try to find experimentally by measuring misclassification rate on a holdout set (a random ...


0

with k algorithms and n domains (datasets): 1 - You can use chi-square Table with k-1 degrees of freedom for large n (usually > 15) and k (usually > 5). 2 - In the case of smaller n and k, the chi-square approximation is imprecise and a table lookup is advised from tables of Friedman values approximated specifically for the Friedman test. Minimal ...


1

No, number of observations has nothing to do with feature selection or dimensionality reduction methods. You'll only reduce the number of features in your data. Number of observations are typically reduced by procedures like outlier elimination or undersampling.


2

Late answer, but some key points can be added. I personnally think of model stacking as "the natural sequel" of model averaging. And there is a reason why model averages are often better than single models. Model averaging Usually, two (different) models with similar performance often perform better than the best model when the average of the ...


0

Sometimes, the RGB channels in an image store independent information -- for example, the amount of green light versus the amount of red light at a certain location. Other times, we might choose to visualize some scalar field, like elevation or density, by converting scalar values into an RGB color, giving us an RGB image which is more pleasing to the eye. ...


1

The gap between train and validation isn't what overfitting is. If you work under that incorrect definition you'll almost always choose underfitted models. An overfit model is one where another would yield equal or greater generalization (which validation loss proxies) with less memorization of training data (higher training loss).


0

I think you misunderstand the asymptotic unbiasedness. Consider the following estimator of the variance: $\tilde S^2_n=\frac 1 n \sum_{i=1}^n(x_i-\bar x)^2$ as opposed to $S^2_n=\frac 1 {n-1} \sum_{i=1}^n(x_i-\bar x)^2$. While $S^2_n$ is unbiased its brother $\tilde S^2_n$ is biased but asymptotically unbiased , because when $n\to\infty$ we have $\frac 1 n=\...


0

Unfortunately I don't think there is an easy way of doing this. It all comes down to this: you need some way to evaluate how good your GAN is, which isn't an easy task. Why not use the discriminator? The way evaluation is done during training is through the discriminator. So, one first thought would be to hold out one part of the training samples, generate ...


4

In your comments you ask why GLMs fit correlated data well. I think we can answer your question by examining how GLMs are fit and how that fitting procedure changes under correlated features. A GLM is fit by maximizing the log likelihood function. For Binomial Data, this looks like $$ \ell(\beta; y,x) = \sum_i y_i \log(\theta(x_i,\beta)) + (1-y_i)\log(1-\...


1

As Therneau and Grambsch say in "Modeling Survival Data: Extending the Cox Model," Springer (2000), defining the covariates as a function of time is tricky (p. 278): Examples where the covariate path is guaranteed are the exception, however. A major concern ... is whether the hypothetical path represents any patient at all. Survival curves based ...


Top 50 recent answers are included