# Tag Info

0

$w^t w$ term can be written as: $$w^t w = \sum_j w_{j}^2$$ This becomes smaller when $w_j$ is closer to zero. That is why the weight tend to be close to zero due to the regularization term.

0

The $w^Tw$ is a quadratic function with minimum at $0$. If the $w$ is "big" (in term of its distance from origin), the $\lambda w^Tw$ term grows and make loss bigger.As we are trying to minimize the loss, we will prefer $w$ to be small (i.e. closer to origin).

0

I think this issue is perfectly discussed and explained by Dr. Kilian Weinberger from Cornel University. Here are the lecture notes: http://www.cs.cornell.edu/courses/cs4780/2018fa/lectures/lecturenote12.html and http://www.cs.cornell.edu/courses/cs4780/2018fa/lectures/lecturenote11.html You can also find video recordings of the course, so you can follow up.

1

Accuracy is a misleading KPI for predictive performance. Note that every criticism raised in that thread against accuracy applies equally to the $F1$ (or more generally, any $F\beta$) score. As Sycorax comments, use proper scoring rules on probabilistic predictions. That is also my recommendation in the linked thread. Contrary to user2672299, I'm all for ...

4

Your thinking is correct. I recommend Gneiting & Raftery (2007, JASA) for an in-depth discussion of scoring rules. A scoring rule $S$ is a mapping that takes a probabilistic prediction $\hat{p}$ and a corresponding observed outcome $y$ to a loss value $S(\hat{p},y)$. In our application, $\hat{p}$ is just a single number (that will depend on predictors, ...

1

There is nothing special about using convolutional layers. In fact, for simple problems like MNIST, you could use simple neural network. Why we use convolutional layers, is because they use sliding windows that detect features of images no matter where are they located in the picture. If the image is centred, that's not the case. Notice that each of the ...

1

You can do one of three things: try different approaches still, like RNNs, CNNs, word vectorization... depending on your sample size and on your problem, they can be quite beneficial make an ensemble of the models you used. Simple model averaging increases generalizability and can easily improve performance in the training and validation sets as well. On ...

0

You want Athey's casual forests to get the average treatment effect. The calculation of ATE in these settings can be a little tricky otherwise. They've set it up nicely as an R package. https://arxiv.org/abs/1902.07409 One might think that simply manually calculating a forecast of Y with RF and then manually evaluating $E(Y|X=1)-E(Y|X=0)$ ought to work (by ...

2

Yes, you are overfitting, and very much so. Use a holdout sample to assess your model's predictive capabilities. Optimizing in-sample fits will usually lead to overfitting, especially with a large number of predictors, as in your case. Consider using a regularized/penalized model to deal with your large number of predictors, like GLMNET or similar. Finally, ...

2

Concordance probability (AUROC; c-index) are not appropriate for comparing two models, because of lack of statistical power. See fharrell.com/post/addvalue for measures that are sensitive enough to be used for such comparisons.

1

If you do have sparse data, filling it with 0 could affect your model. It is possible to fill the gaps by interpolation or collaborative filtering. That being said, if you naturally have a lot of zeros in your dataset, any Anomaly Detection model can work with that. Depending on how many dimensions and samples you have, you most likely don't need special ...

1

Use classification when the number of categories are limited and nothing in between makes sense. For example a class is either a dog or a cat nothing in between. But when it comes to something like ratings, 3 is as likely acceptable as 3.5 ((so is 3.56, etc) so you are not bound to only one value among others. it can be in between as well. Apart from this, ...

1

Your model includes too many predictors for the number of observations. A rule of thumb for logistic regression is to have no more than 1 predictor for each 15 or so members of the minority class. With about 60 total cases (based on the null degrees of freedom), you have no more than 30 members of the minority class. So anything over about 2 or 3 predictors ...

9

There’s no contradiction. The fact that something is easy to interpret has nothing to do with how accurate is it. The most interpretable model you could imagine is to predict constant, independently of the data. In such case, you would always be able to explain why your model made the prediction it made, but the predictions would be horrible. That said, it’s ...

2

The curve is valid. The most commonly seen ROC curves have a curve above the diagonal, which indicates that large predicted scores are associated with the label, and low predicted scores are not associated with the label. The diagonal, i.e. the line $\text{TPR}=\text{FPR}$ of a ROC curve corresponds to the model that has a completely random relationship ...

0

The answer should be 3. VCDim(H)$\geq$3 because any labelling (even of the form (1,-1,1)) is realisable by H. Just consider $h_{b,b,-1}$, where b is the middle point of your chosen subset. On the other hand, VCDim(H)<4 because the labelling of (1,-1,1,-1) is not realisable by any function in H.

2

# the collected data ages.16.35 <- c(20,30,60,15,90) ages.36.55 <- c(15,50,30,20,40) ages.older <- c(35,30,10,10, 5) total.16.35 <- 100 total.36.55 <- 60 total.older <- 40 Let the new data be coded as $D=\{v_1, v_2, \ldots, v_5\}$ where $v_i$ is 1 if the video $i$ was liked, 0 otherwise. In the example you give, that is $D=\{0,1,1,0,1\}$...

2

just create list of all files, then sample this list and voila!, as for calculations: probability sampling image from class $A$ sampling from whole set is $P(class\_A)=size(A)/size(whole\_set)$ but in second sampling/drawing $P(class\_A)=\frac{size(A)-1}{size(whole\_set)-1}$ because I presume you are sampling without replacement, of course after picking ...

4

The vanishing gradient problem: [1] As more layers using certain activation functions are added to neural networks, the gradients of the loss function approaches zero, making the network hard to train. The reason that recurrent neural networks (RNN) suffer from the vanishing gradient problem is due to the nature of backpropagation through time. [2] An ...

0

The Root Mean Squared Error is the square root of summed squared errors. It is in a way an "expected error". You could say "on average, the error between the observed data and the model prediction will be 5 units". Could it be smaller/larger? Yes. But on average it would be 5.

0

Broadly speaking, you are correct that in machine learning we are given some data $\mathbf{X}$ and labels $y$ and we want to learn some function $f$, such that $$y = f(\mathbf{X}) + \varepsilon$$ where $\varepsilon$ is some noise. More formally, this is an oversimplification, as noise does not have to be additive, and the structure of noise may be mroe ...

0

Quoting from here: Converting a binary variable into a one-hot encoded one is redundant and may lead to troubles that are needless and unsolicited. Although correlated features may not always worsen your model, yet they will not always improve it either.

1

This problem is much more complicated than it may look. Try to construct a solution for your simple example. In general, this problem in general can be interpreted as multidimensional knapsack problem and thus even approximations are very hard to find - the one-dimensional knapsack problem is already NP-hard. In your toy example one can simply try all ...

1

The easiest path forward is to just reduce the size of the MNIST data by 99% or more by discarding most of the examples. You could even do something as extreme as only keeping 20 images, 1 of each class as training data and another of each class as test data. This is the smallest you could possibly make the MNIST data without doing something fancy like ...

0

My general advice would be to transparent in how you report your findings. There are standards in different scientific communities, but there is no single way of performing cross-validation. That being said, let me try my best to answer your three questions: Generally, when using cross-validation, you need to split in to THREE groups. Training, validation, ...

0

To understand why we use the relative and subjective notions of under- and overfitting, we must remember that, as George Box said, "all models are wrong" (see here for an explanation of this aphorism), but some of them are useful. When confronted to data whose generative model is unknown, we can define a set of plausible and competing models to ...

0

I was digging about this question, and I found the following: I think KNN cannot be penalized. During testing, knn is supposed to find the closest example in the training set and return the corresponding label; it finds the k-nearest neighbors and returns the majority vote of their labels. There are many approaches to selecting the best model: We can choose ...

0

So, first of all, this is an answer based on the one by @cbeleites above, this one here, and the question itself (all these contributions helped me understand). There is nothing original in it, and although it makes sense to me, I am still a student in this topic so I am not 100% sure of it. Therefore, any feedback is appreciated. However, it gives a ...

1

The logit is used here as the link function. The RHS of (1) can range over the real numbers, and this is the reason for the logarithm, to ensure that the LHS also ranges over the reals—without the logarithm it would only range over the non-negative real numbers. Further, this link function is the canonical link function. Other link functions include the ...

0

I made a picture (see sources for original pictures) showing cells as classically represented in tutorials (Source1: Colah's Blog) and an equivalent cell with 2 units (Source2: Raimi Karim 's post). Hope it will clarify confusion between cells/units and what really is the network architecture.

0

Your polynomial regression has a lower mean absolute ERROR, lower mean squared ERROR, and higher $R^2$, which is a function of mean squared error, thus redundant. ($R^2$ is useful in linear regressions, which your support vector regression is not, as it has an interpretation about the "amount of variance explained” that does not apply to nonlinear ...

1

Graph convolution of the spatial variety can be applied to arbitrary sized graphs, the same way that normal "grid" convolution can be run on arbitrary sized raster inputs. The same can also be said of most (all?) spectral versions of graph convolution. In both cases, you can see that the size of the weights are independent from the size of the ...

1

ERM is equivalent to MLE when the risk is defined as the negative of the likelihood. Other than that I think the differences are subtle and differ from author to author. I'd expect that usually we would only call a method MLE if it optimizes some likelihood function by selecting $\eta(x) = \mathbb{P}(Y | X=x)$ while empirical risk minimization (in the ...

0

Training regression models, and closed-form analytical solutions (what you called "formulas") for computing the $parameters$ of regression models ($m$ and $b$) are merely equivalent approaches, that's all. To be honest, most multiple linear regression scientific packages (programs) carry out approach $2$ under the hood

0

There are a few simple approaches you might try. They all exploit different kinds of patterns in your data, and they all have at least one parameter you'll need to tweak. Fixed-duration windows around a peak. As you've said, this doesn't work very well. Set a y-axis threshold. A peak starts when the value goes above this threshold, and ends when it goes ...

0

I don't think there are papers about feature extraction, probably there are only for single components to be applied into AE in order to reduce reconstruction error. Besides that, if you want to extract features from an image I also suggest an AE using the following approach Train AE to reconstruct images Get encoder and use that to find meaningful features ...

0

One of the advantages of MARS, is that it automatically performs variable selection. This can also be viewed as a disadvantage in certain applications, when you know that the response variable is a function of all parameters. Would a Bayesian approach be acceptable to you? The BASS package in R offers a fast implementation of Bayesian MARS, with at least two ...

4

For instance, would it make sense or add any benefit to train multiple RFCs on different subsets of a training data set, [...] Random forest does by default train each tree on a different subset of data. I can't see how doubling this procedure would add anything. Or, would it be sufficient to do a random train/test split on a percentage of a training data ...

1

As stated in the comment, same as during training you take outputs of first network as inputs to second one, during inference time you could do the same. This is simple function composition, if you treat the two networks as functions $f$ and $g$, and your output is $g(f(x)) = y$ then they form a composition $(f \circ g)(x) = y$. You don't need any fancy math ...

4

I used to develop these models professionally for a major casualty insurer, and probably had a part in developing the data for one of the Kaggle competition's you're referencing. So I'm relatively well positioned for this question. Can someone please explain the use/need for using Tweedie or poisson instead of the regular mean squared loss as objective. ...

0

The previous answer is wrong in the part of bias derivation. I think the correct and full answer should be the following. Firstly, I should note that for kNN we use an important assumption that all $X_i$ are fixed in the training set, i.e. $\mathcal{T}=(x_i,Y_i)_{i=1}^N$ (all randomness arises from the $Y_i$). Hence, here the training set $\mathcal{T}=(x_i,... 1 In the wikipedia page, it says ... where function Z(p), p ∈ [0,1], is the inverse of the cumulative distribution function of the Gaussian distribution. So, the input to$Z$function is a single value in$[0,1]$, which can be hit rate, false alarm rate or AUC, which means you don't need a distribution. It doesn't say but it should be the inverse CDF of ... 2 I disagree with user2974951's answer. You should aim at probabilistic classifications that are well-calibrated and sharp. Oversampling the minority class will bias your predictions, so don't do it. Use a representative sample of your initial data for training your model. Once you have your probabilistic classifications, you can calculate your AUC. In ... 1 You are doing hierarchical forecasting, specifically using the top-down method. That is, you first forecast the aggregate, then break it down. An alternative would be the bottom-up method, where we first forecast daily orders, then aggregate them up. In top-down forecasting, there are of course different ways of disaggregating. What you are doing is ... 0 If you must follow the requirements, then the first option would be better for the simple reason that you have more data for the minority class, so you should be able to capture more of the variability in this class. As opposed to the second option where you would first have to (find a way to) select a small subset of all the data (and throw the other data ... 0 Your flaw is in step 3 + 4: if you again want to select a model as optimal, you need to reserve yet another test set that is independent of steps 1 - 4. Moreover, the selection in step 4 is based on performance estimates that have only 1/4 of the test sample size compared to the selection in step 3 behind them. This makes that estimate more uncertain. ... 0 First of all is need to say that for prediction evaluation, then out of sample, the usual$R^2$is not adequate. It is so because the usual$R^2$is computed on residuals, that are in sample quantities. We can define:$R^2 = 1 – RSS/TSS\$ RSS = residual sum of square TSS = total sum of square The main problem here is that residuals are not a good proxy for ...

0

Someone explained like; The batch size is a hyperparameter that defines the number of samples to work through before updating the internal model parameters. Think of a batch as a for-loop iterating over one or more samples and making predictions. At the end of the batch, the predictions are compared to the expected output variables and an error is calculated....

1

For "performing k-means" you don't need visualisation. You can just run it, 26 columns are not a problem. How you can "perform" it and how to visualise the result are two different questions. You could visualise the result for example on the first two principal components.

0

You should train on all your training data at once. Hyperparameter tuning is done by many runs of the training each time on the same dataset, just different parameters. See this explanation: https://www.jeremyjordan.me/hyperparameter-tuning/

Top 50 recent answers are included