# Tag Info

195

From Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. https://arxiv.org/abs/1609.04836 : The stochastic gradient descent method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a ...

44

I assume you're talking about reducing the batch size in a mini batch stochastic gradient descent algorithm and comparing that to larger batch sizes requiring fewer iterations. Andrew Ng. provides a good discussion of this and some visuals in his online coursera class on ML and neural networks. So the rest of this post is mostly a regurgitation of his ...

21

TL;DR: Too large a mini-batch size usually leads to a lower accuracy! For those interested, here's an explanation. There are two notions of speed: Computational speed Speed of convergence of an algorithm Computational speed is simply the speed of performing numerical calculations in hardware. As you said, it is usually higher with a larger mini-batch ...

20

You should split before pre-processing or imputing. The division between training and test set is an attempt to replicate the situation where you have past information and are building a model which you will test on future as-yet unknown information: the training set takes the place of the past and the test set takes the place of the future, so you only get ...

15

Stratified sampling aims at splitting one data set so that each split are similar with respect to something. In a classification setting, it is often chosen to ensure that the train and test sets have approximately the same percentage of samples of each target class as the complete set. As a result, if the data set has a large amount of each class, ...

15

There are two things mentioned in the CalibratedClassifierCV docs that hint towards the ways it can be used: base_estimator: If cv=prefit, the classifier must have been fit already on data. cv: If “prefit” is passed, it is assumed that base_estimator has been fitted already and all data is used for calibration. I may obviously be interpreting this ...

13

Part 1: How to read learning curve Firstly, we should focus on the right side of the plot, where there are sufficient data for evaluation. If two curves are "close to each other" and both of them but have a low score. The model suffer from an under fitting problem (High Bias) If training curve has a much better score but testing curve has a lower score, i....

12

I'm adding another answer to this question to reference a new (2018) ICLR conference paper from Google which almost directly addresses this question. Title: Don't Decay the Learning Rate, Increase the Batch Size https://arxiv.org/abs/1711.00489 The abstract from the above paper is copied here: It is common practice to decay the learning rate. Here we ...

10

Stream Mining is one answer. It is also called: Data Stream Mining Online Learning Massive Online Learning Instead of putting all data set in memory and training from it. They put chunks of it in memory and train classifier/clusters from these stream of chunks. See following links. Data_stream_mining from wikipedia. MOA: Massive Online Analysis Article ...

10

I do not have knowledge in ML. After a little web searching, I found a reddit thread that lists the following books - all of which are legally downloadable for free. You can research the titles of your interest for details. Also comment if you find any of the books helpful (and why). Machine Learning Elements of Statistical Learning Hastie, Tibshirani, ...

10

I am interested in this question as well and wanted to add some experiments to better understand CalibratedClassifierCV (CCCV). As has already been said, there are two ways to use it. #Method 1, train classifier within CCCV model = CalibratedClassifierCV(my_clf) model.fit(X_train_val, y_train_val) #Method 2, train classifier and then use CCCV on ...

9

In keras, you can save your model using model.save and then load that model using model.load. If you call .fit again on the model that you've loaded, it will continue training from the save point and will not restart from scratch. Each time you call .fit, keras will continue training on the model. .fit does not reset model weights. I would like to point ...

8

With a similar outcome frequency I have found that data splitting can work if $n > 20,000$. And it provides an unbiased estimate of model performance, properly penalizing for model selection (if you really need model selection; penalization is still more likely to result in a better model) if you only use the test sample once. BUT don't use the test ...

8

Yes, you are correct. If you want to look at the details: For observing the results over parametrization, and the final model chosen, you can compare fit$results with fit$bestTune and fit$finalModel (with same performance the less complex model is chosen). For observing the performance of the final model parametrization per partition and resample, look at ... 7 There is a saturation point. Increasing the size of your training set can't help you surpass the assumptions of your modeling method. For example, if you use a linear model to classify data that is separable in a nonlinear way, you will never get perfect accuracy. As we almost never know the underlying process to its full extent, model mismatch is the norm. ... 7 You should work with each image as a volume of dimensions 224x224x5. You still do 2D convolution over the first 2 dimensions as usual, but keep the entire 3rd dimension. For instance, if you use a 7x7 convolution window, each filter will produce a 224x224x1 volume as output (with stride = 1 and zero padding), and the convolutional layer as a whole will ... 7 But can't the same thing be accomplished with a random.seed function? ... What advantage does using a hash function have over using random seed? Sampling is less straight forward when you can't fit the entire dataset in memory. In the context of a DBMS, this article suggests that using RAND() with a seed may not be reproducible when writing SQL. This is ... 6 Instead of using just one subset, you could use multiple subsets as in mini-batch learning (e.g. stochastic gradient descent). This way you would still make use of all your data. 6 I am posting quite late, but I wanted to provide an answer just in case someone else has this problem. Check that you are turning off dropout when you are evaluating on the validation/test set or if you want to compute error on the training set. Dropout was designed with the express intent of reducing overfitting, so if you are evaluating training loss ... 6 In the reference at the bottom$^*$, I see the training involves the following: Initialize the HMM & GMM parameters (randomly or using prior assumptions). Then repeat the following until convergence criteria are satisfied: Do a forward pass and backwards pass to find probabilities associated with the training sequences and the parameters of the GMM-HMM.... 6 If one is using XGBoost in the default mode (booster:gbtree) it shouldn't matter as the splits won't get affected by the scaling of feature columns. But if the booster model is gblinear, there is a possibility that the largely different variance of a particular feature column/attribute might screw up the small regression done at the nodes. 6 The problem is that L1 regularization requires absolute values of parameters, and the absolute value function is not differentiable at zero. So the usual gradient-based methods (including second-order methods) are not applicable. Non-differentiable functions have to be handled in some other way. In particular, L1 regularization can be framed as a quadratic ... 6 This will depend on how your training and test sets are composed. If the test set is rather big and reflects the "application case" data diversity correctly, I would not argue like this. But if the test data is rather small, you could of course achieve some good or bad results by chance. Using more test data would be helpful is such cases (or using a ... 6 [I]sn't the number of epochs something specific to the training phase of a neural network? You're correct -- the number of epochs is just the jargon describing how many total passes the model has made over the entire training data set. Purely in terms of mathematical terminology, an "epoch" is not really a relevant concept for the purposes of prediction, ... 5 i think it is biased. What about applying FS in N-1 partition and test on last partition. and combine the features from all fold in some way(union/intersection/ or some problem specific way). 5 The conditions under which the output of a neural net can be treated as estimates of posterior probabilities are fairly broad, I remember the following paper as being pretty interesting and informative (caveat: but I've not read it since 2002) Marco Saerens, Patrice Latinne, Christine Decaestecker: Any reasonable cost function can be used for a posteriori ... 5 I can't add a comment to @Emre's answer because I don't have enough points. You can train shared-weight networks in torch, be that using CUDA or not. The weight-sharing is supported for any tensor type. Training is done in parallel when you wrap the two shared modules in a nn.Parallel container We use this in torch quite a lot, to build siamese networks. 5 This is a problem called "label noise" in the machine learning literature. There is a nice paper by Bootkrajang and Kaban, called "Learning Kernel Logistic Regression in the Presence of Class Label Noise" (http://www.cs.bham.ac.uk/~axk/Jo_Patrec.pdf) that is probably a good place to start. 5 Looking at the link you provided gets you the definition of normal loss as: $$\frac{-n}{2} \log(\sigma^{2}) - \frac{1}{2 \sigma^{2}} \sum_{i=1}^{n} (y_{i}-x_{i} \beta)^{2}$$ If you assume that$\sigma^2$(the variance) is fixed this reduces to a square error loss. However if you want to perform inference on$\sigma^2\$ you'll optimize the loss function with ...

Only top voted, non community-wiki answers of a minimum length are eligible