Questions tagged [pre-training]
Unsupervised pre-training initializes a discriminative neural network from one which was trained using an unsupervised criterion, such as a deep belief network or a deep autoencoder.
51
questions
0
votes
0
answers
19
views
Leaving duplicated entries in a dataset at pretraining stage
I'm adopting a fine-tuning approach after having pretrained a deep learning model (transformer) on a source dataset (let's call it dataset A) and then fine-tuning it on a target dataset (B). Dataset A ...
- 647
0
votes
0
answers
825
views
How to calculate loss in pre-training gpt-2
As I know BERT model that calculates loss by compare between an original word that was masked with the predicted result for retrieve the loss and update the weight in pre-train model.
But GPT-2 uses ...
- 1
7
votes
2
answers
496
views
Is pretraining on test set texts (without labels) ok?
Edit: after skimming this paper6, I narrowed the scope of this question to NLP problems. Relevant excerpt from the abstract (emphasis my own):
We demonstrate that unsupervised preprocessing can, in ...
- 1,430
1
vote
0
answers
66
views
Transfer Learning "from scratch"
I've recently started to work in machine learning and this is my first post here. Excuse me in advance for duplicates and/or slang mistakes. My question is about transfer learning (although in this ...
- 13
1
vote
0
answers
33
views
References for training BERT-like models from scratch
As the title of the question suggests, I'm interested in training a BERT-like model (and then use it to make some experiments on text-similarity).
Question: Could you share some references on the ...
- 337
1
vote
0
answers
3k
views
Where can I find pre-trained Pathways Language Model (PaLM)?
I am amazed by the possibilities coming from the PaLM model after reading the following post: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html . I was trying to find the ...
- 141
1
vote
0
answers
14
views
Suggestion regarding usage of pre-trained BERT
I have recently been working with the pre-trained BERT. It produces quite good results on supervised tasks with just a bit fine tuning.
But now I wonder if I want to perform some unsupervised task on ...
- 111
1
vote
0
answers
147
views
How to handle problem of different random seeds giving drastically different test scores in machine learning model?
For a rigorous empirical analysis, I am training a model with three different seeds - 0, 1 and 2. In each case, I found that the model obtained through early stopping (lowest validation loss) had an ...
- 143
1
vote
0
answers
37
views
Best way to train a model having train/test/val set
I'm new in the machine learning field and I'm trying to understand what are the best practices to manipulate the time series. I currently have a dataset structured as well:
X0: the first part of my ...
- 135
2
votes
1
answer
731
views
Transfer learning for regression problems
I have trained a regression model with 7 features for a given problem.
Now, I have another regression problem (quite similar to the previous one) where I have only 6 samples in hand, but with 3 more ...
- 153
3
votes
2
answers
881
views
Why is the NSP task in BERT inconsistent or ineffective?
The NSP task is one of the two tasks in BERT which has been revolutionizing NLP, but many pretrained models abandoned that task, for instance
First, XLNet removed NSP
XLNet-Large does not use the ...
- 6,840
1
vote
0
answers
412
views
Model retraining: reset state of optimizer or not?
I am retraining a neural net periodically with new datapoints that were collected. The initial training was performed using an Adam optimizer. My question is: when feeding a new batch of data to the ...
- 11
2
votes
1
answer
513
views
How to resume training in neural networks properly?
I'm working on training a network to identify different kinds of cells. For each experimental batch, I would take my previous model weight, and then train a few new pictures on it. Since the model is ...
- 23
2
votes
0
answers
679
views
How is inpainting for self-supervised pre-training of convolutional neural networks usually done?
I read a nice blog post on self-supervised learning and computer vision, which suggests in-painting (amongst other ideas) as a possible self-supervised task for a neural network to "adapt" ...
- 34.2k
1
vote
0
answers
24
views
Pre-training on a subset and after train again with whole
I am implementing a Context Encoder on CelebA dataset, due my hardware limitation the whole dataset takes too long for training the Generator and Discriminator, Can I pre-training my models on a ...
- 11
2
votes
0
answers
30
views
Pre-training without seeing data
Is there a solid reference on pre-training methods in deep neural networks which never see the actual inputs? Any such known thing in literature?
I guess a more correct term is "initialization ...
- 183
4
votes
1
answer
2k
views
what is the scoring variable called for aucpr?
i am trying to conduct a grid search for an imbalanced problem however i cannot find the aucpr (area under curve precision recall) scoring metric for gridsearch. e.g. you have 'roc-auc', 'neg-brier-...
- 579
2
votes
1
answer
44
views
contaminate data with label then take it away
I have an idea for a training strategy (for an ML model), can you please tell me whether it has a name, and whether it makes sense.
I need a model for binary classification with a massive class ...
- 773
4
votes
3
answers
8k
views
What is the difference between position embedding vs positional encoding in BERT?
This post about the Transformer introduced the concept of "Positional Encoding", while at the same time, the BERT paper mentioned "Position Embedding" as an input to BERT (e.g. in Figure 2).
First, ...
- 141
8
votes
1
answer
2k
views
Is initializing the weights of autoencoders still a difficult problem?
I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it.
I have been reading different articles. In one of Hinton's papers (2006)...
- 211
0
votes
1
answer
840
views
Pre-trained shallow CNN with Imagenet
For my research I need some pre-trained model shallower than VGG-16.
Resnet, facenet won't be useful. I found VGG-11 but is there anything else?
What are some pre-trained shallow CNN based on image ...
- 13
0
votes
0
answers
31
views
How to train the deep neural network efficiently when the input data are unstructured
Background
So the background is that I want to use a deep neural network to model a system. In a traditional way to observe the system character, we will use the Gaussian noise as the inputs of the ...
- 36
1
vote
0
answers
391
views
Would training 1024x1024 images with pretrained resnet (224x224) be appropriate?
I want to use Resnet50 (or 101, or 152..) backbone for a segmentation task. My problem requires a lot of context, hence tiling the large image into 224x224 defeats the purpose.
I was wondering if I ...
- 11
1
vote
1
answer
487
views
Why resolution is not important for pre-trained models
As far as I understand (and even successfully applied in Kaggle competition), it's possible to feed images of any resolution into the pre-trained model (e.g. ResNet34).
But I do not understand, why it ...
- 113
0
votes
0
answers
12
views
Contradiction between accuracy obtained from my pretrained conv base network and pretrained conv base network in Deep Learning with Python
I trained a pretrained convnet model on the cats and dogs dataset and the following are the accuracies obtained:
Freezed Conv Base ~ 90%
Unfreezed Conv Base ~ 96%
However this is in contradiction to ...
0
votes
1
answer
348
views
Summary of Pre-Training a Neural Network with Stacks of RBMs
I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done.
Say you have a basic neural network with a ...
- 133
0
votes
1
answer
504
views
Tips on preparing data for training on neural networks? [closed]
Still feeling a bit new to the world of neural networks. I am working with a CNN model right now (working with Keras), and would like to train it to identify certain types of objects from a dataset. I ...
- 113
2
votes
1
answer
259
views
Pre-trained RNN for removing seasonality like VGG16
In image classification there are Pre-trained networks like VGG16. Are there any such networks for time series operations like removing seasonality?
Edit: I found the following paper TimeNet: Pre-...
- 291
8
votes
3
answers
8k
views
Feature Selection in unbalanced data
I was always taught 3 things:
Training algorithms (rf, trees, etc) don't perform well with unbalanced data.
I should balance data only after performing feature selection (mainly to keep variables ...
- 453
3
votes
1
answer
157
views
Rationale for different activation function neural network pretraining vs. supervised training?
I was reading a paper that used neural networks to predict protein conformation, Improving prediction of secondary
structure, local backbone angles,
and solvent accessible surface
area of proteins by ...
- 457
3
votes
0
answers
83
views
State of the Art Status of Deep Boltzmann Machine and Pretraining
I have been reading some old papers by Hinton on deep Boltzmann machine and deep belief networks, but I wonder what the current status is regarding these models:
Are DBM and DBN totally outdated? I ...
- 2,063
3
votes
2
answers
2k
views
Data Augmentation and Balancing Dataset in a context of Object Detection
I have a dataset of object detection (bounding box + class) with 2 classes (excluding "background" class).
I am worried about two things :
First, my dataset counts only 196 samples (I am not too ...
- 163
7
votes
0
answers
8k
views
Change image input size of a pre-trained convnet
maybe this question will sound a bit as a newbie one but I'd like to have some clarification.
I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
- 171
3
votes
2
answers
406
views
Difference between re-training initial layers and final layers of NN
I was going through a brief tutorial on "transfer learning" available here.
In this blog, a distinction has been made between training the initial layers and training the dense layers of a ...
- 1,956
1
vote
1
answer
1k
views
LDA as the dimension reduction before or after partitioning
I am doing a classification and I have this question about using linear discriminant analysis (LDA) just for dimension reduction:
Shall the LDA be applied on whole feature matrix including train and ...
- 133
3
votes
1
answer
138
views
Visualizing model trajectories for Neural Networks using function approximator
Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15
In there, they compare different neural network ...
- 657
3
votes
1
answer
48
views
Using pretrained segmantation network for unseen motives
For a research project, I need to do a segmentation on images. Since the motivation is nothing any of the big networks was ever trained on, I would ask if it still makes sense to use pretrained ...
- 705
2
votes
1
answer
98
views
Is it possible to improve training error by removing features in a GBT?
The case for test data is clearly explained in many places, but I'm just thinking about training error here. I believe it's impossible in a decision tree, and possible (though unlikely) in a random ...
- 41
1
vote
0
answers
315
views
Retraining CNN : Classification to Regression
I am currently working on CNNs and would like to create a Regression model. However I have a relatively small amount of new pictures so I was thinking about retraining.
Is it possible to retrain a ...
- 31
2
votes
1
answer
7k
views
CNN: ReTraining and Fine Tuning
I am getting into CNNs for several months already, and I am currently wondering myself if there is a difference between Retraining a DCNN and Fine Tuning it ?
I am working on a project in which my ...
- 31
1
vote
0
answers
123
views
Is there any place for DBN / Wake-Sleep / Up-Down / Pretraining in modern ML?
I've been looking at DBNs:
first a greedy (unsupervised) layerwise pretraining.
now split weights into recognition R and generative G, and apply Wake-Sleep (again unsupervised, i.e. Unlabelled Data)
...
- 131
1
vote
0
answers
62
views
Does it make sense to use auto-encoders to reconstruct GIST features?
I am trying to extract good low dimensional representation of CIFAR-10 images in an unsupervised way. It is a project requirement that I use 512-d GIST features, reduce the dimensions to 32 using PCA ...
- 11
5
votes
2
answers
974
views
Why does pre-training help avoid the vanishing gradient problem?
I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
- 747
2
votes
2
answers
4k
views
What If I train with Multiple Copies of same data?
I am training a char-rnn to yield some nice generative output. Can I make it memorize context by using the same copies of data in training, multiple times?
- 1,385
9
votes
1
answer
5k
views
Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?
Is it possible to achieve state of the art results by using back-propagation only (without pre-training) ?
Or is it so that all record breaking approaches use some form of pre-training ?
Is back-...
user70990
0
votes
1
answer
204
views
Second layer pre-training for a stacked autoencoder has to reconstruct a badly scaled data
For my greedy layer-wise pre-training using sparse autoencoder, the first layer training seems to be okay since it can fairly reconstruct my test set.
However, because I use "sparse" autoencoder, the ...
- 1
1
vote
1
answer
342
views
Do weights learned from pre-training using a Denoising Autoencoder need rescaling when using dropout in complete NN?
I have a question related to this question.
I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the weights learned in the pre-training ...
- 151
57
votes
3
answers
62k
views
What is pre training a neural network?
Well the question says it all.
What is meant by "pre training a neural network"? Can someone explain in pure simple English?
I can't seem to find any resources related to it. It would be great if ...
- 1,143
2
votes
0
answers
833
views
Stopping Criteria for Pre-Training using Stacked Autoencoders
In stacked autoencoders during greed layer-wise training of individual autoencoders using gradient descent and backpropagation to minimize the reconstruction error(squared error or cross entropy) what ...
11
votes
3
answers
3k
views
What is pretraining and how do you pretrain a neural network?
I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
- 251