Questions tagged [pre-training]

Unsupervised pre-training initializes a discriminative neural network from one which was trained using an unsupervised criterion, such as a deep belief network or a deep autoencoder.

Filter by
Sorted by
Tagged with
1 vote
0 answers
10 views

References for BERT training from scratch

As the title of the question suggests, I'm interested in training a BERT-like model (and then use it to make some experiments on text-similarity). Question: Could you share some references on the ...
user avatar
0 votes
0 answers
75 views

Where can I find pre-trained Pathways Language Model (PaLM)?

I am amazed by the possibilities coming from the PaLM model after reading the following post: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html . I was trying to find the ...
user avatar
  • 121
1 vote
0 answers
9 views

Suggestion regarding usage of pre-trained BERT

I have recently been working with the pre-trained BERT. It produces quite good results on supervised tasks with just a bit fine tuning. But now I wonder if I want to perform some unsupervised task on ...
user avatar
0 votes
0 answers
42 views

How to handle problem of different random seeds giving drastically different test scores in machine learning model?

For a rigorous empirical analysis, I am training a model with three different seeds - 0, 1 and 2. In each case, I found that the model obtained through early stopping (lowest validation loss) had an ...
user avatar
0 votes
0 answers
7 views

Methods for selecting best data points for training language models

I have a neural network model (a language model) that is already pretrained using a huge amount of data (think of BERT, for example.) I added some domain-related data (~120,000 extra text sentences) ...
user avatar
  • 133
0 votes
0 answers
11 views

Does model on MLM task train context too - or only target?

Assume a masked language model like BERT. Further assume I have two sentences The cat eats food and ...
user avatar
1 vote
0 answers
20 views

Best way to train a model having train/test/val set

I'm new in the machine learning field and I'm trying to understand what are the best practices to manipulate the time series. I currently have a dataset structured as well: X0: the first part of my ...
user avatar
  • 115
2 votes
1 answer
76 views

Transfer learning for regression problems

I have trained a regression model with 7 features for a given problem. Now, I have another regression problem (quite similar to the previous one) where I have only 6 samples in hand, but with 3 more ...
user avatar
  • 153
0 votes
2 answers
92 views

Why is the NSP task in BERT inconsistent or ineffective?

The NSP task is one of the two tasks in BERT which has been revolutionizing NLP, but many pretrained models abandoned that task, for instance First, XLNet removed NSP XLNet-Large does not use the ...
user avatar
  • 5,296
0 votes
0 answers
5 views

Intrinsic metrics to evaluate pretrained language models?

I learned that there are intrinsic and extrinsic evaluation methods for vector models. Although the most important evaluation is the extrinsic the intrinsic metrics are also useful. There are three ...
user avatar
  • 5,296
1 vote
0 answers
90 views

Model retraining: reset state of optimizer or not?

I am retraining a neural net periodically with new datapoints that were collected. The initial training was performed using an Adam optimizer. My question is: when feeding a new batch of data to the ...
user avatar
  • 11
2 votes
1 answer
170 views

How to resume training in neural networks properly?

I'm working on training a network to identify different kinds of cells. For each experimental batch, I would take my previous model weight, and then train a few new pictures on it. Since the model is ...
user avatar
  • 23
1 vote
0 answers
292 views

How is inpainting for self-supervised pre-training of convolutional neural networks usually done?

I read a nice blog post on self-supervised learning and computer vision, which suggests in-painting (amongst other ideas) as a possible self-supervised task for a neural network to "adapt" ...
user avatar
  • 23k
1 vote
0 answers
16 views

Pre-training on a subset and after train again with whole

I am implementing a Context Encoder on CelebA dataset, due my hardware limitation the whole dataset takes too long for training the Generator and Discriminator, Can I pre-training my models on a ...
user avatar
  • 11
2 votes
0 answers
26 views

Pre-training without seeing data

Is there a solid reference on pre-training methods in deep neural networks which never see the actual inputs? Any such known thing in literature? I guess a more correct term is "initialization ...
user avatar
2 votes
1 answer
1k views

what is the scoring variable called for aucpr?

i am trying to conduct a grid search for an imbalanced problem however i cannot find the aucpr (area under curve precision recall) scoring metric for gridsearch. e.g. you have 'roc-auc', 'neg-brier-...
user avatar
  • 319
2 votes
1 answer
37 views

contaminate data with label then take it away

I have an idea for a training strategy (for an ML model), can you please tell me whether it has a name, and whether it makes sense. I need a model for binary classification with a massive class ...
user avatar
  • 336
1 vote
3 answers
2k views

What is the difference between position embedding vs positional encoding in BERT?

This post about the Transformer introduced the concept of "Positional Encoding", while at the same time, the BERT paper mentioned "Position Embedding" as an input to BERT (e.g. in Figure 2). First, ...
user avatar
8 votes
1 answer
890 views

Is initializing the weights of autoencoders still a difficult problem?

I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it. I have been reading different articles. In one of Hinton's papers (2006)...
user avatar
  • 211
0 votes
1 answer
713 views

Pre-trained shallow CNN with Imagenet

For my research I need some pre-trained model shallower than VGG-16. Resnet, facenet won't be useful. I found VGG-11 but is there anything else? What are some pre-trained shallow CNN based on image ...
user avatar
0 votes
0 answers
21 views

How to train the deep neural network efficiently when the input data are unstructured

Background So the background is that I want to use a deep neural network to model a system. In a traditional way to observe the system character, we will use the Gaussian noise as the inputs of the ...
user avatar
1 vote
0 answers
215 views

Would training 1024x1024 images with pretrained resnet (224x224) be appropriate?

I want to use Resnet50 (or 101, or 152..) backbone for a segmentation task. My problem requires a lot of context, hence tiling the large image into 224x224 defeats the purpose. I was wondering if I ...
user avatar
1 vote
1 answer
244 views

Why resolution is not important for pre-trained models

As far as I understand (and even successfully applied in Kaggle competition), it's possible to feed images of any resolution into the pre-trained model (e.g. ResNet34). But I do not understand, why it ...
user avatar
  • 113
0 votes
0 answers
12 views

Contradiction between accuracy obtained from my pretrained conv base network and pretrained conv base network in Deep Learning with Python

I trained a pretrained convnet model on the cats and dogs dataset and the following are the accuracies obtained: Freezed Conv Base ~ 90% Unfreezed Conv Base ~ 96% However this is in contradiction to ...
user avatar
0 votes
1 answer
277 views

Summary of Pre-Training a Neural Network with Stacks of RBMs

I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done. Say you have a basic neural network with a ...
user avatar
  • 133
0 votes
1 answer
474 views

Tips on preparing data for training on neural networks? [closed]

Still feeling a bit new to the world of neural networks. I am working with a CNN model right now (working with Keras), and would like to train it to identify certain types of objects from a dataset. I ...
user avatar
2 votes
1 answer
206 views

Pre-trained RNN for removing seasonality like VGG16

In image classification there are Pre-trained networks like VGG16. Are there any such networks for time series operations like removing seasonality? Edit: I found the following paper TimeNet: Pre-...
user avatar
  • 281
7 votes
3 answers
4k views

Feature Selection in unbalanced data

I was always taught 3 things: Training algorithms (rf, trees, etc) don't perform well with unbalanced data. I should balance data only after performing feature selection (mainly to keep variables ...
user avatar
3 votes
1 answer
130 views

Rationale for different activation function neural network pretraining vs. supervised training?

I was reading a paper that used neural networks to predict protein conformation, Improving prediction of secondary structure, local backbone angles, and solvent accessible surface area of proteins by ...
user avatar
  • 407
3 votes
0 answers
78 views

State of the Art Status of Deep Boltzmann Machine and Pretraining

I have been reading some old papers by Hinton on deep Boltzmann machine and deep belief networks, but I wonder what the current status is regarding these models: Are DBM and DBN totally outdated? I ...
user avatar
  • 1,663
2 votes
2 answers
1k views

Data Augmentation and Balancing Dataset in a context of Object Detection

I have a dataset of object detection (bounding box + class) with 2 classes (excluding "background" class). I am worried about two things : First, my dataset counts only 196 samples (I am not too ...
user avatar
  • 153
7 votes
0 answers
7k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
user avatar
  • 171
2 votes
2 answers
66 views

Difference between re-training initial layers and final layers of NN

I was going through a brief tutorial on "transfer learning" available here. In this blog, a distinction has been made between training the initial layers and training the dense layers of a ...
user avatar
  • 1,586
1 vote
1 answer
724 views

LDA as the dimension reduction before or after partitioning

I am doing a classification and I have this question about using linear discriminant analysis (LDA) just for dimension reduction: Shall the LDA be applied on whole feature matrix including train and ...
user avatar
  • 133
3 votes
1 answer
103 views

Visualizing model trajectories for Neural Networks using function approximator

Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15 In there, they compare different neural network ...
user avatar
3 votes
1 answer
46 views

Using pretrained segmantation network for unseen motives

For a research project, I need to do a segmentation on images. Since the motivation is nothing any of the big networks was ever trained on, I would ask if it still makes sense to use pretrained ...
user avatar
0 votes
1 answer
62 views

Is it possible to improve training error by removing features in a GBT?

The case for test data is clearly explained in many places, but I'm just thinking about training error here. I believe it's impossible in a decision tree, and possible (though unlikely) in a random ...
user avatar
1 vote
0 answers
280 views

Retraining CNN : Classification to Regression

I am currently working on CNNs and would like to create a Regression model. However I have a relatively small amount of new pictures so I was thinking about retraining. Is it possible to retrain a ...
user avatar
  • 21
1 vote
1 answer
6k views

CNN: ReTraining and Fine Tuning

I am getting into CNNs for several months already, and I am currently wondering myself if there is a difference between Retraining a DCNN and Fine Tuning it ? I am working on a project in which my ...
user avatar
  • 21
1 vote
0 answers
99 views

Is there any place for DBN / Wake-Sleep / Up-Down / Pretraining in modern ML?

I've been looking at DBNs: first a greedy (unsupervised) layerwise pretraining. now split weights into recognition R and generative G, and apply Wake-Sleep (again unsupervised, i.e. Unlabelled Data) ...
user avatar
  • 121
1 vote
0 answers
59 views

Does it make sense to use auto-encoders to reconstruct GIST features?

I am trying to extract good low dimensional representation of CIFAR-10 images in an unsupervised way. It is a project requirement that I use 512-d GIST features, reduce the dimensions to 32 using PCA ...
user avatar
5 votes
2 answers
840 views

Why does pre-training help avoid the vanishing gradient problem?

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
user avatar
2 votes
2 answers
3k views

What If I train with Multiple Copies of same data?

I am training a char-rnn to yield some nice generative output. Can I make it memorize context by using the same copies of data in training, multiple times?
user avatar
  • 1,109
9 votes
1 answer
5k views

Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?

Is it possible to achieve state of the art results by using back-propagation only (without pre-training) ? Or is it so that all record breaking approaches use some form of pre-training ? Is back-...
user avatar
0 votes
1 answer
175 views

Second layer pre-training for a stacked autoencoder has to reconstruct a badly scaled data

For my greedy layer-wise pre-training using sparse autoencoder, the first layer training seems to be okay since it can fairly reconstruct my test set. However, because I use "sparse" autoencoder, the ...
user avatar
0 votes
1 answer
286 views

Does weights learned from pre-training using a Denoising Autoencoder needs rescaling when using dropout in complete NN

I have a question related to this question which is also yet to be answered. I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the ...
user avatar
  • 141
44 votes
3 answers
48k views

What is pre training a neural network?

Well the question says it all. What is meant by "pre training a neural network"? Can someone explain in pure simple English? I can't seem to find any resources related to it. It would be great if ...
user avatar
2 votes
0 answers
751 views

Stopping Criteria for Pre-Training using Stacked Autoencoders

In stacked autoencoders during greed layer-wise training of individual autoencoders using gradient descent and backpropagation to minimize the reconstruction error(squared error or cross entropy) what ...
user avatar
11 votes
3 answers
3k views

What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
user avatar
7 votes
1 answer
779 views

How to pretrain Convolution filter

I was implementing convolutional neural network, For classification of natural images like face, car, flower etc of about 10 categories. I read(from Andrew NG notes) that pre trained convolutional ...
user avatar
  • 173