Questions tagged [pre-training]

Unsupervised pre-training initializes a discriminative neural network from one which was trained using an unsupervised criterion, such as a deep belief network or a deep autoencoder.

Filter by
Sorted by
Tagged with
2
votes
0answers
18 views

Pre-training without seeing data

Is there a solid reference on pre-training methods in deep neural networks which never see the actual inputs? Any such known thing in literature? I guess a more correct term is "initialization ...
0
votes
0answers
6 views

How to properly retrain a model with quantization aware training

I am trying to tune a model via quantization aware training (QAT). The model is from rcmalli. It is a ResNet50 architecture. The model was trained by them on the vggface2 dataset. I use the model to ...
1
vote
1answer
41 views

what is the scoring variable called for aucpr?

i am trying to conduct a grid search for an imbalanced problem however i cannot find the aucpr (area under curve precision recall) scoring metric for gridsearch. e.g. you have 'roc-auc', 'neg-brier-...
0
votes
0answers
8 views

Positive cases without variable data

I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
1
vote
1answer
30 views

contaminate data with label then take it away

I have an idea for a training strategy (for an ML model), can you please tell me whether it has a name, and whether it makes sense. I need a model for binary classification with a massive class ...
1
vote
0answers
63 views

What is the difference between position embedding vs positional encoding in BERT?

This post about the Transformer introduced the concept of "Positional Encoding", while at the same time, the BERT paper mentioned "Position Embedding" as an input to BERT (e.g. in Figure 2). First, ...
0
votes
0answers
18 views

How can I train SRGAN with Synthetic Aparture Radar grayscale images?

I have been reading and looking at implementations of the SRGAN, from Photo-realistic Single Image Super Resolution with Generative Adversarial Networks. I implemented the PyTorch implementation of ...
8
votes
1answer
280 views

Is initializing the weights of autoencoders still a difficult problem?

I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it. I have been reading different articles. In one of Hinton's papers (2006)...
0
votes
1answer
297 views

Pre-trained shallow CNN with Imagenet

For my research I need some pre-trained model shallower than VGG-16. Resnet, facenet won't be useful. I found VGG-11 but is there anything else? What are some pre-trained shallow CNN based on image ...
0
votes
0answers
17 views

How to train the deep neural network efficiently when the input data are unstructured

Background So the background is that I want to use a deep neural network to model a system. In a traditional way to observe the system character, we will use the Gaussian noise as the inputs of the ...
1
vote
0answers
107 views

Would training 1024x1024 images with pretrained resnet (224x224) be appropriate?

I want to use Resnet50 (or 101, or 152..) backbone for a segmentation task. My problem requires a lot of context, hence tiling the large image into 224x224 defeats the purpose. I was wondering if I ...
1
vote
1answer
55 views

Why resolution is not important for pre-trained models

As far as I understand (and even successfully applied in Kaggle competition), it's possible to feed images of any resolution into the pre-trained model (e.g. ResNet34). But I do not understand, why it ...
0
votes
0answers
12 views

Contradiction between accuracy obtained from my pretrained conv base network and pretrained conv base network in Deep Learning with Python

I trained a pretrained convnet model on the cats and dogs dataset and the following are the accuracies obtained: Freezed Conv Base ~ 90% Unfreezed Conv Base ~ 96% However this is in contradiction to ...
0
votes
1answer
201 views

Summary of Pre-Training a Neural Network with Stacks of RBMs

I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done. Say you have a basic neural network with a ...
0
votes
1answer
398 views

Tips on preparing data for training on neural networks? [closed]

Still feeling a bit new to the world of neural networks. I am working with a CNN model right now (working with Keras), and would like to train it to identify certain types of objects from a dataset. I ...
2
votes
1answer
140 views

Pre-trained RNN for removing seasonality like VGG16

In image classification there are Pre-trained networks like VGG16. Are there any such networks for time series operations like removing seasonality? Edit: I found the following paper TimeNet: Pre-...
3
votes
2answers
1k views

Feature Selection in unbalanced data

I was always taught 3 things: Training algorithms (rf, trees, etc) don't perform well with unbalanced data. I should balance data only after performing feature selection (mainly to keep variables ...
2
votes
1answer
71 views

Rationale for different activation function neural network pretraining vs. supervised training?

I was reading a paper that used neural networks to predict protein conformation, Improving prediction of secondary structure, local backbone angles, and solvent accessible surface area of proteins by ...
3
votes
0answers
74 views

State of the Art Status of Deep Boltzmann Machine and Pretraining

I have been reading some old papers by Hinton on deep Boltzmann machine and deep belief networks, but I wonder what the current status is regarding these models: Are DBM and DBN totally outdated? I ...
2
votes
2answers
997 views

Data Augmentation and Balancing Dataset in a context of Object Detection

I have a dataset of object detection (bounding box + class) with 2 classes (excluding "background" class). I am worried about two things : First, my dataset counts only 196 samples (I am not too ...
6
votes
0answers
7k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
2
votes
2answers
46 views

Difference between re-training initial layers and final layers of NN

I was going through a brief tutorial on "transfer learning" available here. In this blog, a distinction has been made between training the initial layers and training the dense layers of a ...
1
vote
1answer
535 views

LDA as the dimension reduction before or after partitioning

I am doing a classification and I have this question about using linear discriminant analysis (LDA) just for dimension reduction: Shall the LDA be applied on whole feature matrix including train and ...
3
votes
1answer
90 views

Visualizing model trajectories for Neural Networks using function approximator

Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15 In there, they compare different neural network ...
0
votes
1answer
30 views

Is it possible to improve training error by removing features in a GBT?

The case for test data is clearly explained in many places, but I'm just thinking about training error here. I believe it's impossible in a decision tree, and possible (though unlikely) in a random ...
1
vote
0answers
223 views

Retraining CNN : Classification to Regression

I am currently working on CNNs and would like to create a Regression model. However I have a relatively small amount of new pictures so I was thinking about retraining. Is it possible to retrain a ...
1
vote
1answer
3k views

CNN: ReTraining and Fine Tuning

I am getting into CNNs for several months already, and I am currently wondering myself if there is a difference between Retraining a DCNN and Fine Tuning it ? I am working on a project in which my ...
1
vote
0answers
85 views

Is there any place for DBN / Wake-Sleep / Up-Down / Pretraining in modern ML?

I've been looking at DBNs: first a greedy (unsupervised) layerwise pretraining. now split weights into recognition R and generative G, and apply Wake-Sleep (again unsupervised, i.e. Unlabelled Data) ...
1
vote
0answers
57 views

Does it make sense to use auto-encoders to reconstruct GIST features?

I am trying to extract good low dimensional representation of CIFAR-10 images in an unsupervised way. It is a project requirement that I use 512-d GIST features, reduce the dimensions to 32 using PCA ...
4
votes
2answers
684 views

Why does pre-training help avoid the vanishing gradient problem?

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
2
votes
2answers
1k views

What If I train with Multiple Copies of same data?

I am training a char-rnn to yield some nice generative output. Can I make it memorize context by using the same copies of data in training, multiple times?
8
votes
1answer
5k views

Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?

Is it possible to achieve state of the art results by using back-propagation only (without pre-training) ? Or is it so that all record breaking approaches use some form of pre-training ? Is back-...
0
votes
1answer
147 views

Second layer pre-training for a stacked autoencoder has to reconstruct a badly scaled data

For my greedy layer-wise pre-training using sparse autoencoder, the first layer training seems to be okay since it can fairly reconstruct my test set. However, because I use "sparse" autoencoder, the ...
0
votes
1answer
218 views

Does weights learned from pre-training using a Denoising Autoencoder needs rescaling when using dropout in complete NN

I have a question related to this question which is also yet to be answered. I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the ...
27
votes
3answers
30k views

What is pre training a neural network?

Well the question says it all. What is meant by "pre training a neural network"? Can someone explain in pure simple English? I can't seem to find any resources related to it. It would be great if ...
2
votes
0answers
690 views

Stopping Criteria for Pre-Training using Stacked Autoencoders

In stacked autoencoders during greed layer-wise training of individual autoencoders using gradient descent and backpropagation to minimize the reconstruction error(squared error or cross entropy) what ...
11
votes
3answers
2k views

What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
7
votes
1answer
708 views

How to pretrain Convolution filter

I was implementing convolutional neural network, For classification of natural images like face, car, flower etc of about 10 categories. I read(from Andrew NG notes) that pre trained convolutional ...