Questions tagged [pre-training]

Unsupervised pre-training initializes a discriminative neural network from one which was trained using an unsupervised criterion, such as a deep belief network or a deep autoencoder.

Filter by
Sorted by
Tagged with
6
votes
1answer
80 views

Is initializing the weights of autoencoders still a difficult problem?

I was wondering if initializing the weights of autoencoders is still difficult and what the most recent strategies are for it. I have been reading different articles. In one of Hinton's papers (2006)...
0
votes
1answer
102 views

Pre-trained shallow CNN with Imagenet

For my research I need some pre-trained model shallower than VGG-16. Resnet, facenet won't be useful. I found VGG-11 but is there anything else? What are some pre-trained shallow CNN based on image ...
0
votes
0answers
15 views

How to train the deep neural network efficiently when the input data are unstructured

Background So the background is that I want to use a deep neural network to model a system. In a traditional way to observe the system character, we will use the Gaussian noise as the inputs of the ...
1
vote
0answers
66 views

Would training 1024x1024 images with pretrained resnet (224x224) be appropriate?

I want to use Resnet50 (or 101, or 152..) backbone for a segmentation task. My problem requires a lot of context, hence tiling the large image into 224x224 defeats the purpose. I was wondering if I ...
1
vote
1answer
37 views

Why resolution is not important for pre-trained models

As far as I understand (and even successfully applied in Kaggle competition), it's possible to feed images of any resolution into the pre-trained model (e.g. ResNet34). But I do not understand, why it ...
0
votes
0answers
12 views

Contradiction between accuracy obtained from my pretrained conv base network and pretrained conv base network in Deep Learning with Python

I trained a pretrained convnet model on the cats and dogs dataset and the following are the accuracies obtained: Freezed Conv Base ~ 90% Unfreezed Conv Base ~ 96% However this is in contradiction to ...
0
votes
1answer
156 views

Summary of Pre-Training a Neural Network with Stacks of RBMs

I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done. Say you have a basic neural network with a ...
0
votes
1answer
334 views

Tips on preparing data for training on neural networks? [closed]

Still feeling a bit new to the world of neural networks. I am working with a CNN model right now (working with Keras), and would like to train it to identify certain types of objects from a dataset. I ...
2
votes
1answer
93 views

Pre-trained RNN for removing seasonality like VGG16

In image classification there are Pre-trained networks like VGG16. Are there any such networks for time series operations like removing seasonality? Edit: I found the following paper TimeNet: Pre-...
2
votes
2answers
632 views

Feature Selection in unbalanced data

I was always taught 3 things: Training algorithms (rf, trees, etc) don't perform well with unbalanced data. I should balance data only after performing feature selection (mainly to keep variables ...
2
votes
1answer
60 views

Rationale for different activation function neural network pretraining vs. supervised training?

I was reading a paper that used neural networks to predict protein conformation, Improving prediction of secondary structure, local backbone angles, and solvent accessible surface area of proteins by ...
3
votes
0answers
70 views

State of the Art Status of Deep Boltzmann Machine and Pretraining

I have been reading some old papers by Hinton on deep Boltzmann machine and deep belief networks, but I wonder what the current status is regarding these models: Are DBM and DBN totally outdated? I ...
1
vote
2answers
734 views

Data Augmentation and Balancing Dataset in a context of Object Detection

I have a dataset of object detection (bounding box + class) with 2 classes (excluding "background" class). I am worried about two things : First, my dataset counts only 196 samples (I am not too ...
6
votes
0answers
6k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
1
vote
0answers
31 views

difference between re-training initial layers and final layers of NN

I was going through a brief tutorial on "transfer learning" available on https://www.analyticsvidhya.com/blog/2017/06/transfer-learning-the-art-of-fine-tuning-a-pre-trained-model/ In this blog, a ...
1
vote
1answer
463 views

LDA as the dimension reduction before or after partitioning

I am doing a classification and I have this question about using linear discriminant analysis (LDA) just for dimension reduction: Shall the LDA be applied on whole feature matrix including train and ...
2
votes
1answer
73 views

Visualizing model trajectories for Neural Networks using function approximator

Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15 In there, they compare different neural network ...
0
votes
1answer
26 views

Is it possible to improve training error by removing features in a GBT?

The case for test data is clearly explained in many places, but I'm just thinking about training error here. I believe it's impossible in a decision tree, and possible (though unlikely) in a random ...
1
vote
0answers
200 views

Retraining CNN : Classification to Regression

I am currently working on CNNs and would like to create a Regression model. However I have a relatively small amount of new pictures so I was thinking about retraining. Is it possible to retrain a ...
1
vote
1answer
2k views

CNN: ReTraining and Fine Tuning

I am getting into CNNs for several months already, and I am currently wondering myself if there is a difference between Retraining a DCNN and Fine Tuning it ? I am working on a project in which my ...
1
vote
0answers
80 views

Is there any place for DBN / Wake-Sleep / Up-Down / Pretraining in modern ML?

I've been looking at DBNs: first a greedy (unsupervised) layerwise pretraining. now split weights into recognition R and generative G, and apply Wake-Sleep (again unsupervised, i.e. Unlabelled Data) ...
1
vote
0answers
54 views

Does it make sense to use auto-encoders to reconstruct GIST features?

I am trying to extract good low dimensional representation of CIFAR-10 images in an unsupervised way. It is a project requirement that I use 512-d GIST features, reduce the dimensions to 32 using PCA ...
3
votes
2answers
569 views

Why does pre-training help avoid the vanishing gradient problem?

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
2
votes
2answers
1k views

What If I train with Multiple Copies of same data?

I am training a char-rnn to yield some nice generative output. Can I make it memorize context by using the same copies of data in training, multiple times?
8
votes
1answer
4k views

Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?

Is it possible to achieve state of the art results by using back-propagation only (without pre-training) ? Or is it so that all record breaking approaches use some form of pre-training ? Is back-...
0
votes
1answer
138 views

Second layer pre-training for a stacked autoencoder has to reconstruct a badly scaled data

For my greedy layer-wise pre-training using sparse autoencoder, the first layer training seems to be okay since it can fairly reconstruct my test set. However, because I use "sparse" autoencoder, the ...
0
votes
1answer
186 views

Does weights learned from pre-training using a Denoising Autoencoder needs rescaling when using dropout in complete NN

I have a question related to this question which is also yet to be answered. I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the ...
22
votes
3answers
24k views

What is pre training a neural network?

Well the question says it all. What is meant by "pre training a neural network"? Can someone explain in pure simple English? I can't seem to find any resources related to it. It would be great if ...
2
votes
0answers
641 views

Stopping Criteria for Pre-Training using Stacked Autoencoders

In stacked autoencoders during greed layer-wise training of individual autoencoders using gradient descent and backpropagation to minimize the reconstruction error(squared error or cross entropy) what ...
10
votes
2answers
2k views

What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
7
votes
1answer
677 views

How to pretrain Convolution filter

I was implementing convolutional neural network, For classification of natural images like face, car, flower etc of about 10 categories. I read(from Andrew NG notes) that pre trained convolutional ...