Questions tagged [pre-training]

Unsupervised pre-training initializes a discriminative neural network from one which was trained using an unsupervised criterion, such as a deep belief network or a deep autoencoder.

0
votes
0answers
14 views

How to train the deep neural network efficiently when the input data are unstructured

Background So the background is that I want to use a deep neural network to model a system. In a traditional way to observe the system character, we will use the Gaussian noise as the inputs of the ...
0
votes
0answers
25 views

Would training 1024x1024 images with pretrained resnet (224x224) be appropriate?

I want to use Resnet50 (or 101, or 152..) backbone for a segmentation task. My problem requires a lot of context, hence tiling the large image into 224x224 defeats the purpose. I was wondering if I ...
1
vote
1answer
24 views

Why resolution is not important for pre-trained models

As far as I understand (and even successfully applied in Kaggle competition), it's possible to feed images of any resolution into the pre-trained model (e.g. ResNet34). But I do not understand, why it ...
0
votes
0answers
11 views

Contradiction between accuracy obtained from my pretrained conv base network and pretrained conv base network in Deep Learning with Python

I trained a pretrained convnet model on the cats and dogs dataset and the following are the accuracies obtained: Freezed Conv Base ~ 90% Unfreezed Conv Base ~ 96% However this is in contradiction to ...
0
votes
1answer
116 views

Summary of Pre-Training a Neural Network with Stacks of RBMs

I understand that pre-training with stacks of RBMs is now (mostly) obsolete but I'm still interested in knowing if I have the right idea on how it is done. Say you have a basic neural network with a ...
0
votes
1answer
232 views

Tips on preparing data for training on neural networks? [closed]

Still feeling a bit new to the world of neural networks. I am working with a CNN model right now (working with Keras), and would like to train it to identify certain types of objects from a dataset. I ...
2
votes
1answer
66 views

Pre-trained RNN for removing seasonality like VGG16

In image classification there are Pre-trained networks like VGG16. Are there any such networks for time series operations like removing seasonality? Edit: I found the following paper TimeNet: Pre-...
2
votes
1answer
269 views

Feature Selection in unbalanced data

I was always taught 3 things: Training algorithms (rf, trees, etc) don't perform well with unbalanced data. I should balance data only after performing feature selection (mainly to keep variables ...
2
votes
1answer
52 views

Rationale for different activation function neural network pretraining vs. supervised training?

I was reading a paper that used neural networks to predict protein conformation, Improving prediction of secondary structure, local backbone angles, and solvent accessible surface area of proteins by ...
3
votes
0answers
68 views

State of the Art Status of Deep Boltzmann Machine and Pretraining

I have been reading some old papers by Hinton on deep Boltzmann machine and deep belief networks, but I wonder what the current status is regarding these models: Are DBM and DBN totally outdated? I ...
1
vote
2answers
572 views

Data Augmentation and Balancing Dataset in a context of Object Detection

I have a dataset of object detection (bounding box + class) with 2 classes (excluding "background" class). I am worried about two things : First, my dataset counts only 196 samples (I am not too ...
5
votes
0answers
5k views

Change image input size of a pre-trained convnet

maybe this question will sound a bit as a newbie one but I'd like to have some clarification. I'm using a VGG16-like convnet, pre-trained with VGG16 weights and edited top layers to work with my ...
1
vote
0answers
27 views

difference between re-training initial layers and final layers of NN

I was going through a brief tutorial on "transfer learning" available on https://www.analyticsvidhya.com/blog/2017/06/transfer-learning-the-art-of-fine-tuning-a-pre-trained-model/ In this blog, a ...
1
vote
1answer
393 views

LDA as the dimension reduction before or after partitioning

I am doing a classification and I have this question about using linear discriminant analysis (LDA) just for dimension reduction: Shall the LDA be applied on whole feature matrix including train and ...
2
votes
1answer
67 views

Visualizing model trajectories for Neural Networks using function approximator

Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15 In there, they compare different neural network ...
0
votes
1answer
24 views

Is it possible to improve training error by removing features in a GBT?

The case for test data is clearly explained in many places, but I'm just thinking about training error here. I believe it's impossible in a decision tree, and possible (though unlikely) in a random ...
1
vote
0answers
161 views

Retraining CNN : Classification to Regression

I am currently working on CNNs and would like to create a Regression model. However I have a relatively small amount of new pictures so I was thinking about retraining. Is it possible to retrain a ...
1
vote
1answer
2k views

CNN: ReTraining and Fine Tuning

I am getting into CNNs for several months already, and I am currently wondering myself if there is a difference between Retraining a DCNN and Fine Tuning it ? I am working on a project in which my ...
1
vote
0answers
66 views

Is there any place for DBN / Wake-Sleep / Up-Down / Pretraining in modern ML?

I've been looking at DBNs: first a greedy (unsupervised) layerwise pretraining. now split weights into recognition R and generative G, and apply Wake-Sleep (again unsupervised, i.e. Unlabelled Data) ...
1
vote
0answers
52 views

Does it make sense to use auto-encoders to reconstruct GIST features?

I am trying to extract good low dimensional representation of CIFAR-10 images in an unsupervised way. It is a project requirement that I use 512-d GIST features, reduce the dimensions to 32 using PCA ...
3
votes
2answers
481 views

Why does pre-training help avoid the vanishing gradient problem?

I read that a problem with the Classic approach to deep NN is the vanishing gradient, which is caused by the derivative of the logistic activation function - broadly speaking, the update flowing down ...
2
votes
2answers
873 views

What If I train with Multiple Copies of same data?

I am training a char-rnn to yield some nice generative output. Can I make it memorize context by using the same copies of data in training, multiple times?
8
votes
1answer
4k views

Is Greedy Layer-Wise Training of Deep Networks necessary for successfully training or is stochastic gradient descent enough?

Is it possible to achieve state of the art results by using back-propagation only (without pre-training) ? Or is it so that all record breaking approaches use some form of pre-training ? Is back-...
0
votes
1answer
121 views

Second layer pre-training for a stacked autoencoder has to reconstruct a badly scaled data

For my greedy layer-wise pre-training using sparse autoencoder, the first layer training seems to be okay since it can fairly reconstruct my test set. However, because I use "sparse" autoencoder, the ...
0
votes
1answer
159 views

Does weights learned from pre-training using a Denoising Autoencoder needs rescaling when using dropout in complete NN

I have a question related to this question which is also yet to be answered. I am using a denoising autoencoder for pre-training of a neural network for dimensionality reduction. I want to use the ...
21
votes
3answers
19k views

What is pre training a neural network?

Well the question says it all. What is meant by "pre training a neural network"? Can someone explain in pure simple English? I can't seem to find any resources related to it. It would be great if ...
2
votes
0answers
586 views

Stopping Criteria for Pre-Training using Stacked Autoencoders

In stacked autoencoders during greed layer-wise training of individual autoencoders using gradient descent and backpropagation to minimize the reconstruction error(squared error or cross entropy) what ...
10
votes
2answers
1k views

What is pretraining and how do you pretrain a neural network?

I understand that pretraining is used to avoid some of the issues with conventional training. If I use backpropagation with, say an autoencoder, I know I'm going to run into time issues because ...
7
votes
1answer
645 views

How to pretrain Convolution filter

I was implementing convolutional neural network, For classification of natural images like face, car, flower etc of about 10 categories. I read(from Andrew NG notes) that pre trained convolutional ...