Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [autoencoders]

Feedforward neural networks trained to reconstruct their own input. Usually one of the hidden layers is a "bottleneck", leading to encoder->decoder interpretation.

1
vote
1answer
47 views

VAE: why we do not sample again after decoding and before reconstruction loss?

In many of the VAE schematics and in the original paper, a sampling step is present after decoding and before the reconstruction loss as shown in the image below. The image comes from Stanford CS321n. ...
1
vote
0answers
28 views

Why aren't auto-encoders also considered generative models?

Auto-encoders (AEs) are composed of an encoder and a decoder (often represented by a neural network). The encoder produces a vector representation $z$ of its input $x$ (e.g. an image). The decoder ...
1
vote
0answers
17 views
+50

Where does the prior distribution $p(z)$ for adversarial autoencoders come from?

I am trying to understand how an adversarial autoencoder works. The discriminator takes as an input the aggregated posterior $q(z)$ generated from the decoder and matches this against the prior ...
2
votes
1answer
20 views

What is the relation between ELBO and SGVB?

Evidence lower bound (ELBO) can be minimised, so that to find the most appropriate approximative distribution of the target distribution, which is equivalent to the maximisation of the corresponding ...
0
votes
1answer
23 views

What is meant by Low-Order combination of features?

I came across a Machine Learning paper that talks about input with low-order combination of features. A statement says: The initial feature is used as the input of the model, and the non-linear ...
1
vote
1answer
8 views

The best number of nodes in bottleneck layer in Autoencoder

I would like to perform dimensionality reduction using autoencoders (similar to PCA) and I am not sure how many components are optimal i.e. what should be the size of the bottleneck layer. I was ...
0
votes
0answers
19 views

Dealing with images of variable resolution in CNN autoencoders

Let's suppose would like to build a CNN autoencoder that would be able to turn greyscale images into coloured ones. The final model should be able to accept images of any resolution. Also, note that ...
0
votes
1answer
19 views

Correct way to calculate MSE for autoencoders with batch-training

Suppose you have a network representing an autoencoder (AE). Let's assume it has 90 inputs/outputs. I want to batch-train it with batches of size 100. I will denote my input with ...
1
vote
0answers
16 views

Can you use VAEs to produce deep word embeddings?

There are many articles about applications of VAE such as image reconstruction, denoising, data compression / augmentation. However, I have not seen an example of embeddings for high dimensional data ...
0
votes
0answers
14 views

How to train the deep neural network efficiently when the input data are unstructured

Background So the background is that I want to use a deep neural network to model a system. In a traditional way to observe the system character, we will use the Gaussian noise as the inputs of the ...
0
votes
0answers
40 views

Convolutional Conditional Variational Autoencoder Implementation

This may be a rather trivial question, but I am somewhat confused. I have been able to implement a convolutional variational autoencoder. I have also been able to implement a conditional variational ...
0
votes
0answers
40 views

Many-hot encoding for Hamming codes using 1D convolutional autoencoder

I am trying a simple experiment where I take 16 numbers [0 to 15] that represent the Hamming codebook, and try to reconstruct it using an autoencoder. Instead of using a one-hot encoding scheme (16 ...
0
votes
0answers
10 views

Autoencoder keeping constant vector as predict in keras [duplicate]

I'm new in keras and deep learning field. In fact, I want to make a dense vector for each document in my data so that i built a simple autoencoder using keras library. The input data are normalized ...
2
votes
1answer
26 views

Do fully connected layers in the middle of a network impede optimization?

I submitted a paper that uses an auto-encoder network with several convolutional layers in both the encoder and the decoder and a fully connected layer (FCL) in between. Besides the FCL being useful ...
0
votes
0answers
7 views

Autoencoder - reconstructed image not matching the input image

I have trained a convolutional autoencoder on cifar10 dataset. The reconstruction loss on the test data is quite less (around 0.0225). However, the reconstructed training images do not look like ...
1
vote
0answers
16 views

What autoencoder architectures are effective for large images?

I've had easy success in the past making autoencoders for small images using necked down dense networks, but my new application has images of ~1M pixels, which is impractical to address (I think) with ...
0
votes
0answers
64 views

Reparametrization trick for VAE, prooving that the resulting vector follows a normal distribution

So I've been reading about Variational AutoEncoders and I'm stuck on a little exercise meant to help understand the reparameterization trick. Z is a random vector of K elements with a distribution $q(...
0
votes
0answers
9 views

Efficient way to do Autoencoder on large sparse matrix

I have a large csr_matrix of shape (60,000, 180,000) and about 99.7% sparsity. I was trying to train an autoencoder for this matrix via mini-batch optimization. I tried batch size of 6000 with ...
0
votes
1answer
28 views

Unsupervised VAE model? [closed]

I would like to use VAE model in unsupervised learning to generate new feature. Most of the examples are supervised and semi-supervised learning. Where can I find for unsupervised learning or can it ...
0
votes
1answer
31 views

On evaluating variational autoencoders with prior likelihood and reconstruction error

A common evaluation metric for variational autoencoders (VAEs) is estimating the marginal likelihood of some held-out data, i.e. $p(x)$. This is difficult and often one can only get a lower bound. It'...
0
votes
0answers
12 views

faulty autoencoder [duplicate]

I am developing an autoencoder for CIFA10 dataset, without adding noise at the input (which is 2nd goal). The Convnet based autoencoder is not converging: Any suggestions ...
1
vote
0answers
52 views

training prior network on conditional variational autoencoder

The objective function of the conditional VAE is defined as: $L_{CVAE} = -KL(q(z|x,y)||p(z|x)) + \frac{1}{L}\sum_{l=1}^{L}\text{log } p(y|x,z^{(l)})$ Here x is input; y is output; z is latent ...
0
votes
1answer
42 views
0
votes
0answers
17 views

Autoencoder for sparse data

Suppose I have a big (1,000x20,000) sparse (95% of elements are zeros) matrix with counts. I want to use autoencoder to encode-decode this matrix. How should I do it? Are there any tricks or ...
0
votes
0answers
10 views

How to calculate precision-recall curve for multiple binary classifier for multiple classes

I am using autoencoder based multiple binary classifier for text regeneration each trained on data related to multiple classes. In other words, each classifier is trained to classify only one class. I ...
1
vote
0answers
19 views

Training Variational Autoencoders in two steps

I started to use Variational Autoencoder in a project, (and I have a hard time determining the weight for the reconstruction loss and KL-loss). I have an idea of training the VAE in two steps.: Train ...
0
votes
0answers
31 views

how to prepare text data for LSTM autoencoder

My main goal is to come up with some topics using LSTM autoencoder. I want to use 20 news_group data set. after reading lots of material and looking at some GitHub project, I am still not clear how ...
2
votes
3answers
207 views

Why is binary cross entropy (or log loss) used in autoencoders for non-binary data

I am working on an autoencoder for non-binary data ranging in [0,1] and while I was exploring existing solutions I noticed that in many people (e.g., the keras ...
2
votes
1answer
436 views

KL Divergence loss in variational autoencoders

I was studying VAEs and came accross the Loss function that consists of KL Divergence. I wanted to intuitively make sense of the KL divergence part of the loss function. It would be great if somebody ...
0
votes
1answer
381 views

How to achieve variational autoencoder (VAE) with unrestricted input?

For a normal VAE an input and a reconstruction with values in the range of $[0, 1]$ are expected. This is necessary since the log loss only makes sense for this range. If the input is not within $[0, ...
2
votes
1answer
53 views

Are VAE used for missing data imputation in multivariate time series? If not, what is used?

Multivariate time series are, to the best of my understanding, one of the few cases where Deep Learning still hasn't had its AlexNet moment. I'm especially interested to the case where most of the ...
0
votes
2answers
116 views

Stacked shallow autoencoders vs. deep autoencoders

In LeCun et. all "Deep Learning", Chapter 14, page 506, I found the following statement: "A common strategy for training a deep autoencoder is to greedily pretrain the deep architecture by training a ...
0
votes
0answers
108 views

rewriting ELBO to highlight the role of priors

I am reading this paper which rewrites ELBO. I am stuck in verifying the mathematics used for doing the rewriting. Essentially, the paper writes the KL term involved in ELBO as follows (equations 13 ...
0
votes
1answer
165 views

advantage of variational autoencoder

I know that VAE is generative model. However it is also used as a dimensionality reduction method. In this case, what are advantages of VAE?? Also I saw that well-applied vae on mnist, and it was ...
1
vote
1answer
76 views

Removing noise with Variational Autoencoders

I have one question that is related to variational autoencoders: can they be used as a denosing algorithm in the same way as standard denosing autoencoders? I generally see people removing the ...
2
votes
1answer
173 views

Where to include Dropout in stacked autoencoder

I'm using Keras to implement a stacked autoencoder, and I think it may be overfitting. I wanted to include dropout, and keep reading about the use of dropout in autoencoders, but I cannot find any ...
2
votes
1answer
146 views

Image feature extraction using an Autoencoder combined with PCA

Background: I have fairly large dataset of biomedical images (around 10,000 images) of 1920x1920 pixels (after cropping parts of black borders out). My task is to extract the 200 most important ...
0
votes
1answer
41 views

How to reconstruct negative acceleration values using a simple autoencoder?

I am trying to reconstruct the acceleration values of a tri-axial accelerometer using a simple autoencoder. As acceleration values are often negative (e.g, -3.4) therefore using a ReLU activation ...
5
votes
1answer
141 views

Writing PCA as a special kind of auto-encoder

I would like to know if it's possible to view PCA as a particular type of neural network, however, there's one major stumbling block that I haven't yet been able to get past. Define the following "...
1
vote
0answers
20 views

Constraints on low dimensional representations of data

Is there literature discussing introduction of constraints to loss functions in order to specify certain structures on low dimensional representations? If so, how do they compare the efficacy of the ...
0
votes
1answer
73 views

Variational Autoencoder and Covariance Matrix

Why does the encoder from a variational autoencoder map to a vector of means and a vector of standard deviations? Why does it not instead map to a vector of means and a covariance matrix? Is it ...
0
votes
1answer
36 views

Generative model to generate hidden activations coming from a previously trained hidden layer

I need to train a generative model to generate vectors which resemble the activations of a particular hidden layer of a neural network which has been previously trained. In particular, the hidden ...
1
vote
1answer
43 views

What kind of impact do autoencoders have on final model performance when compared to models trained only on supervised data? [closed]

For example, say we have two datasets, a labeled set (I will call it df_labeled) of nrows=200k and an unlabeled dataset (df_unlabeled) of nrows=800k and we want to build a binary classifier. I clearly,...
0
votes
0answers
36 views

What is the best input for denoise autoencoder for sound/audio data?

I am currently trying to build an autoencoder to de-noise audio data. However I have not found any good articles explaining about the input to the autoencoder, i.e. feature vector. As in speech ...
0
votes
1answer
142 views

When does my unsupervised autoencoder start to overfit?

I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has ...
0
votes
0answers
34 views

What's the difference between random and deterministic encoder in autoencoders?

I read this paper "Wasserstein Auto-Encoders", and they mention deterministic encoder and random encoder but without stating the difference between them. How can we tell the difference?
0
votes
0answers
128 views

Autoencoder as an optimization (search) problem

We all know that machine learning problems can be modeled as an optimization problem where we are searching for the best set of parameter values in the parameter space that optimizes our objective ...
0
votes
1answer
169 views

Help in Understanding Variational Autoencoder Size of Latent Variables

I'm trying to understand further how a variational autoencoder works beyond the conceptual level. However I'm still confused as to what the "vector of mean and variances" can look like and to digest ...
1
vote
0answers
47 views

Equity Risk Model using an autoencoder

I am trying to create a statistical equity risk model using an autoencoder in a similar fashion to how one would use PCA to derive the systematic and specific risk components of a stock's returns. I ...
0
votes
1answer
128 views

Minimum Input Dimension for Autoencoder Neural Network

Model: Assume we want to learn patterns using an autoencoder neural network. In the simplest case, such a network is "shallow" with 1 hidden layer, takes a $d$-dimensional numerical input vector $x$, ...