Skip to main content

Questions tagged [autoencoders]

Feedforward neural networks trained to reconstruct their own input. Usually one of the hidden layers is a "bottleneck", leading to encoder->decoder interpretation.

Filter by
Sorted by
Tagged with
0 votes
1 answer
18 views

Exploring vae latent space

I recently trained a AE and a VAE and used the latent variables of each for a clustering task. It seemed to work well, sensible clusters. The main reason for training the VAE was too gain more ...
Nathan Thompo's user avatar
0 votes
0 answers
15 views

difference between l2 penalty and l2 loss in SAE

I was reading this paper from Anthropic https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html and in the paper loss is defined like this :$$ L = \mathbb{E}_x \left[ \| x - \hat{x} \|...
Mrnobody's user avatar
2 votes
0 answers
99 views

Anomaly detection for Multivariate Time-Series data from multiple sensors

I work with tabular time-series data from multiple sensors and my goal is to detect abnormal behavior in battery discharge. Here is an example of data (example contains records only for one device ...
mz2300's user avatar
  • 71
1 vote
0 answers
22 views

Multi-task learning-Loss function

0 I am training a convolutional autoencoder with two velocity fields (2D array) as inputs and outputs. These fields represent wind velocities in both the x and y directions within a square domain. My ...
Sarah's user avatar
  • 11
2 votes
1 answer
46 views

Why is the forward process referred to as the "ground truth" in diffusion models?

I've seen in many tutorials on diffusion models refer to the distribution of the latent variables induced by the forward process as "ground truth". I wonder why. What we can actually see is ...
Daniel Mendoza's user avatar
2 votes
2 answers
64 views

Why does Variational Inference work?

ELBO is a lower bound, and only matches the true likelihood when the q-distribution/encoder we choose equals to the true posterior distribution. Are there any guarantees that maximizing ELBO indeed ...
Daniel Mendoza's user avatar
2 votes
1 answer
40 views

VAEs: Why do we need the encoder for image generation?

I'm probably missing something obvious, but if we're only looking to generate images and are not interested in the latent space, why do we even need the encoder in VAEs? In my understanding, the ...
Jannik's user avatar
  • 125
0 votes
0 answers
14 views

ShapeNet VAE KL Divergence issues

I am trying to train a VAE on shapenet but I can't seem to make it work. Any help or ideas would be highly appreciated. Now the problem is whenever I apply the KL divergence loss the network seems to ...
Youssef's user avatar
3 votes
1 answer
131 views

Are there any situations where orthogonality is not optimal?

Data reduction is often used to avoid overfitting and to enhance explainability. Popular data reduction techniques, such as SVD or PCA map/project high-dimensional data to a lower-dimensional ...
Chris M's user avatar
  • 39
0 votes
0 answers
9 views

Creating a light image generation model for a specific distribution

I am currently working on how a user can introduce bias in a neural network model. To do so, I am creating an image2image model that only works on the training distribution. For example, let's say I ...
Adrien's user avatar
  • 19
0 votes
0 answers
9 views

Clarification about varitional autoencoders training

I have one specific question about VAEs as I try to work through the math on my own. I found from this paper that to training a VAE model involves optimizing a lower-bound of the marginal log-...
Brooklyn Sheppard's user avatar
1 vote
0 answers
27 views

1 dimensional autoencoder as a clustering tool?

I am looking for references (as I prefer to sit over the giant's shoulders...) to something it "seems" to work well... When we do clustering to analyse some data, to understand its structure ...
Antonello's user avatar
  • 403
0 votes
0 answers
21 views

Multivariate Variational Autoencoder and Positive Definite Covariance Matrix

This might be a naive question from a non-statistician but here we go. I understand that the challenges that hamper the use multivariate variational encoder where a covariance matrix is used instead ...
applied_env's user avatar
1 vote
0 answers
25 views

why Conditional VAE require conditioning of the encoder

looking at this blogpost and in many other, the cVAE looks like this: Now, my question is... why do we need the label on the encoder level? Clearly that information is already inside the image, thus ...
Alberto's user avatar
  • 1,217
0 votes
0 answers
13 views

How can probability variables of different dimensions share the same parameter in VAE?

I am confused while studying VAE(Auto-Encoding Variational Bayes). My problem is as follows: We have a continuous variable $x$ and a random variable $z$. The dimensions of $x$ and $z$ are different, ...
Seonil Choi's user avatar
0 votes
0 answers
6 views

Characterise distribution overlap

I'm applying VAEs to sections genomic data (haplotypic vcf format, so binary variables), with one model being trained on each section. Depending on which section I'm applying these methods to, my ...
Whitehot's user avatar
2 votes
0 answers
73 views

Do discontinuous functions have subgradients also?

Typically, the subgradient is defined for convex functions. And convex functions are continuous. However, DeepMind's VQ-VAE paper defines a model with a discontinuous vector quantization (VQ) layer, ...
MWB's user avatar
  • 1,337
0 votes
0 answers
89 views

VQ-VAE - Commitment Loss

Why the commitment loss is necessary in the VQ-VAE objective function? I understand that it's serving the role of keeping the encoder outputs(continuous latent representations) close to the codebook. ...
mcr0yal's user avatar
2 votes
0 answers
55 views

VAE latent variables apper highly correlated

I am attempting to train a Beta-VAE on historical interest rate curves to map interest rate curves to a lower dimension latent space. I have chosen to use VAE over PCA or autoencoders because I want a ...
Alex's user avatar
  • 21
2 votes
1 answer
40 views

Is the assumption of a diagonal covariance matrix on the latent space in a variational autoencoder in any way restrictive?

The covariance matrix in an autoencoder is assumed to be diagonal. And, I see it mentioned in good places that this is a fairly restrictive assumption. To quote However, in order to simplify the ...
figs_and_nuts's user avatar
0 votes
0 answers
60 views

VAE with linear decoder and nonlinear encoder, does this just learn a linear decomposition of the data?

There are a number of variational autoencoder(VAE) methods that have nonlinear encoders and linear decoders. The concept of using the linear decoder is to improve the interpretability (which features ...
sanK's user avatar
  • 1
1 vote
0 answers
40 views

Understanding Variational inference and EM in relation to each other

I have read several answers like here but, somehow I still have a few doubts. I hope to present my understanding and ask a few questions to clear my doubts EM: A maximization maximization algorithm E-...
figs_and_nuts's user avatar
1 vote
1 answer
132 views

Nonlinear PCA vs Encoder in Autoencoders

I see that encoders have the benefit over PCA that they can transform both linear and non-linear data. However, isn't non-linear PCA designed to work with non-linear data? So, why do we still prefer ...
AliM's user avatar
  • 131
0 votes
0 answers
19 views

Why doesn't BERT give me back my original sentence?

I've started playing with BERT encoder through the huggingface module. I passed it a normal unmasked sentence and got the ...
AlanSTACK's user avatar
  • 640
0 votes
0 answers
13 views

Challenges with Predictor (regression) Performance: Persistent MAE of 0.26 and Inaccurate Prediction of Binary Vectors

I am trying to work on building an variational autoencoder in Keras, with an input shape of ...
stevGates's user avatar
  • 111
1 vote
0 answers
24 views

Regression of a binary vector to obtain another binary vector [duplicate]

I am working in the security field. I have a dataset, called X, of binary victors. So the sample is a binary vector, for example sample in X = [1,0,0,0,1,0,....n] ...
stevGates's user avatar
  • 111
0 votes
0 answers
19 views

Why is the KL Divergence term present in the Variational Auto Encoder Loss? [duplicate]

I am trying to understand VAEs. A youtube video and a paper that I read about it defined the loss as roughly: $$L=\sum||x-Dec(Enc(x))||_2^2 + D_{KL}(\mathcal N(\mu, \sigma)|\mathcal N(0, 1))$$ The ...
Wolfuryo's user avatar
  • 133
0 votes
1 answer
128 views

How to do dimension reduction from a variational autoencoder

I am thinking about a variational autoencoder. As far as I understand it, in the encoding section you compress to a px1 tensor and then you create a $\mu$ and $\...
user1357015's user avatar
  • 1,744
0 votes
0 answers
28 views

What is the expectation of P(x|z) under distribution of z parameterized by x?

This question stems from Section 2.1 of this VAE tutorial. The problem stated in the paper is to compute the data likelihood using law of total probability: $$ P(X) = \int P(X,z) \,\mathrm dz = \int P(...
Kaiwen's user avatar
  • 81
0 votes
0 answers
8 views

increasing the capacity of an autoencoder

I have an autoencoder model with 5 layers in encoding and 5 layers in the decoding section. I am using this model for signal processing the problem is that it is making the signal way more smooth that ...
rrSep's user avatar
  • 1
0 votes
0 answers
24 views

Trained network always predicts zero [duplicate]

I have an encoder model and I'm training it with a dataset of signals with size (500,1). The data set is normalized and then used to train the model but the problem is that after the model is trained, ...
rrSep's user avatar
  • 1
1 vote
1 answer
199 views

The loss of VAE is negative. is it normal?

the function loss of VAE is : ...
Ramzy's user avatar
  • 21
0 votes
0 answers
29 views

VAE for Motion Sequence Generation - Convergence Issue with Scheduled Sampling

I implemented a Variational Autoencoder (VAE) in PyTorch for motion sequence generation using human pose data (joint angles and angular velocities in radians) from the CMU dataset. The VAE ...
RTn's user avatar
  • 1
0 votes
0 answers
27 views

On the expressivity of latent variable models

Empirically, we have seen that VAEs can approximate very complex distributions. I am interested in knowing if there are any theoretical results showing how expressive latent variable models can be. ...
Saeed Hedayatian's user avatar
0 votes
0 answers
71 views

KL Divergence in VAE [duplicate]

The basic KL-Divergence between two distributions is as: $KL(N(\mu_1,\sigma_1) || N(\mu_2, \sigma_2)) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{...
Martin Perry's user avatar
0 votes
1 answer
245 views

Image pre-processing for Variational Autoencoder

Setting I am training a Variational Autoencoder (VAE) on the CIFAR10 dataset, which has RGB colors. The VAE uses convolution and transposed convolution layers as well as linear layers to encoder and ...
joba2ca's user avatar
  • 143
0 votes
0 answers
17 views

Deconvolution vs tf.Reshape

I currently have a 1D-CNN which produces a 1D output due to the dense layers at the end of said CNN but want it to produce a 2D output. Instead of reshaping my tensor elements using tf.reshape, would ...
ryl06's user avatar
  • 1
0 votes
1 answer
173 views

Understanding a beta-variational autoencoder

I'm working on a beta-variational autoencoder using car images from the Vehicle Color Recognition Dataset. At this point, I'm just exploring different architectures and values for beta. (If you're ...
KirkD_CO's user avatar
  • 1,158
1 vote
0 answers
56 views

Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?

I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though $q_{\phi}(z|x)$ and $p_{\theta}(x|z)$ are Gaussian. I understand that the parameters ...
Decaying Tails's user avatar
0 votes
0 answers
12 views

How can a linear autoencoder with $h=1$ hidden unit reconstruct any rank 1 matrix?

I've had this as a homework problem as a true or false type of question and I'm trying to wrap my head around why this is true. Is the reason simply represent each datapoint as a scaled version of a ...
Oliver's user avatar
  • 310
0 votes
0 answers
27 views

How do I check that two ways of expressing the same random variable lead to the same distribution?

When reading the explanation for the "The reparametrization trick" on the Stanford's cs228 notes, I saw a claim that It is easy to check that the two ways of expressing the random variable ...
blueberryfields's user avatar
0 votes
0 answers
21 views

Can a linear autoencoder perform projection as PCA with less layers than components? [duplicate]

Assume that you have a matrix $X$ and you want to project it onto a subspace by using PCA. It will work. Then you are trying to use a linear autoencoder to projecting $X$ onto the same subspace. It ...
euraad's user avatar
  • 425
1 vote
0 answers
15 views

A ML classifier for predicting the hourly direction for a group of stocks where training stocks don't match out of sample stocks [closed]

I have historical data for 100 stocks (call them A). I would like to train a model jointly on all stocks which will learn from the cross sectional historical activity, to predict another set of stocks ...
GlaceCelery's user avatar
0 votes
0 answers
11 views

Linear autoencoders - Will they only preserve linear separable data?

I'm looking for to compress an image $X$ into a smaller image $x$. But not only compress, also reduce its view into a simplier view that are linear separable. My question is: Can I use linear ...
euraad's user avatar
  • 425
2 votes
1 answer
241 views

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output

How can be decoder of VAE represent probability distribution p(x|z) eventhough we directly get image as output. Also if my current understanding is right than we get same image for same value of z ...
Hamit Des's user avatar
0 votes
2 answers
979 views

How to Resolve Variational Autoencoder (VAE) Model Collapse in Reconstruction Task Using Sensor Data?

I am currently experiencing a suspected model collapse in a Variational Autoencoder (VAE) model I am working with. Below are details on the project setup and the issue at hand: Project Goal: Exploring ...
Bae Browns's user avatar
2 votes
1 answer
47 views

What purpose do higher dimensional mappings serve compared to lower dimensional mappings?

In Neural Network Architectures I understand that lower dimensional mappings(for example mapping and input to a space of lower dimension) can serve the purpose of decreasing dimensionality while ...
Kiran Manicka's user avatar
0 votes
2 answers
94 views

Rank Neurons Importance of the latent space of an Autoencoder using PCA

I am trying to extract only the important neurons from the latent space of an Autoencoder to be converted later to a pattern for a model pattern recognizer. PCA Loadings helps in finding the highest ...
hatahetahmad's user avatar
0 votes
1 answer
80 views

In factor VAE, do you freeze the discriminator weights during the back propagation step for the FVAE loss?

In factor vae, Disentangling by Factorising, there are two losses that are minimized. One is the VAE loss (eq. 2 in the paper) that includes (1) reconstruction loss, (2) KL divergence and (3) Total ...
S R's user avatar
  • 33
1 vote
0 answers
81 views

Is it correct to do SVD from the latent space of an autoencoder?

Is it correct to do SVD from the latent space of an autoencoder? I am asking because I think that by performing SVD from a latent space, and plotting the singular values, it is possible to know the ...
Carlos's user avatar
  • 11

1
2 3 4 5
13