# Tag Info

0

Have you tried lowering the learning rate of SGD to 0.001? It seems like the training loss decays too fast (due to a potentially high learning rate) and that potentially explains the divergence of the training and validation loss.

0

The difference would be in the first step in weights when you calculate the back propagation. To calculate the gradient NN will make a step in weights $\Delta\theta$. Although the product $X\theta$ is the same when you start, the first step $X\Delta\theta$ will be very different because the scales of $X$ are different, by two orders of magnitude in fact. ...

1

If a unit with ReLU activation gives $0$ for all the training inputs (that means also $0$ gradient based on the ReLU, which is the whole issue), it can't recover. This is dying ReLU problem, which is addressed by leaky ReLU activation. In your case, once the neuron parameters are pushed out of the data region, they're dead forever. A better way to tackle ...

1

The procedure, that you described, has a similar effect to artificially inflating the sample size. It's akin to cloning the observations of the sample. Say, you take each observation and create 9 more copies of it and run the Bayesian update on the 10 times bigger sample. Here's an example from the Wiki article. Suppose there is a school having 60% boys ...

1

They normalize the pixel values and then multiply by some hyper parameter that is chosen arbitrarily. Actually, they're scaling and shifting an interval [0,255] to be [$\lambda,1-\lambda$], then applying logit to. Logit can't handle 0 or 1 as you know: $\mathrm{logit}(x)=\ln\frac x {1-x}$, so they have to put a floor and a ceiling on its inputs. They ...

1

In Bayesian inference we typically are interested in the conditional distribution of our parameters $\theta$ given data. In the case of regression, we condition on our predictors $X$ and outcomes $y$. Given known Gaussian noise with known variance, this reduces to inferring the conditional distribution of regression coefficients $\beta$ given the data $X,y$:...

0

You can compute gradient of cross-entropy loss and softmax activation combo or logistic loss and sigmoid activation combo in single step. You will see no gradient between loss and activation of last layer but you will have simple gradient function for combo, like just (labels - predictions).

1

Okay, cross-validation is a great starting point for hyperparameter tuning. There is no one right answer about this, but here are some general thoughts about your methods: 1. Train on the full train/validation dataset and use the test set as "new" validation. I'm assuming this means that you train on the best hyperparameters, and test the resultant model ...

0

The terminology "stochastic gradient descent" is inconsistent. In some sources, SGD is exclusively the case of using 1 observation randomly-chosen without replacement per epoch to update a model. In other sources, stochastic gradient descent refers to using a randomly-selected sample of observations for updating the model, of any size, including a mini-batch ...

0

Yes, those methods are valid. You can even inject those auxiliary inputs earlier into the network by "tiling them" -- that is, if you have some K-dimensional vector of auxiliary inputs, copy it on the height and width axes until you have a KxHxW tensor, and concatenate it to some intermediate feature map of your CNN. You could use one-hot coding for ...

0

Should I keep images of my dataset for this purpose or should I use a real life dataset ? Depends on what you're trying to accomplish. If you ultimately care about the performance on your dataset (perhaps you wish to compare with other models also benchmarked on this dataset), then you should evaluate on it. If you ultimately want to deploy the model on ...

0

You appear to be referring to the view of the computation graph provided by tensorboard or a similar visualization tool. Typically, these visualization tools don't draw every weight as a separate edge -- that would not really be feasible, since neural networks can have hundreds of millions to billions of parameters, which I doubt most plotting software ...

3

When the weights are zero-initialized, it's certain that you get $-\log 0.5$ in the first batch. In normal, this is not guaranteed, but you'll get similar results on average. Because, on average, each input to logistic regression will be $E[w^Tx+b]=E[w^T]x+E[b]=0$, because $E[w]=E[b]=0$. Actually, each input to sigmoid function is going to be normally ...

0

Let's first look at the cell state memory of a vanilla RNN: $$a_t = g(W[a_{t-1}, x_t] + b_a)$$ where $g$ is some activation function (sigmoid, tanh), $a_{t-1}$ the previous state and $W$ and $b_a$ the weights and biases. For a vanilla RNN, we incur a vanishing gradient because as we continually compute the activation function, our previous state $a_{t-n}$...

1

In my case the initial training set was probably too difficult for the network, so it was not making any progress. I have prepared the easier set, selecting cases where differences between categories were seen by my own perception as more obvious. The network picked this simplified case well. After it reached really good results, it was then able to ...

1

No, there is no connection between the two. The backfitting algorithm that is used to fit GAMs, iteratively improves the estimator $\hat {f_j}$ by subtracting the contribution of each predictor $j$ from the response $y$ that is influenced by all, say '$p$' predictors, followed by smoothing using optimal bandwidth, which is yet solving a linear equation ...

0

In general, you should normalize your input using some method. This happens for at least two reasons: 1) Inputs of different magnitudes will affect differently the weights of the neural network. This will make the convergence more difficulty. 2) If the input are larger it can cause the saturation of the intermediate units of the neural network.

0

Correct me if I'm wrong, but it appears as though you're question comes as a result of the research you have done on convolutional neural networks, where, in many explanations, the convolutional layers are described as certain kernel convolutions, for example, a sobel edge detection. However, in actual reality, these convolutions are not completing a ...

1

The reason you need softmax is because your last layer of outputs has more than one neuron. Suppose, you have a sigmoid activation in the last layer but it has two neurons. How are you going to plug this into a loss function? What is your scalar prediction? So, with more than neuron you need a way to aggregate the outputs of the neurons into one number, a ...

1

Is the NN trying to estimating the distribution of the classes? No, you are not estimating the parameters of a distribution. The idea is similar to the logistic regression. The logistic regression provides the probability the dependent variable is one or zero using a generalized linear model. In the case of the logistic distribution, the dependent variable ...

0

Usually, in a Markov Decision Process, $S_t \text{ and } R_t$ are defined as random variables that depend only on the preceding state and action (Markov Property). $S_t \text{ and } R_t$ have well defined discrete probabilities, which determine the probability of assuming values at any given time step $S_t = s$ and $R_t = r$. Similarly, also the actions are ...

1

The first loss equation is meant to update the parameters of the Q-network $\theta_i$, in which the target network of parameters $\theta_i^-$ is used to approximate, according to the Bellman equation, the optimal Q-function $Q^*$. So if $\theta_i^-=θ_i$ in the first loss equation, does the gradient still hold and will the update still be called ...

6

I think you confuse the purpose of the two methods. Maximizing the ELBO leads to a parameterized class of densities that approximates closely the true distribution, in terms of Kullback-Leibler divergence. If you instead just do SGD on the target, what you will achieve is just a (local) maximum of parameters, but no approximate probability distribution. ...

1

That particular slide is about Perceptron algorithm, where initially $w$ is $0$ and you update it for each misclassified sample with the following update rule (there are slightly changed versions of this, but sticking with the slides): $$w\leftarrow w+y_ix_i$$ Because we start from $0$ and every update made is in terms of $y_ix_i$, the final version of the ...

0

Both the encoder and decoder is used when computing the reconstruction error. So to compute anomaly score you just pass new samples through the trained autoencoder. If you have enough data you might periodically retrain the model to compensate for trends. For example use the last N weeks as the window for training data, and retrain once per week. Detecting ...

0

Probably because the model becomes more complicated than necessary. While deep networks can accommodate more non-linearity which makes them more powerful in certain tasks, adding more non-linearity than necessary will increase training cost with no significant improvement in performance. Deep Averaging Networks (DANs) are used to obtain sentence/document ...

2

My answer is certainly NO for cryptography, and maybe YES sometimes for compression. Artificial neural nets (ANN) can't be trained to decrypt data. The reason is that the inverse problem is too stiff. Encryption In cryptography, by design, encryption transforms input X into output Y=f(X) using a very rough function f(). That's the whole idea of encryption ...

-1

your data has outliers ,use z score or IQR to remove outliers here on how to and normalization is also important to scale all columns

0

Disclaimer: the approach presented is not feasible for continuous values, but I do believe bears some weight in decision making for the project Smarty77 brings up a good point about utilizing a rescaled sigmoid function. Inherently, the sigmoid function produces a probability, which describes a sampling success rate (ie 95 out of 100 photos with these ...

0

Well, you can use a label encoder on training data, but I typically don't. If there are just two categories, you could use it safely (I just tend to apply a function that recodes the data to 0s and 1s.) However, if there are more than two categories and and the categories don't represent an ordinal scale like tickets (tickets could be first-class, second-...

0

No, independent of the model you use, it's not a good idea. It's not the case that MLP handles or understands it. Imposing an ordinality may create unexpected problems. Your model performance is probably good because of another feature subset.

0

I don't think either of the answers provides a clear definition, so I will attempt to answer it because I stumbled into the same problem finding a clear definition of a hidden unit in the context of a Convolutional Neural Network. Hidden units in this context are the feature maps or filters. So for Tensorflow or Keras it would be tf.nn.Conv2D(**...

1

For your first question -- probably just a typo. For your second question, the lagrangian dual of the first formulation would be $$\max_{\lambda > 0} \min_{r} |r|+ \lambda \text{loss}(x+r,l)$$ If we set $c = 1/\lambda$ and multiply through, then we have $$\min_{c,r} c|r|+\text{loss}(x+r,l)$$ To gain better intuition on this part, it might help to ...

0

This would require defining a distance metric on embedding/latent space, since there is no "built-in" distance. Assuming you pick any reasonable distance like L2, and assuming that your neural network mapping images to latent space is injective, then yes, you will also have a valid distance metric between images. If your network is not injective, then of ...

1

This happens because your data contains missing values. For example, minimum of np.array([np.nan, 10, -4.5]) is nan, and likewise the maximum. Any sum or difference of nan is also nan. Dividing by nan yields nan, so all of your data values are nan where the feature contains at least 1 nan value. You need to somehow fix your missing values. The preferred ...

0

Regarding your first question: "How does parameter sharing help the model?" The main purpose of sharing is the radical reduction of parameters, one of the main selling points was to reduce the impractical size of Bert. The authors do mention that the parameter reduction also acts as regularisation/generalisation. It does not improve the performance of the ...

1

I don't think this makes much sense. You can't set other layers of the CNN to 0 and expect any meaningful output. As an analogy, suppose you try to analyze what each organ of a human body does by shutting down the rest and testing how the human performs -- likely nothing good will happen and you won't gain any insight. All parts of the system need to ...

2

I don't think there is a strict answer in terms of sample size. It depends on the complexity of the analysis. A neural network (NN) will find patterns in whatever you give it. The question is, have you given the NN enough observations to find the patterns of interest. What are you looking for? If you're asking a NN to classify images of humans, then yes, 55 ...

1

The difference is that for $KL(p||q)$ minimum over $p$ is attained (it's zero when the $p = q$) but maximum may not exist (or make sense in usual way). KL-divergence is a convex function. Convex functions on bounded sets attain their minima. There is a whole branch of math that deals with convex optimization. The same doesn't hold for maxima - for KL ...

3

In fact Hinton et al (2014) in their paper, as inspiration refer to paper by Caruana et al (2006), who described distilling knowledge from an ensemble of models. Moreover, distilling is a very basic and versatile approach that can be applied to any model. The point is that you have noisy data and train a big model on this data, this model learns some kind of ...

0

All a neural network does is minimize some loss function. A sum (or multiple) of loss functions is a perfectly fine loss function in its own right. In fact, this is quite a common setup. L(whatever) regularization is an extra term in the loss function, for one example. A VAE doesn't even work properly unless you use a sum of two losses, for another. If ...

1

If you focus on the generative part, GANs and VAEs are actually mathematically the same object (1), i.e. Gaussian latent variable models, where $z$ is a latent Gaussian random variable pointing to an observed $x$: The difference is that VAEs are prescribed models that output a random variable $x$ with a probability density, while GANs are likelihood-free ...

0

After posting my question I realised I know how to recover both the unscaled mean ($\mu$) and variance ($\sigma^2$), which I can then use to compute the unscaled $\theta$. We have $$\mu = k\mu' \;\;\;\;\;\; \sigma^2 = k^2\sigma'^2 \;\;\;\;\;\; \theta = \frac{\mu^2}{\sigma^2 - \mu}$$ where $\mu'$ and $\sigma'^2$ denote the scaled values. And doing some ...

0

Weights in Deep Learning infer the connection strength between the neurons. They have a direct influence on the output. So if the weights are near to zero, that means changing this input will not have any significant influence on the output. In short, Positive weights indicate a positive influence on the output and vice versa. Bias, on the other hand, ...

0

Yes, it does -- for some binary classification procedures. For example, gradient boosting is formulated around the assumption that the labels are -1 or 1 (See "Boosting and Additive Trees" Chapter in Elements of Statistical Learning). That's mainly because of the type of loss that boosting uses ($e^{-y{F(x)}})$, where you can easily see that using $\{0, 1\}$...

1

One important thing to note as well is that the cross entropy is not a bounded loss. Which means that a single very wrong prediction can potentially make your loss "blow up". In that sense it is possible that there are one or a few outliers that are classified extremely badly and that are making the loss explode, but at the same time your model is still ...

2

Short overview about Expectation maximization : Marginal likelihood Expectation maximization contrasts with 'regular' likelihood maximization by refering to the maximization of a marginal likelihood. \underbrace{p(X\vert \theta)}_{\substack{\text{marginal likelihood}\\\text{ $\mathcal{L}(\theta \vert X)$}}} = \int_z \underbrace{p(X, z \vert \theta)}_{\...

0

Found this library that works as an alternative to fastai vision's unet_leraner classifier: https://segmentation-models.readthedocs.io/en/latest/api.html#unet

0

I believe this is the answer. Hopefully, someone more knowledgeable can confirm. What I was looking for was a one-tailed p-value derived from a two-sample t-test with the assumption of equal variances. I found this calculator: http://www.usablestats.com/calcs/2samplet&summary=1 And this more detailed explanation: https://www.itl.nist.gov/div898/...

1

I'd like to recommend this limpid article: CS231n Convolutional Neural Networks for Visual Recognition, and let me compare the (simplified) vanilla network with the (simplified) residual network as follows. Here is a diagram I borrowed from that page: where the green numbers above the lines are indicating the forward pass, and the red numbers the ...

Top 50 recent answers are included