New answers tagged

0 votes

If fine tuning produces better performance than feature extraction, is there any advantage of using feature extraction?

A very common scenario is that we simply do not have the resources or time (or at least it's a get most of the benefit for minimal effort situation) to fine-tune a model appropriately, but can run ...
  • 24.1k
0 votes

If fine tuning produces better performance than feature extraction, is there any advantage of using feature extraction?

If the dataset is of similar nature, the features extracted for the original (possibly very large) dataset, which is conceptually achieved in the first layers of a neural net, should be representative ...
  • 53.1k
7 votes

$\sin(x)$ is a counterexample to the universal approximation theorem

The classical (Cybenko) universal approximation theorem has a condition about the function being approximated on a compact space. On the real line, the Heine-Borel theorem says that compacts sets are ...
  • 35.6k
0 votes

Cross-Entropy or Log Likelihood in Output layer

I think that @user650654 made a mistake in his formulation of the Cross Entropy and therefore his conclusion is incorrect. In the case of hard labels (i.e., using one-hot vectors for ground truth, ...
0 votes

Why Deep Learning needs to be performed in Graphical representations?

Because traditional deep learning methods do not take into account a crucial property of the adjacency matrix representation: node permutation equivariance. One axiom of deep learning is that ...
  • 16.1k
1 vote

Why Deep Learning needs to be performed in Graphical representations?

Same could have been said about image data, but CNNs were born. Models that exploit the structure of the underlying data are likely to be more successful than generic methods.
  • 53.1k
0 votes

Autoencoder accuracy with standardized data

In the input of the network it's fine to input normalized data. This is exactly what batch normalization does. In the output layer you don't add an activation. Instead you output the values from the ...
  • 814
0 votes

Can naive Bayes model this type of (approx. circular) decision boundary?

You were right. Naive Bayes can actually create a circular decision boundary as the variance of red and black data points is different. For Gaussian distribution estimation, if NB is forced to use ...
1 vote

Interpret neural network like the linear regression equation such as how much will Y change if we change X1 and keep the other variables fixed

You can interpret NN like that but it is pointless generally. The seeming utility in doing so for OLS is in decoupling of the partial derivative from other variables, I.e. $\partial/\partial x_1 f(x_1,...
  • 57.1k
1 vote

Bias Variance tradeoff in neural networks

Bias-variance trade-off is an old fashioned concept from classical statistics which fails to be useful in high-dimensional setting. Here's an example of famous statistician being surprised that ...
0 votes

Does neural network work this way in general?

If I understand you correctly, you are asking "what is the black box". First of all, not only neural networks are "black boxes", but this would be the case also for simple machine ...
  • 121k
1 vote

Why do graph convolutional neural networks use normalized adjacency matrices?

I adopt the authors notation and use $\tilde A$ for the normalized adjacency matrix. The largest eigenvalue $\lambda_1$ of the normalized adjacency matrix $\tilde A$ is $\lambda_1 \le 1$. This ...
  • 81.2k
4 votes

Bias Variance tradeoff in neural networks

The existence of a bias-variance tradeoff has been assumed as inevitable (i.e., an axiom) in any model using data, including neural networks. However, it has been observed since about 2018 that ...
  • 361
0 votes

Is a neural network's hidden layer the same thing as the hidden state?

The hidden layer is a set of operations (like ODE-Solver operations in Neural ODE models), and the state layer is a set of results (set of numbers) from these operations. In other words, the hidden ...
0 votes

Importance for Color in X ray imaging for Detection of Pneumonia

transforming RGB to grayscale lead to loss of important information of colours It would be easier to ask this to a radiologist instead of a statistician. From the statistical point of view you try ...
1 vote

Importance for Color in X ray imaging for Detection of Pneumonia

Are we talking classic X ray radiographs or some fancy modern technique ( MRI, PET-CT, Spectral-CT, ...). Classic X ray radiography has no color information. They measure how much of an electric ...
  • 7,947
0 votes

For neural networks, is mini-batching done purely because of memory constraints?

It is difficult to prove anything and many results assume some network architectures, unless you can use fit everything into 1 batch. But from my experience, for small batch sizes relative to the ...
8 votes
Accepted

Interpret neural network like the linear regression equation such as how much will Y change if we change X1 and keep the other variables fixed

One of the issues when you introduce nonlinearities and interactions is that the change resulting in a change in a variable of interest depends on the starting value of that variable of interest and ...
  • 35.6k
7 votes

Interpret neural network like the linear regression equation such as how much will Y change if we change X1 and keep the other variables fixed

This is a nice question, that touches on some interesting points in the history of neural networks (which I can only briefly mention here). First, what you say is absolutely right if and only if the ...
  • 6,161
1 vote

How to account for varying levels of “confidence” among data points during training

Here's how I'd address this. From your description it sounds like you're trying to predict a "conversion rate" for each image, and once you have the conversion rate you treat the clicks / ...
2 votes

For neural networks, is mini-batching done purely because of memory constraints?

It might be useful to consider the extreme cases of mini-batch sizes: using a single sample vs taking all $N$ samples in one go. These are sometimes also referred to as on-line and batch learning, ...
  • 1,649
9 votes

For neural networks, is mini-batching done purely because of memory constraints?

It's complicated. We're trying to balance multiple effects. Yes, you are right in your reasoning as far as it goes, but there are other considerations as well. GPU utilization favors larger batches, ...
  • 6,194
1 vote

NN type/architecture needed for inverse covariance matrix approximation

Related to your question, there are times when you want to use something like inverse covariance estimation to perform link prediction across time series. For this you can include an inverse ...
  • 11
2 votes

For neural networks, is mini-batching done purely because of memory constraints?

so you have to understand that neural networks are used as 'computation-bound' statistical models. ie they are solving the problem how accurate can I get with a budget of X compute-hours. reasoning ...
  • 4,444
3 votes
Accepted

Neural networks - what does learning rate exactly mean and is it applied over batches or epochs?

Gradient descent updates the model parameters $\theta$ using the rule $$ \theta^{(t+1)} = \theta^{(t)}-\eta \nabla \mathcal{L}(f(X,\theta^{(t)}),y) $$ where $\eta$ is the learning rate, $\mathcal L$ ...
  • 81.2k
11 votes
Accepted

For neural networks, is mini-batching done purely because of memory constraints?

There is evidence that supports the proposition that it is best to use the biggest batch size your machine can handle. See e.g. Goyal et al. (2018). However, that paper (and another) reveal ...
  • 310
-1 votes

Bias Variance tradeoff in neural networks

Sidenote: It depends on the situation The question is a bit of a loaded question. It presupposes that neural networks are better. But, whether neural networks are better depends on the situation. If ...
0 votes

Why is it important to include a bias correction term for the Adam optimizer for Deep Learning?

All the above answers are helpful. Why not just visualize the claims? Here is an animation that I created to demonstrate the following statement from the paper lack of initialisation bias correction ...
3 votes
Accepted

Should we scale the data if our response(Y) is numeric, a large number, and 99% of other variables are dummy variables?

Standardizing would be advisable. Neural networks usually do not behave well with large numbers due to disproportionate gradient steps.
  • 53.1k
2 votes
Accepted

On using the same tokenizer for train and test data

The point of using a test set is to validate your model on unseen data. To do this, you need to apply exactly the same preprocessing and prediction function to the test set. If you used different ...
  • 121k
0 votes

GANS: Using Discriminator for prediction

I have a different opinion than above and I think you got misunderstood. I believe it is possible. Training of GAN is done when the generator fool the discriminator. The Generator is useless at this ...
  • 1
1 vote
Accepted

why is the receptive field size not depend on the input image?

To answer the second part of the question: In addition, what would be the affect of stride and padding on the receptive fields? As explained in Nikolas Adaloglou's blog on AI Summer Understanding ...
  • 987
1 vote

Does Discriminator in GAN Train only on Real Data or it also Trains on Fake Generated Data

The input to the complete GAN is only the real data, so the complete GAN only trains on the real data. But the discriminator part of the GAN learns to distinguish between the generated and real data, ...
  • 7,576
3 votes

why is the receptive field size not depend on the input image?

"Receptive fields are defined portion of space or spatial construct containing units that provide input to a set of units within a corresponding layer. The receptive field is defined by the ...
  • 81.2k
3 votes

Is label smoothing equivalent to adding a KL divergence term or a cross entropy term?

You are just one step away! Following Lei Mao's derivation $\eqref{eq1}$, we can replace the second term in the right hand with $\eqref{eq2}$. $$L^{'} = \sum_{i=1}^n \left[ (1-\epsilon)H_i(p, q_\theta)...
1 vote

What machine learning architectures can be used when dealing with variable input size?

Transformer neural networks seem like an obvious candidate. They are most famous for being used for text data, where of course the input size is variable and the order is fixed. However, note that ...
  • 24.1k
1 vote
Accepted

What machine learning architectures can be used when dealing with variable input size?

Here are some possible approaches: Use Deep Sets. Estimate a probability density for each set of your $n$ $m$-dimensional vectors (the columns) and take the parameters of this density as input for a ...
  • 7,576
3 votes

Reframing a HMM problem as an RNN

Neural networks can be used to amortize the optimization part, effectively learning an adaptive solution given a corpus of data. The connection with VAEs is pretty easy to see here. So, in your ...
  • 16.1k
4 votes

Reframing a HMM problem as an RNN

The hidden nodes (states) in an HMM are random variables, while in an RNN only the input nodes could be considered random variables, all the other nodes are just deterministic nonlinear functions. ...
  • 7,576
1 vote

Using AI for human like mouse movement between two points

This is a very nice example/application of cGAN, conditional generative adversarial network... you have a dataset made of mouse tracking, so you set as labels initial and final point, and you feed ...
1 vote

Using AI for human like mouse movement between two points

This isn't a proper answer, but if I had to solve this problem (and only had a few hours to do so) I would just find the trajectory in the training set that was most similar to the target in terms of ...
  • 6,161
1 vote

Matrix form of backpropagation with batch normalization

Forward pass of batch normalization: Suppose $\mathbf{X}$ has shape $n\times m$ where $n$ is number of nodes in the layer and $m$ is number of samples in a batch, $\mathbf{J}$ is matrix of ones (...
0 votes

Choosing parameters for an artificial neural network with a likelihood ratio test

If you are using a neural network, I recommend that you don't do feature selection. The usual approach is to just include all the available features, and let the neural network learn which ones are ...
  • 6,194
0 votes

Do you tune hyperparameters for neural networks one at a time?

Typically people fix a network architecture, and then tune the hyperparameters (learning rate, number of epochs, etc.) for that architecture. There are many ways to do hyperparameter tuning, and no ...
  • 6,194
1 vote
Accepted

Overfitting on small dataset to check if model is good

I use the proposed method just as a bug detection: if my neural network does not provide good results, I try to fit a small batch to see if there is any bug in the network architecture or training ...
  • 713
4 votes

How to train neural network classifier to have a low false positive ratio, when false negative ratio is almost irrelevant?

Train a probabilistic classifier with a proper scoring rule as a loss. If you then want a hard 0-1 classification (which I would argue is not part of the statistical modeling step, but of the ...
0 votes

How to calculate the Transposed Convolution?

In 3x3 images, just like in 2x2, each pixel of the input is multiplied with the kernel matrix and accumulated. A naive implementation is given by this page: ...
  • 53.1k
1 vote
Accepted

How to update betas each time step with optimizer adam?

$t$ is most possibly the time step, which would mean a decreasing $\beta_i$ as time goes.
  • 53.1k
2 votes

How predictions are made in time-varying survival analysis?

These are panel data, with the same individuals evaluated at a set of discrete points in time, 13 time points in this case. The Health and Retirement Study supplements that with "Exit" ...
  • 68.7k
3 votes

What are the advantages of combining BiLSTM and CRF?

Totally agree with Jindrich, I work in Chinese Grammatical Error Correction (CGEC) and BiLSTM-CRF is the fundamental structure of our network. Basically, BiLSTM is used to take the context into ...

Top 50 recent answers are included