Questions tagged [activation-function]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Train model on negative numbers in labels when real data contains none

Assume a dataset where the input x is a vector with values in the range [0, 1], and the label ...
user avatar
  • 103
1 vote
1 answer
14 views

How are centers in an RBF Network chosen?

I am struggling to understand how RBF (radial basis functions) work. My first question concerns the weights: are the learnable weights the same as the centres? So, is the algorithm essentially ...
user avatar
  • 63
1 vote
0 answers
19 views

When I do not add an activation function to my convolutional layer the model gets quickly stuck in a local optima, why?

I have model A: ...
user avatar
2 votes
1 answer
12 views

How do I check if the weights of my perceptron/step activation function are correct

I am new to stack overflow and deep learning so I hope I am doing this the right way. I tried to find the solution myself but it has not been successful so I am seeking some help. This is the ...
user avatar
  • 23
0 votes
0 answers
14 views

what is average unit activation and how to measure it?

recently I read a research paper about exponential linear units named "FAST AND ACCURATE DEEP NETWORK LEARNING BY EXPONENTIAL LINEAR UNITS (ELUS)". in section 4.1.1 they train 8 layers (128 ...
user avatar
1 vote
0 answers
52 views

Deep learning : Can I make an all positive network (features and weights)?

I am trying to train a deep neural network that will always use positive weights and the user will be obliged to enter a feature vector as input that will always be positive (normalized to 0-1). In ...
user avatar
0 votes
1 answer
19 views

If all computed features ( or components ) in Neural Network nodes are positive numbers , does using Relu meaningful?

I am trying to understand the following issue. The reason we use activation functions such as sigmoid,tanh or relu in neural networks is to obtain a nonlinear combination of input features ( x's). My ...
user avatar
0 votes
0 answers
9 views

Which NLP methods use gradient and activation methods?

I am doing a literature review of gradient-based methods for NLP. Yet, apart from linear and logistic regression, I have little knowledge of other methods using the gradient. So I have no knowledge of ...
user avatar
0 votes
0 answers
19 views

If the weight and bias gradients are stuck at zero throughout training, is this an indication of dying ReLu?

A high learning rate when combined with a ReLu activation function is known to lead to the 'dying ReLu' problem. Is this a reasonable conclusion to arrive at if the gradient with respect to weights ...
user avatar
2 votes
0 answers
69 views

What are the advantages and disadvantages of higher order neuron activation functions?

I've been reading about different types of neurons that the traditional linear one. One example I came across is the Sigma-Pi neuron, where the activation function includes higher order terms, such as ...
user avatar
0 votes
0 answers
19 views

How do you scale the activation function of an auto-encoder when using a custom normalization fitted on the data?

I'm working on a convolutional auto encoder. The input is an image The output is a reconstructed image During the training phase, we feed the same image in and out The loss is the Mean Squared Error ...
user avatar
  • 111
1 vote
0 answers
14 views

What activation function or pre-processing to use for features describing when a certain event occurred in the past?

I have a series of features that describe how long ago a certain event happened and whether it happened at all. Of course we could break down this features into two, whether it did happen or no, and ...
user avatar
  • 171
10 votes
5 answers
3k views

Can a neural network work with negative and zero inputs?

As the title suggests, I have several features which have values of either -1, 0 or 1. If I feed this data into a neural network where I use ReLu as the activation ...
user avatar
  • 255