# Tag Info

222

These are not very strict terms and they are highly related. However: Loss function is usually a function defined on a data point, prediction and label, and measures the penalty. For example: square loss $l(f(x_i|\theta),y_i) = \left (f(x_i|\theta)-y_i \right )^2$, used in linear regression hinge loss $l(f(x_i|\theta), y_i) = \max(0, 1-f(x_i|\theta)y_i)$, ...

169

First of all, there's no reason that an agent has to do the greedy action; Agents can explore or they can follow options. This is not what separates on-policy from off-policy learning. The reason that Q-learning is off-policy is that it updates its Q-values using the Q-value of the next state $s'$ and the greedy action $a'$. In other words, it estimates ...

46

First of all, what actually policy (denoted by $\pi$) means? Policy specifies an action $a$, that is taken in a state $s$ (or more precisely, $\pi$ is a probability, that an action $a$ is taken in a state $s$). Second, what types of learning do we have? 1. Evaluate $Q(s,a)$ function: predict sum of future discounted rewards, where $a$ is an action and $s$ ...

18

The fundamental difference is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, unlike in bagging where all features are considered for splitting a node.

18

Disclosure: I didn't have time to carefully read the full paper yet. From the abstract: Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. Foundational models is a term with a good hype potential, but ...

16

Quoting from section 4.3 in "Deep Learning" book by Ian Goodfellow, Yoshua Bengio, Aaron Courville (emphasis in the original): The function we want to minimize or maximize is called the objective function, or criterion. When we are minimizing it, we may also call it the cost function, loss function, or error function. In this book, we use these ...

16

This is just terminology, no need to think about it too much as different people classify different areas into different categories. For example a lot of statisticians would consider machine learning to be a sub-area of statistics, people from AI would consider machine learning to be a sub area of AI research, and people working with computer science ...

15

Make the identity of the agent one of the features, and train on all data. Probably train on a mini-batch of eg 128 agents at a time: run through the time-series from start to finish for those 128 agents, then select a new mini-batch of agents. For each mini-batch, run a slice of say 50 timesteps, then backprop. Keep the end states from that slice, and run ...

15

Here is a quick summary on the Reinforcement Learning taxonomy: On-policy vs. Off-Policy This division is based on whether you update your $Q$ values based on actions undertaken according to your current policy or not. Let's say your current policy is a completely random policy. You're in state $s$ and make an action $a$ that leads you to state $s'$. Will ...

14

According to Wikipedia, there are 4 main types of artificial neural network learning algorithms: supervised, unsupervised, reinforcement and deep learning. Unsupervised learning algorithms: Perceptron, Self-organizing map, Radial basis function network Supervised learning algorithms: Backpropagation, Autoencoders, Hopfield networks, Boltzmann machines, ...

14

The only difference is that : The sigmoid neuron is able to output values between 0 and 1. The stochastic binary neurons only can output either a 0 or a 1 with the probability given by the sigmoid activation function.

13

As you mentioned, most neural networks are based on general simple abstractions of the brain. Not only are they lacking in mimicking characteristics like plasticity, but they do not take into account signals and timing as real neurons do. There's a fairly recent interview, that I felt was appropriate for your specific question, Machine-Learning Maestro ...

13

According to Prof. Andrew Ng (see slides on page 11), Function h(X) represents your hypothesis. For fixed fitting parameters theta, it is a function of features X. I'd say this can also be called the Objective Function. The Cost function J is a function of the fitting parameters theta. J = J(theta). According to the Hastie et al.'s textbook "Elements of ...

13

In Andrew NG's words- "Finally, the loss function was defined with respect to a single training example. It measures how well you're doing on a single training example. I'm now going to define something called the cost function, which measures how well you're doing an entire training set. So the cost function J which is applied to your parameters ...

13

I think there's some confusion here. The reason you have vanishing gradients in neural networks (with say, softmax) is wholly different from RNNs. With neural networks, you get vanishing gradients because most initial conditions make your outputs end up on either the far left or far right of your softmax layer, giving it a vanishingly small gradient. In ...

12

Not much --- arguably nothing --- has so far been learnt about brain functioning from artificial neural networks. [Clarification: I wrote this answer thinking about neural networks used in machine learning; @MattKrause (+1) is right that neural network models of some biological neural phenomena might have been helpful in many cases.] However, this is perhaps ...

10

The difference between Off-policy and On-policy methods is that with the first you do not need to follow any specific policy, your agent could even behave randomly and despite this, off-policy methods can still find the optimal policy. On the other hand on-policy methods are dependent on the policy used. In the case of Q-Learning, which is off-policy, it ...

10

I don't think it would be possible to answer this question with respect to proprietary models used by private enterprise. But there is a vein of scholarship that focuses on flawed practices, such as in this paper. Zachary C. Lipton, Jacob Steinhardt "Troubling Trends in Machine Learning Scholarship" Collectively, machine learning (ML) researchers ...

10

The Bitter Lesson is that in the long term, progress is dependent on leveraging more and more computational power. This is not to say that algorithmic and modeling progress isn't important, but they aren't the limiting factor -- neural networks have been since the 1950s (or earlier), and it's only now that increasing computation resources have let us ...

9

Bagging in general is an acronym like work that is a portmanteau of Bootstrap and aggregation. In general if you take a bunch of bootstrapped samples of your original dataset, fit models $M_1, M_2, \dots, M_b$ and then average all $b$ model predictions this is bootstrap aggregation i.e. Bagging. This is done as a step within the Random forest model algorithm....

9

I would like to provide clarification, there is a disctinction between bagging and bagged trees. Bagging (bootstrap + aggregating) is using an ensemble of models where: each model uses a bootstrapped data set (bootstrap part of bagging) models' predictions are aggregated (aggregation part of bagging) This means that in bagging, you can use any model of ...

8

It is certainly not true that the human brain only uses "a few" convolutional layers. About 1/3 of the primate brain is somehow involved in processing visual information. This diagram, from Felleman and Van Essen is a rough outline of how visual information flows through the monkey brain, beginning in the eyes (RGC at the bottom) and ending up in the ...

8

My father went to university shortly after World War II and he used to tell me about how he once did a long Fourier transformation and how they used those very long paper rolls, as they did it in a paper pencil approach. When I went to school in the 80s, people used to think, that in some time in the future everybody would have to deal with computers. And ...

7

In RL, for value functions, the bias and variance refer to behaviour of different kinds of estimate for the value function. The value function's true value is the expected return from a specific starting state (and action for action values), assuming that all actions are selected according to the policy being evaluated. For control problems you can just aim ...

6

The whole point of that initialisation scheme (sometimes called "Xavier initialisation") is to mitigate the problem of disappearing gradients caused by the form of many activation functions. Consider the common sigmoidal activation function $s(z) = \frac{1}{1+e^{-z}} = \frac{e^z}{e^z + 1}$. Its gradient $\sigma'(z) = s(z)[1-s(z)]$ approaches $0$ away from ...

5

The idea is to learn the bias weights but have the activation fixed at 1. Anything else would make it an additional ordinary unit.

5

The recommended thing to do when using ReLUs is to clip the gradient, if the norm is above a certain threshold, during the SGD update (suggested by Mikolov, see http://arxiv.org/pdf/1211.5063.pdf) This requires another hyperparameter, the threshold. The suggestion from the referenced paper is to sample some gradients to get an idea of the (non-exploding) ...

5

That depends on the image. In general the answer is 100%, but in different strengths, depending on both the image and the specific network. In fact, encouraging full-network usage is one of the objectives of the "dropout" technique. This has been written about extensively in many places, but this blog post is a nice quick introduction. I'm not aware of in-...

Only top voted, non community-wiki answers of a minimum length are eligible