6
If you think of $L$ as a column vector, then I think both your sources agree that $\frac{dJ}{dL}$ should be a row vector.
But what if you really want $L$ as a row vector. Surely, the math shouldn't "care" about how you arrange your collection of numbers. One way to clarify this is by designating dimensions of your objects as "covariant" ...
5
Unfortunately, I didn't come across a resource that doesn't leave gaps. It's a disputed area. Even the chain rule may sometimes not make a lot sense, e.g. some terms might be 3D tensors that the matrix multiplication is not well-defined because of matrices differentiated by vectors or vice versa.
Having said that, these rules are also very useful if you ...
5
I understand the question as asking
If we have a loss $L_1$ which is the regression portion and another loss $L_2$ which is the classification portion, how do we train a NN to minimize both?
This touches on the optimization topic of multi-objective optimization, for which one approach is finding the pareto optimal solution. A simple approach would be for ...
3
If the state-space is small, you could indeed solve by dynamic programming (which would require visiting all the states, though). In exponentially large state-spaces (which are the ones you care about in AI), it would not work, and the objectives proposed in the paper are ways around that using the generalization power of deep learning or other ML approaches ...
3
My initial idea then is to use some sort of stacking to combine RF with some linear or polynomial model
Random Forest is not a proper tool for this. I would try Gaussian Process, trying Neural Network and Squared Exponential kernels, modelling the mean as a linear function. See the lower three charts in an example from Golding & Purse 2016:
Surely ...
2
Your input-to-hidden matrix $W_{hx}$ has shape $M \times K$. Your hidden matrix $W_{hh}$ has shape $M \times M$. Then $h_t, b_h, a_t$ all have shape $M$. The output matrix $W_{yh}$ has shape $K \times M$, so $W_{yh}h_t$ has shape $K$. Softmax doesn't change any shapes, so your output is $K$.
You seemed confused about whether to think about $x_t, h_t$, etc as ...
1
When using cross-entropy for classification tasks, that can also be viewed as a conditional probability distribution (categorical - the output values are the parameters giving the probability of each catageory) and when using MSE for regression that is also specifying a conditional probability distribution (the conditional mean of a Gaussian distribution - ...
1
Turning my comment into an answer:
This is doable, but it's not common.
The assumption in an RNN is that the hidden state already carries with it information from all previous states. This isn't true in (e.g.) a $k$th-order HMM. By the Markov assumption, you absolutely can only look back $k$ steps.
I don't think the higher-order RNN would have much value, ...
1
I don't no the real answer if there is any.
I'll try to answer on the square root normalization.
Lets say that we have two independent random variable, x and y. Both have the variance. What would be the variance of their sum?
Var(x + y) = Var(x) + Var(y) = 2*const.
Now, lets define another two random variables: x' = x/sqrt(2), y' = y/sqrt(2).
What would be ...
1
As already mentioned, it depends on what data you train your GAN. But it also depends on what you expect as an outcome of the GAN. Most methods focus on complete new synthetic data, but that's not the only option. This approach of landing AI seems to be more promising than just generating new artificial data, they augment existing data using a GAN to enrich ...
Only top voted, non community-wiki answers of a minimum length are eligible