I have the following ANN architecture, the neuron is a sigmoid neuron:

enter image description here

Where the weight and parameter matricies are given by:

$$ \begin{vmatrix} & x1& & x2& & x3& \end{vmatrix} \begin{vmatrix} & w1& & w2& & w3& \end{vmatrix} $$

My cost function for one training example is given as such, it is just MSE: $$ C(a,y)=(a-y)^2 $$ Where y is the actual output and a is defined below,

$$ a=σ(w∙x) $$

I have two questions: What is the dimensionality of my cost function in this case?

How does this calculation extend to N training examples, how are the weights updated through backprop in that scenario? Is the average of the gradient across all training examples taken and used to update the weights?

  • $\begingroup$ What do you mean by the "dimensionality" of a function? $\endgroup$ – D.W. Mar 28 '20 at 20:16
  • $\begingroup$ @D.W. Like how a function y= f(x) is two dimensional. In this case, the cost function is y = f(w, x, y ...). I need to know the specific dimensionality of the cost function in this case $\endgroup$ – Aditya Mehrotra Mar 28 '20 at 20:20
  • 1
    $\begingroup$ y=f(x) is univariate if x is a real number. why do you say that it's two dimensional? $\endgroup$ – gunes Mar 28 '20 at 20:22
  • $\begingroup$ Also, please ask only one question per post. You can learn how training of neural networks and backpropagation works in standard resources that are widely available, so there is little point in us repeating it here. $\endgroup$ – D.W. Mar 28 '20 at 20:23
  • $\begingroup$ @gunes Because the graph of a univariate function can be graphed on a Cartesian grid, which means it exists in two dimensions -x,y. $\endgroup$ – Aditya Mehrotra Mar 28 '20 at 20:24

Given $y$ and $\mathbf x$, the cost is in the form $C=f(w_1,w_2,w_3)$, which means this is a multivariate function with domain $\mathbb R^3$, having dimensionality $d=3$. Since the function value is a scalar, if you plot your function, you'll be plotting (or at least storing) in 4D. So, it's $4$, but I refrain from calling it as dimensionality.

For $N$ training examples (in the batch), your cost function becomes as follows, which can be found in many posts in our from as @D.W. suggests: $$C=\sum_{i=1}^N C(\sigma(\mathbf{w}^T\mathbf{x}_i),y_i)$$

  • $\begingroup$ So the actual output y and predicted value x don’t function as variables but actually constants in the scope of one training example? $\endgroup$ – Aditya Mehrotra Mar 28 '20 at 20:55
  • $\begingroup$ in ml literature, yes, because you don’t differentiate wrt them. $\endgroup$ – gunes Mar 28 '20 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.