3
$\begingroup$

I have the following ANN architecture, the neuron is a sigmoid neuron:

enter image description here

Where the weight and parameter matricies are given by:

$$ \begin{vmatrix} & x1& & x2& & x3& \end{vmatrix} \begin{vmatrix} & w1& & w2& & w3& \end{vmatrix} $$

My cost function for one training example is given as such, it is just MSE: $$ C(a,y)=(a-y)^2 $$ Where y is the actual output and a is defined below,

$$ a=σ(w∙x) $$

I have two questions: What is the dimensionality of my cost function in this case?

How does this calculation extend to N training examples, how are the weights updated through backprop in that scenario? Is the average of the gradient across all training examples taken and used to update the weights?

Improve this question
$\endgroup$
5
  • $\begingroup$ What do you mean by the "dimensionality" of a function? $\endgroup$
    – D.W.
    Commented Mar 28, 2020 at 20:16
  • $\begingroup$ @D.W. Like how a function y= f(x) is two dimensional. In this case, the cost function is y = f(w, x, y ...). I need to know the specific dimensionality of the cost function in this case $\endgroup$
    – Aditya Mehrotra
    Commented Mar 28, 2020 at 20:20
  • 1
    $\begingroup$ y=f(x) is univariate if x is a real number. why do you say that it's two dimensional? $\endgroup$
    – gunes
    Commented Mar 28, 2020 at 20:22
  • $\begingroup$ Also, please ask only one question per post. You can learn how training of neural networks and backpropagation works in standard resources that are widely available, so there is little point in us repeating it here. $\endgroup$
    – D.W.
    Commented Mar 28, 2020 at 20:23
  • $\begingroup$ @gunes Because the graph of a univariate function can be graphed on a Cartesian grid, which means it exists in two dimensions -x,y. $\endgroup$
    – Aditya Mehrotra
    Commented Mar 28, 2020 at 20:24

1 Answer 1

Reset to default
1
$\begingroup$

Given $y$ and $\mathbf x$, the cost is in the form $C=f(w_1,w_2,w_3)$, which means this is a multivariate function with domain $\mathbb R^3$, having dimensionality $d=3$. Since the function value is a scalar, if you plot your function, you'll be plotting (or at least storing) in 4D. So, it's $4$, but I refrain from calling it as dimensionality.

For $N$ training examples (in the batch), your cost function becomes as follows, which can be found in many posts in our from as @D.W. suggests: $$C=\sum_{i=1}^N C(\sigma(\mathbf{w}^T\mathbf{x}_i),y_i)$$

Improve this answer
$\endgroup$
2
  • $\begingroup$ So the actual output y and predicted value x don’t function as variables but actually constants in the scope of one training example? $\endgroup$
    – Aditya Mehrotra
    Commented Mar 28, 2020 at 20:55
  • $\begingroup$ in ml literature, yes, because you don’t differentiate wrt them. $\endgroup$
    – gunes
    Commented Mar 28, 2020 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.