ANN Cost Function Notation

Question

I have been following This book on the fundamentals of NNs. It is currently outlining the MSE Cost function, and the Notation is tripping me up some.

$$ C(w, b) = \dfrac{1}{2n} \sum_x \vert\vert y(x)-a\vert\vert^2 $$

** Note x is tied with sum operator, LaTex was weird

** Note if any other notation is still confusing the book does its best to explain. I am in This section, just after the start.

I have some questions.

• The book read that "The notation ∥v∥ just denotes the usual length function for a vector v" what exactly is the "length function" of a vector.

• The book also denotes that y(x) is a column vector representing the desired output of the network with the input x while a is the actual output. I know you can subtract vectors, but how can you square one to get a single value. Or am I missing something.

• Additionally, why do we divide by 2n rather than just n to get the mean value. (n being the number of training inputs you provide the network with.

• Finally, I have been coding a Network with what I know, and for the time being I have been avoiding biases. As for now I am just using the sign activation function to simplify the process. How would the function change with only weights as an input? I have provided my code, if it helps with this problem. ** its is not complete now

from random import randint
from math import exp
import numpy as np

def map(f,mat):
    arr = np.asarray_chkfinite(mat)

    for i in range(len(arr)):
        for k in range(len(arr[0])):
            arr[i][k] = f(arr[i][k])

    mat = np.asmatrix(arr)

    return mat

def sigmoid(z):

    return 1/(1 + exp(-z))

def sign(z):

    if z > 0:
        return 1

    else:
        return 0


class Point:

    def __init__(self,x,y):

        self.x = x
        self.y = y

        if x > y:
            label = 1

        elif x == y:
            x += 1
            label = 1

        else:
            label = 0


training_data = []
testing_data = []

for i in range(100):
    p = Point(randint(0,500),randint(0,500))
    training_data.append(p)

for i in range(100):
    p = Point(randint(0,500),randint(0,500))
    testing_data.append(p)




class NeuralNetwork:

    def __init__(self,num_inputs,num_hiddens,num_outputs):

        self.num_inputs = num_inputs
        self.num_hiddens = num_hiddens
        self.num_outputs = num_outputs

    def initiate_weights(self):

        self.weights_ih = np.zeros((self.num_hiddens,self.num_inputs))
        for i in range(self.num_hiddens):
            for j in range(self.num_inputs):
                self.weights_ih[i][j] = randint(-5,6)
        self.weights_ih = np.asmatrix(self.weights_ih)

        self.weights_ho = np.zeros((self.num_outputs,self.num_hiddens))
        for i in range(self.num_outputs):
            for j in range(self.num_hiddens):
                self.weights_ho[i][j] = randint(-5,6)
        self.weights_ho = np.asmatrix(self.weights_ho)


    def guess(self,inputs):

        inputs_matrix = np.asmatrix(inputs)
        inputs_matrix = np.reshape(inputs_matrix,(self.num_inputs,1))

        weighted_ih = np.matmul(self.weights_ih,inputs_matrix)

        self.activations_h = map(sign,weighted_ih)

        weighted_ho = np.matmul(self.weights_ho,self.activations_h)

        outputs = map(sign,weighted_ho)

        return outputs

    def train(self,inputs,labels):

        n = self.num_inputs

        error_sum = 0

        guess = self.guess(inputs)
        guess = np.asarray_chkfinite(guess)

        for i in range(n):

** I would also appreciate any feedback on the code itself, I'm open to any suggestions

Thanks in advance, I know it's a lot

Answer 1

1) This length is the usual Euclidean distance, basically the Pythagorean Theorem.

$$\vert\vert (x_1, \dots , x_n) \vert\vert = \sqrt{x_1^2+\dots+x_n^2} $$

Be aware that the double vertical bar notation means a norm, which is a specific operation in linear algebra and functional analysis. Without context saying otherwise, it usually would mean this usual Euclidean distance, but it has a more general definition. (That will matter if you want to get into ridge, lasso, or elastic net regularization, or if you explore the MAE loss function.)

2) The norm operation outputs a number, so you are just squaring a number, not a vector. Again, this is very related to Pythagoras.

3) This is not universal, but that’s for mathematical convenience when you take down the derivative and bring down the $2$. But it doesn’t matter much; what you want to is find the set of parameters that gives the lowest mean squared error, regardless of what the MSE is. (You’ll care about the value of the MSE later for assessing if your mode is useful or better than another model, and then you’ll need to interpret the performance metric and make sure you’re using the same performance metric for each model under consideration.)

I will link you to a post of mine where I explain this. I very much prefer the notation used in the question there. That’s a much easier way to understand MSE, which is not a neural net concept. (It comes up in neural nets, but it also comes up in ordinary least squares regression and random forest regression and every other type of regression.)

Finally, you may see the denominator of MSE written as $n-p$, the number of observations minus the number of parameters. This has to do with getting an unbiased estimate of the variance when you do ordinary least squares, which is may not interest you if you’re doing a neural net. (A common assumption in OLS is that the errors have equal variances, which is less of an assumption in neural networks.) However, all denominators one the MSE formula will, except for numerical issues related to doing math on a computer, give the same parameter estimates in your regression.

4) Nothing changes. The $w$ weights and $b$ biases determine the $y(x)$ values, but once you have the $y(x)$ values, the MSE machinery doesn't care how you got there. The weights and biases determine how $y$ acts on $x$, but then you just have a number (or a vector, as my edit below discusses) to run through the MSE equation. You can apply the MSE equation to predictions from neural net regressions, random forest regressions, linear regressions, elastic net regressions...

EDIT

Looking at (1) a second time, I want to mention that this is a very general way of writing the MSE that only becomes particularly useful when the response variable is a vector. Most of the time the form that will make sense is the form in the question I linked, which is a special case of the form in your question. Anyway, when the response variable is a vector, your predictions are vectors, so you find the mean squared error by considering each error to be the distance between the predicted vector and the actual vector. Then you add up those squared errors to get the sum of squared errors and divide by $2n$ (or $n$ or whatever) to get the mean squared error.

REMARK

Do note that if you take the square root of the MSE, even the MSE you get when you divide by $n$ instead of $2n$, you do not get the average amount by which a given prediction misses the actual value. This is a common, easy misconception, and it is wrong.