# Help in problem formuation :Hebb's learning

In the supervised learning problem, the goal is, given a training set, to learn a function $h : X \mapsto Y$ so that $h (x)$ is a “good” predictor for the corresponding value of $y$. If $y$ takes discrete values, then it is a classification problem.

$X = \{x_1,x_2,...,x_N \}$ is the training set of $N$ examples where each example $x_i \in R^D$ has $D$ number of elements in binary. Then the problem formulation becomes : $$y^{(i)} = \theta^T x^{(i)} + e^{(i)} \tag{1}$$ where $i = 1..N$ and $e$ is the prediction error. Please correct me if I am wrong in saying that this is a linear regression formulation. Estimating the unknown parameter $\theta$ for all the training examples can be done using Least Squares. I think $\theta$ will be a vector of size $D$.

I have training data that consists of $N$ time series where each $\{\mathbf{x}_i\}$ is of length $D$. The problem is to learn a transformation coefficient $\mathbf{W}$ for all $N$ examples such that the output $\mathbf{y}_i \approx \mathbf{x}_i$ . The target output should be close to the input for all the examples in such a way that the error between $\mathbf{e}_i = ||\mathbf{x}_i - \mathbf{y}_i||$ is minimum. The data $\{\mathbf{x}_i\}$ comes from a nonlinear dynamical map TEnt map $g(.)$ , so each example $\{\mathbf{x}_i\}$ is obtained from $D$ iterations of $N$ different maps. The question is based on the document http://www.ece.rice.edu/~erzsebet/ANNcourse/handouts502/course-cf-3.pdf which talks about Nonlinear Hebb learning. But it is unclear to me how I can apply to my model in Eq(2)

I cannot represent the problem mathematically.

I tried something like this but not sure if it makes sense and so need help in formulation. Is the following problem formulation correct?

$$\mathbf{y}_i= Wg(\mathbf{x}_i) + \mathbf{e}_i \tag{2}$$

and assuming error to be $i.i.d$ Gaussian distributed. I cannot understand how to derive Hebb's learning law for weight update $\theta$. Can somebody please provide the solution? Thank you.

• What's your goal? Any equation would be arbitrary without having something specific that you're trying to do. – user20160 Jul 25 '16 at 23:49
• I have updated the question and added more information – SKM Jul 25 '16 at 23:56
• You say you want to learn a transformation $W$ such that $y_i \approx x_i$, but $y_i$ is a scalar and $x_i$ is a vector. Can you be more precise in this statement? – Sven Jul 28 '16 at 0:33
• $\mathbf{y}_i$ is not a scalar. $\mathbf{y}_i$ is a vector of length $m$; $W$ is of size $m \times N$. I am not sure if the learning is performed for all the examples at one go, if so, then the dimensions will be $Y \in R^{m \times N}$; $W \in R^{m \times D}$ – SKM Jul 28 '16 at 0:45

first of all you are mixing two different things, linear regression and non linear Hebbs learning (''neural networks''). Even tought both approaches aim to solve the same problem, they way they do it differs. Main difference is that in nonLinear Hebb learning you are training a perceptron which has typically a non linear activation function, as sigmoid or soft sign (or many others). Because of this, it's formulation changes a bit:

\begin{equation} \vec y = f(W\vec x + \vec b) \end{equation}

where $f$ is the activation function, $W$ are the weights and $\vec b$ the bias (please note this a CS notation, in neuroscience notation differs. For instance the bias is called threshold instead). Here you can find a book which provides a very detailed introduction to Neural Networks, I think you could go trough it, especially the first chapters and see how NNs are working (also suggest reading Back-Propagation chapter as if the categories are not linear separable best to use Back-Prop learning, which is really just a generalization of Hebbian learning for non-linear units and multi-layer networks).

For what it concerns the weight update:

\begin{equation} \Delta W = \gamma y(\vec x)\vec x \end{equation}

where $y(\vec x)$ is the output of the netowrk when presented the input $\vec x$.

For a more detailed explanation, with a nice (and more formal than mine) math background, I suggest you see this, and also for a more detailed description of non linear Hebb learning see this.

There is something confusing in your problem statement. I try to rephrase it as I understood it, but feel free to correct me if I got something wrong:

You have an initial dataset $X \in R^d$ with $|X| = n$, which then get transformed into $Y \in R^m$ trough a non-linear $g: R^d \rightarrow R^m$. I guess this function $g$ is unknown to you and you want to estimate it with a Neural Network, as linear Hebb's learning would bring you nowhere close to your goal (can't estimate a non linear function with linear Hebb).

In this case what you have is:

\begin{equation} Y' = f(WX + \vec b) \end{equation}

and your error function would be (the error function is what you try to minimize when learning, in this case is the reconstruction error):

\begin{equation} E = ||Y-Y'||_2 \end{equation}

therefore, what you want to learn is the weight matrix $W$ of size $d \times m$ and the bias vector $\vec b \in R^m$. The function $f$ IS NOT $g$ or trying in any way to be close to it. This is typically a sigmod function or a soft sign and it does not change over time. What changes is the matrix $W$ that is initially set to random and then trained trough your dataset $X$ and expected result (supervised) $Y$.

So to put it in your terms: you want to estimate $W$ such that the recostruction error $E$ is small.

I really suggest you for this problem to take a look at normal Neural Networks and forget for a second about Hebbs.

• I edited answer to give a followup of your comment. – Renthal Aug 9 '16 at 13:15
• Well for a proper answer of your question please refer to books i linked as it is way too complicate to explain shortly. Howevery, just to give you an insight, imaging clustering situation. If you were to train a Neural Network to perform such operation, the error function E might be e.g. the average distance from each sample to the center of their cluster (which you clearly want to mimimize). $Y$ would be the coordinates of the cluster center (class) an observed sample $X_i$ belong to.But then again, it's a long story this is just a toy example to give you an insight. – Renthal Aug 11 '16 at 15:47