I have a question about the shape of the error surface for online gradient descent algorithms. Take into account that I am trying to translate my specific question into a more general and idealized situation so that you can understand it. This means that the situation and constraints you will read here may look a bit odd and simplistic.
Imagine the following situation:
- I have only one parameter to optimize. This is really like having an extremely simple connectionist network with 2 units (input and output).
- The data that are provided to the system are also extremely simple: Both the input and output units can be thought as binary elements (they are either active, value=1, or inactive, value=0). On each trial, I present one exemplar consisting of a pair of numbers, corresponding to the activation of either unit.
- Moreover, let's make it simpler. Imagine that the input unit is always active (value=1) and I just need to find the best parameter (the weight) to predict the current outcome (target = 1 or 0). So each exemplar actually consists of a 1 and a target to be predicted (either 1 or 0).
- My learning algorithm is a variant of the delta rule that performs steepest descent on the error function.
- Note that the parameter is updated online, after presenting each exemplar (as opposite to the more usual batch update method). This is the main complication of my idealized problem. I need my learning algorithm to adapt to unexpected changes in the input stream.
Now, the question:
Given this situation, I have a problem "visualizing" and understanding the real task the algorithm is carrying out. In particular, I am wondering what exactly the error function looks like, even in this extremely simplified situation.
My intuition tells me that the curvature of the surface will depend on the consistency in the input stream. Say, if many consecutive trials consist of the same target (e.g., "1-1, 1-1, 1-1..."), then the surface would be flatter, while if they point to inconsistent directions (e.g., "1-0, 1-1, 1-0...") it would look more curvy / steep.
- Is this intuition right?
- Would you suggest any metaphor or image to help me "visualize" the problem the online algorithm is actually solving?