I've set up a regression model but am not sure if I'm doing it right. I'm using multiple regression to help do multi-class classification.
So far I feel like I've understood the theory, but I'm confused in implementation/interpretation. My confusion is between two possible interpretations of the regression part of a Gentleboost algorithm, which I will show below.
Let me explain my problem.
My data, $X$, is 2D, with $n$ number of observations, and $m$ number of features per observation. Now, the response vector is a 2D array too. In other words, given an observation $x_i$, the response vector doesn't merely tell me what the $y$ value is, but it gives me an array of values, one value for each class. For example, if my data can be classified as being from one of two possible classes, then the response vector might be like this: $y = (0,1)$. So that's telling me that the value for the first class is 0, and for the second class it's 1.
Now, on to my regression function, $g(x)$. Recall that for each observation I have $m$ features, in other words, I need $m$ regression parameters in the regression function. So that means I would store the parameters in a 1D array that holds $m$ values.
To illustrate this, say my data $X$ is as follows, with 10 observations and 3 features each. And the observations can easily be separated into two different classes. So the number of classes for $X$ below is 2:
So this was $X$, a 10x3 array.
This means that my regression function $g(x)$ should have 3 parameters, let it be a 3x1 array. Now because my response vector is 1x2 array, not not merely a single value, then if I put in an observation into $g(x)$, it should also give out a response vector, with one value per class, i.e. a 1x2 array. But in order for $g(x)$ to return a 1x2 array for me, it needs to have more than 3 parameters. It has to have 3 parameters for each class, that means in this case of two classes, it has to be 3x2 array. (I'm not sure if it really has to, I'm just explaining according to my understanding and implementation of this using
statsmodels in python. If I'm wrong, please correct me).
So $g(x)$ was 3x2. Let's say it looks like this:
And let's say that getting this 3x2 array of parameters above was called the "fitting" step.
However, recall that I needed a $g(x)$ for each class. So to store all the regression parameters for my data, given that I have two classes, I decided that I need a 2x3x2 array, which is a 3D array. Like this:
Is this true? Is this the correct understanding of
Step 2(a)(ii) in this algorithm below? (In the algorithm below, the calculated working response is $z$, not $y$).
So my confusion is whether $g(x)$ should be:
- A 2x3x2 array, as I mentioned above. (And if that's the case, then when I input test observations, $g(x)$ gives output of a 2x2 array, and how is that resulting array supposed to be interpretted?)
- Or, instead, should I stick to the 3x2 $g(x)$ that I got in the fitting step, and consider each column to be $g_j(x)$? In other words, given that my 3x2 $g(x)$ was
, should I now consider that the first column is $g_0(x)$, and the second column is $g_1(x)$? This would mean that the parameters for the regression function for the first class are
a, b, c, and for the second class they're
d, e, f. Is this the correct way to implement/understand the above step
Step 2(a)(ii) in the algorithm? If I do this, then I believe that giving test data to $g(x)$ would produce a 1x2 array, which makes more sense as the response vector.