# linear regression - polynomial of higher degree with one hot encoding

I am trying to fit a hypothesis to my data in order to predict the duration for a certain event.

My data is strictly categorical and I used one-hot encoding for all features.

After using one-hot encoding the dataset now has the dimensions 20.000x414.

After plotting the training and cross validation error as a function of the training set size, I know that my hypothesis suffers from a high bias.

I thought about using a polynomial of higher order but that would not make any difference because all of my features are either '0' or '1'.

Is that assumption correct?

Not too sure what your question is. Could you clarify what are your input features and what you are trying to predict. If your output is binary, I would suggest using softmax function and your objective function for optimization should be a cross-entropy. Using a polynomial regressor is not appropriate in this case.

• Sure, my input features are all categorical, so I used one hot encoding. Now they are all binary. The output features is not binary. it is a real number. – cmplx96 Nov 9 '17 at 13:01
• my question is: I think that it doesn't make a difference, if I use a polynomial of a higher order because 1^k or 0^k is still 1 and 0 respectively – cmplx96 Nov 9 '17 at 13:03
• Yes you are right. It does not make sense. I would suggest considering mapping an entire input vector directly rather than mapping them individually using a polynomial mapping. Try to use a series of sigmoid or radial basis functions for the nonlinear mapping. – Karlsson Yu Nov 9 '17 at 13:13
• do you mean that I should apply a sigmoid function to the features in the dataset before actually training them? – cmplx96 Nov 9 '17 at 13:23
• Please Have a read of the linear basis function regression methods method.utstat.utoronto.ca/~radford/sta414.S11/week1b.pdf – Karlsson Yu Nov 9 '17 at 13:48

You wrote:

I thought about using a polynomial of higher order
but that would not make any difference
ecause all of my features are either '0' or '1'.

Is that assumption correct?


Your hypothesis (as I understand it):

If one makes a polynomial fit to data that only contains the values "0" and "1", then a first order polynomial (line) is going to be no more or less effective than an $n^{th}$ order polynomial where $n \ge 2$ because my values are only 1 and zero.

Question:
Can a polynomial function on the corners of a unit hypercube give better classification accuracy than a linear function?

My approach:
You can think of your data as a existing on the corners of a unit hypercube of dimension equal to the number columns.

You are wanting to see if you can make a surface through that cube such that points at corners on one side tend to be 1, and on the other side of the surface tend to be 0.

Can I come up with a polynomial surface that will give better dispositioning than a hyperplane?

The curvature gives advantage if there is an idea of interior. If I can rotate the space, then I could solve the 2d-xor problem with a polynomial, while it has been proved that it requires two separate "perceptrons" (planar cuts of the space). I can't do it in 2d unless I can make a function like $z \left( x,y \right)$.

I can do that. I think that is part of how SVD gets its "powers". Rotation or parametric expressions are allowed.