In order to perform regression you need to find the equation that best fits the data. It is said you can select the equation by plotting the data. But when the data has several input variables how to find the equation that best fits my data since I can't plot all the variables?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If you have no ideas about a form of the equation, it can be good idea to start from linear equation:
a1*x1 + a2*x2 + ... + an*xn = y
Here x1, ..., xn are input variables and y is a variable you are trying to predict. a1, ..., an are the coefficients used to fit the data.
If you then find out that such an equation underfits your data (i.e. yields high error on the training dataset), try adding new polynomial features like x1^2, x1*x2, x2^2, ... starting from low orders.
However, adding too much polynomials can cause overfitting - low error on training dataset and high error of test dataset.
To prevent overfitting, it was proposed a trick, called regularization - adding a special term to the error function that gives high "penalties" for large coefficients near high-order polynomials.
Say, if you're using mean square error function, the modified error function will be like this:
J(a1,...,an) = sum(i)(y[i] - h(a1,...,an, x[i]))^2 + λ * sum(j)(aj^2)
Here h(a1,...,an, x[i]) is the model used to fit y, which includes linear and polynomials from the input variables, a1,...,an are the it's coefficients and λ is called a regularization term. This question has some useful information on how to select the value of the regularization parameter.
Also for more information you can see Andrew Ng's overview lecture on regularization and overfitting from the awesome Machine Learning course.
