Logistic Regression Algorithm? I am studying on some Machine Learning concepts. I am looking for logistic regression(multiclass) and logistic regression classifier and I should learn how to change it to penalize large weights.
I have looked here: http://en.wikipedia.org/wiki/Logistic_regression but doesn't give me information what I want. I checked Alpaydin's, Tom Mitchel's and Duda's book but couldn't find anything like that(maybe I don't know that there is another name for it) I could see some explanation at Bishop's Pattern Recognition book but I don't have full access to book to read.
My question is what is logistic regression classifier, and next, how can I penalize large weights on it?
 A: For your first question: What is a logistic regression classifier, I don't really know what kind of answer do you want.
But if you have some data that can be classified by category, you can use Logistic regression.
$$
J(\Theta ) = -[\frac{1}{m}\sum_{i=1}^{m}y^{(i)} log( h_{\Theta }(x^{(i)})) + (1-y^{(i)}) log (1-h_{\Theta }(x^{(i)}))]
$$
with $h_{\Theta }(x) = \frac{1}{1+exp(-\Theta ^{T}x))}$
and with your data being {($x^{(1)}$, $y^{(1)}$), ($x^{(2)}$, $y^{(2)}$), ..., ($x^{(n)}$, $y^{(n)}$)}
You need to minimize $J(\Theta )$ using gradient descent to find good enough parameters ${\Theta }$
If you want to penalize large weight you can use the same formula and use regularization.
Try to find the course from Andrew Bg on Machine Learning and more specifically the Logistic Regression part.
(I wrote the all thing quickly, I hope I didn't forget anything)
$$
J(\Theta ) = -[\frac{1}{m}\sum_{i=1}^{m}y^{(i)} log( h_{\Theta }(x^{(i)})) + (1-y^{(i)}) log (1-h_{\Theta }(x^{(i)}))] + \frac{\lambda}{2m} \sum_{j=1}^{n}\Theta_{j}^{2}
$$
${\lambda}$ being how much penalty you want to apply to your model
