0
$\begingroup$

I have found this cost function for logistic regression (source of formula: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression):

$\displaystyle \underset{w,c}{min} \ \frac{1}{2} w^{T} w+C\sum ^{n}_{i=1} log\left( exp\left( -y_{i}\left( X^{T}_{i} w+c\right)\right) +1\right)$

I would like to code this function to python, but I have some issues understanding some parts of the formula:

  1. regarding this part: $\displaystyle\frac{1}{2} w^{T} w$

    • If w is vector of weights of features, how it can be transposed? Isn't that same like doing just w * w ?
  2. regarding this part: $\displaystyle x^{T}_{i} w$

    • Here it's similar, ith vector is multiplied with another vector, so why the transpose symbol?
    • Also, most important question: if two vectors are multiplied, result is vector again. Does it mean that all 'outer' (exp, log, sum) operations are done on vectors? But in that case, function output is vector. How do we can minimize function when output is vector?
$\endgroup$
0

1 Answer 1

3
$\begingroup$

If w is vector of weights of features, how it can be transposed? Isn't that same like doing just w * w ?

If you view a column vector as a $k \times 1$ matrix and a row vector as a $1 \times k$ matrix, then the meaning of a transpose should be obvious because it corresponds to the usual usage in linear algebra and matrix arithmetic.

The notation w * w is ambiguous. Do you mean dot product or element-wise product?

If two vectors are multiplied, result is vector again.

This is not what the logistic cost function says. The logistic cost function uses dot products. Suppose $a$ and $b$ are two vectors of length $k$. Their dot product is given by $$ a \cdot b = a ^\top b=\sum_{i=1}^{k} a_i b_i = a_1b_1 + a_2b_2 + \cdots +a_kb_k. $$

This result is a scalar because the products of scalars are scalars and the sums of scalars are scalars.

Does it mean that all 'outer' (exp, log, sum) operations are done on vectors?

No, the result of a dot product is scalar.

How do we can minimize function when output is vector?

You don't need to minimize a vector because the result of the logistic regression cost function is a scalar. You use ordinary scalar minimization.

$\endgroup$
1
  • $\begingroup$ thanks, this is clear $\endgroup$
    – Jan Musil
    Commented Jun 22, 2020 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.