4
$\begingroup$

Python's scikit-learn package has a convenient pipe function that can combine machine learning techniques into one model with fitting and predicting functions. I was following this tutorial for chaining PCA and logistic regression and everything works as expected but I am having trouble describing this in terms of math notation because I am unsure what is really going on.

I know PCA algorithm is as follows:

  1. Normalize/standardize data matrix: $Y=HX$
  2. Calculate covariance matrix: $S=\frac{1}{n-1} Y^T Y$
  3. Eigenvalue Decomposition: $S = Z \Lambda Z^{-1}$
  4. Find PC Scores of original data: $T_L=YZ_L$

Where the transformation matrix is $Z_L$, and $L$ is the number of principal components used. Therefore, we can compress the original data using $T_L = Y Z_L$, where $T_L$ still has the same number of rows but only $L$ columns, thereby resulting in a reduced dataset.

How do I then combine the resulting $T_L$ matrix which is my transformed data with a multinomial ($>2$ classes) logistic regression?

I know multinomial logistic regression model is something like: Suppose the response variables has $K$ levels in the space $G=\{0,1,2,...,K\}$, representing the set of possible classes. The probability of determining a particular class is defined as,

$$Pr(G=k | X=x) = \frac{e^{\beta_{0k}+\beta_k^Tx}}{\sum_{l=1}^K e^{\beta_{0k}+\beta_k^Tx}}$$

But I do not know how to relate the two in a logical manner. (Or how the cost function should be defined.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.