4
$\begingroup$

Python's scikit-learn package has a convenient pipe function that can combine machine learning techniques into one model with fitting and predicting functions. I was following this tutorial for chaining PCA and logistic regression and everything works as expected but I am having trouble describing this in terms of math notation because I am unsure what is really going on.

I know PCA algorithm is as follows:

  1. Normalize/standardize data matrix: $Y=HX$
  2. Calculate covariance matrix: $S=\frac{1}{n-1} Y^T Y$
  3. Eigenvalue Decomposition: $S = Z \Lambda Z^{-1}$
  4. Find PC Scores of original data: $T_L=YZ_L$

Where the transformation matrix is $Z_L$, and $L$ is the number of principal components used. Therefore, we can compress the original data using $T_L = Y Z_L$, where $T_L$ still has the same number of rows but only $L$ columns, thereby resulting in a reduced dataset.

How do I then combine the resulting $T_L$ matrix which is my transformed data with a multinomial ($>2$ classes) logistic regression?

I know multinomial logistic regression model is something like: Suppose the response variables has $K$ levels in the space $G=\{0,1,2,...,K\}$, representing the set of possible classes. The probability of determining a particular class is defined as,

$$Pr(G=k | X=x) = \frac{e^{\beta_{0k}+\beta_k^Tx}}{\sum_{l=1}^K e^{\beta_{0k}+\beta_k^Tx}}$$

But I do not know how to relate the two in a logical manner. (Or how the cost function should be defined.)

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.