1
$\begingroup$

I'm currently following Lindsay Smith's (in large parts very well written) PCA tutorial. I'm a bit confused about the usage of the term "feature vector" in this paper though. A quote from this paper is:

[...] What needs to be done now is you need to form a feature vector, which is just a fancy name for a matrix of vectors. This is constructed by taking the eigenvectors that you want to keep from the list of eigenvectors, and forming a matrix with these eigenvectors in the columns.

However, on Wikipedia one can read the following:

In pattern recognition and machine learning, a feature vector is an n-dimensional vector of numerical features that represent some object.

I'm a bit confused as I interpret Lindsay Smith's feature vector as being a matrix consisting of eigenvectors and Wikipedia's feature vector as an actual representation of each object of the input data set. My interpretation is that in the latter case one row of the feature vector could for example look as follows:

Person A = {Age:30,Height=199 cm,Weight=100 kg, ...}

This representation wouldn't have anything to do with eigenvectors.

Is any of those two definitions wrong or do they actually mean the same thing and I just don't see it? Thanks a lot for any answers.

$\endgroup$
1
$\begingroup$

She defines a feature vector as follows:

$$FeatureVector = (eig_1 eig_2 eig_3 ... eig_n)$$

Let's work through her example to see what she means by $FeatureVector$:

import numpy as np 
import matplotlib.pyplot as plt

data = np.array([[2.5,0.5,2.2, 1.9,3.1,2.3,2,1,1.5,1.1],
                [2.4,0.7,2.9,2.2,3,2.7,1.6,1.1,1.6,0.9]]).T

plt.plot(data[:,0], data[:,1], '.', alpha = 0.2)
plt.title("Original Data")
plt.show()

enter image description here

# normalizing before performing PCA
data_norm = data - data.mean(axis=0)


# number of observations/rows
m = data_norm.shape[0]

# vectorized covariance calculation
cov = data_norm.T.dot(data_norm) / (m - 1)

eig_vals, eig_vects = np.linalg.eig(cov)

# fixing column order to mirror Lindsay Smith's output
eig_vals = np.array([eig_vals[1], eig_vals[0]])
eig_vects = np.array([[eig_vects[:,1]], [eig_vects[:,0]]])
print("Eigenvalues:\n",eig_vals)
print("Eigenvectors:\n:", eig_vects, '\n')

# keeping both eigenvectors
final_data_all = eig_vects.T.dot(data_norm.T).reshape(data.shape)
# keeping only the first eigenvector
final_data_one = eig_vects[0].dot(data_norm.T)

plt.title("Original Data Rotated about principal Components")
plt.plot(final_data_all[:,0], final_data_all[:,1], '+')
plt.show()

enter image description here

# calculating the projection onto first principal component
RowOriginalData = (eig_vects[0].T.dot(final_data_one)) +\
                data.mean(axis=0)[:,None]
RowOriginalData = RowOriginalData.T

plt.plot(data[:,0], data[:,1], '.', alpha = 0.2, label = "Original Data")
plt.plot(RowOriginalData[:,0], RowOriginalData[:,1], '-b')
plt.plot(RowOriginalData[:,0], RowOriginalData[:,1], '+', 
        label = "Projection onto 1st PC")
plt.legend()
plt.show()

Projected Data

In this case, we have two $FeatureVector$s: one using both both principal components/eigenvectors (final_data_all) and one keeping only the 1st principal component (final_data_one). The first results in a rotation of the data about the principal components, and the second projects the data points onto the first principal component.

In this case, we had two instances of $FeatureVector$, final_data_all:

+-----------------------------+
| [[-0.82797019  1.77758033]  |
|  [-0.99219749 -0.27421042]  |
|  [-1.67580142 -0.9129491 ]  |
|  [ 0.09910944  1.14457216]  |
|  [ 0.43804614  1.22382056]  |
|  [-0.17511531  0.14285723]  |
|  [ 0.38437499  0.13041721]  |
|  [-0.20949846  0.17528244]  |
|  [-0.3498247   0.04641726]  |
|  [ 0.01776463 -0.16267529]] |
+-----------------------------+

and final_data_one:

+-----------------+
| [[-0.82797019]  |
|  [ 1.77758033]  |
|  [-0.99219749]  |
|  [-0.27421042]  |
|  [-1.67580142]  |
|  [-0.9129491 ]  |
|  [ 0.09910944]  |
|  [ 1.14457216]  |
|  [ 0.43804614]  |
|  [ 1.22382056]] |
+-----------------+

To answer your question, both use a different definition for feature vector, but they aren't necessarily wrong. You need to pay attention to context to understand what someone means when they use a confusing term. This has become especially problematic recently because every dusty old statistics concept needs a more modern buzzword. So you get multiple names for the same technique.

Code: https://github.com/geofflangenderfer/stack_overflow/blob/master/pca_smith.py

Original text: http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf

$\endgroup$
  • $\begingroup$ Wow, thank you very much for this very comprehensive answer. $\endgroup$ – Hagbard Dec 17 '18 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.