# Determining the influential features for an outcome

I have a small table like this

Outcome     Feature_1_Value  Feature_2_Value   Feature_3_Value   .....
1               1              1.2               3.7
9.7             2.3            3.976             4.65
....            ....           ....              ....


There is an outcome that depends on what values do a set of features have. The above table represents the training data. The outcomes and the features all are continuous values and are not discrete ( it is not 0/1/2 kind of values).

Now, how do I analyze the training set of data to determine which of the features actually influence the output and how do I re-arrange the features in decreasing order of their influence? I would like to represent nicely to the end-user saying Feature_n_Value was twice more influencing the output than Feature_m_Value, but Feature_n_value was less than half influential for the output compared to Feature_p_Value, and so on.

What is the mathematics behind such a calculation. I looked a bit at logistic regression. Would that be a right way to do? If so, then in logistic regression, I get a set of beta vectors (num of output values), each of the size of the dimension of the input (num of features). What do I do with the output beta vectors, though I know that you can predict an output given a fresh set of feature values and these beta values that are calculated using training data.

Thanks Abhishek S

You are asking for a typical "feature selection" analysis; Principal Component Analysis, or PCA, is perhaps the most common (and i think the most straightforward) technique for this type of anlysis.

It is based on an eigenvector decomposition of the covariance matrix (of the original data).

In the example below, i use Python and NumPy, the most widely used numerical computation library for Python. The syntax is very similar to Matlab's.

# generate a data set (4 explanatory variables + 1 response variable)
import numpy as NP
X = NP.random.rand(75).reshape(15, 5)

# calculate the correlation matrix of this data matrix, gives a 4 x 4 matrix
C = NP.corrcoef(X[:,:-1], rowvar=0)

# calculate the eigenvalues of the covariance matrix:
from numpy import linalg as LA

eva, evc = LA.eig(C)    # the key step

# sort the eigenvalue array in descending order:
eva1 = NP.sort(eva)[::-1]

# get value proportion of each eigenvalue:
eva2 = NP.cumsum(eva1/NP.sum(eva1))

# print/display this intermediate result in table:
NP.set_printoptions(precision=3, suppress=True)
evas = NP.arange(1, 5)
evas = evas.reshape(4,1)
eva1 = eva1.reshape(4,1)
eva2 = eva2.reshape(4,1)
q = NP.hstack((evas, eva1, eva2))
q = NP.array(q, dtype=float)

title1 = "ev value proportion"
print(title1)
print( "{0}".format("-"*len(title1)) )
for row in q :
print("{0:1d} {1:2f} {2:2f}".format(int(row[0]), row[1], row[2]))

ev value proportion
-------------------
1 1.596746 0.399186
2 1.379002 0.743937
3 0.843466 0.954803
4 0.180787 1.000000


In other words, according to Table immediately above:

• slightly less than 40% of the variability in the data is accounted for by the first variable,
• 74%, by the first two, and
• 95% of the variability is accounted for by the first three variables.

When the data from the table above is plotted using a bars to show percent, the result is usually referred to as a scree plot.

Finally, i would not use regression analysis for this problem. To begin with, it's not necessary, and second the result from applying those techniques tell you the importance of variables in your regression model, not in the data itself (which is their purpose).

• If I have to then predict a future value of the outcome, given a set of feature values, can you suggest something? Would I need to take the output of PCA and then eliminate those features from the dataset that do not account for a majority of variation? Should the dataset be cleaned this way and then may be we use some prediction algorithm? – Abhishek Shivkumar Sep 28 '11 at 4:24
• @AbhishekShivkumar : so i use PCA a lot in pre-processing prior to a supervised ML technique (e.g., SVM)--to reduce the number of features. In fact, PCA is often enough, by itself--remember, it gives you the axes of maximum variability, so not surprisingly, it is very useful as an 'unsupervised' classification technique. if you need an example, i can put one in my repo. – doug Sep 28 '11 at 5:20
• This reply seems to address a different problem than stated: the PCA explores relationships among all the variables, but it tells us little or nothing about "which features influence the output," which is explicitly a regression problem. As such you will succeed in characterizing correlation among the independent variables and--if you include the outcome in the PCA--you will also learn something about how they are correlated with the dependent variable, but you won't make much progress in ordering the IVs by degree of "influence" on the DV. – whuber Oct 21 '11 at 19:35

Yes you are going in the right direction.

This is a typical regression/correlation analysis case. There are multiple approaches to the problem, depending on the nature of the relation. The easiest is a linear dependency, the next would be functions that contain expressions like y = a * x1^n + b * x2^m... And, depending on the case, there are many more solution possibilities - all depending on the nature of the dependency.

There are many tools freely available to play around to find the best fitting function.

I always like to start with Excel fo simple problems - but there are som quite advanced special software packages available too. All depends on the complexity of the dependency and on the number of data points to verify/test

ps its real fun to look and find the best fitting function. Once you have it you can easily describe each parameters influence to the output.