Determining the influential features for an outcome

Question

I have a small table like this

Outcome     Feature_1_Value  Feature_2_Value   Feature_3_Value   .....
   1               1              1.2               3.7
   9.7             2.3            3.976             4.65
   ....            ....           ....              ....

There is an outcome that depends on what values do a set of features have. The above table represents the training data. The outcomes and the features all are continuous values and are not discrete ( it is not 0/1/2 kind of values).

Now, how do I analyze the training set of data to determine which of the features actually influence the output and how do I re-arrange the features in decreasing order of their influence? I would like to represent nicely to the end-user saying Feature_n_Value was twice more influencing the output than Feature_m_Value, but Feature_n_value was less than half influential for the output compared to Feature_p_Value, and so on.

What is the mathematics behind such a calculation. I looked a bit at logistic regression. Would that be a right way to do? If so, then in logistic regression, I get a set of beta vectors (num of output values), each of the size of the dimension of the input (num of features). What do I do with the output beta vectors, though I know that you can predict an output given a fresh set of feature values and these beta values that are calculated using training data.

Advice is highly appreciated.

Thanks Abhishek S

Answer 1

You are asking for a typical "feature selection" analysis; Principal Component Analysis, or PCA, is perhaps the most common (and i think the most straightforward) technique for this type of anlysis.

It is based on an eigenvector decomposition of the covariance matrix (of the original data).

In the example below, i use Python and NumPy, the most widely used numerical computation library for Python. The syntax is very similar to Matlab's.

# generate a data set (4 explanatory variables + 1 response variable)
import numpy as NP
X = NP.random.rand(75).reshape(15, 5)

# calculate the correlation matrix of this data matrix, gives a 4 x 4 matrix 
C = NP.corrcoef(X[:,:-1], rowvar=0)

# calculate the eigenvalues of the covariance matrix:
from numpy import linalg as LA

eva, evc = LA.eig(C)    # the key step

# sort the eigenvalue array in descending order:
eva1 = NP.sort(eva)[::-1]

# get value proportion of each eigenvalue:
eva2 = NP.cumsum(eva1/NP.sum(eva1))

# print/display this intermediate result in table:
NP.set_printoptions(precision=3, suppress=True)
evas = NP.arange(1, 5)
evas = evas.reshape(4,1)
eva1 = eva1.reshape(4,1)
eva2 = eva2.reshape(4,1)
q = NP.hstack((evas, eva1, eva2))
q = NP.array(q, dtype=float)

title1 = "ev value proportion"
print(title1)
print( "{0}".format("-"*len(title1)) )
for row in q :
    print("{0:1d} {1:2f} {2:2f}".format(int(row[0]), row[1], row[2]))

    ev value proportion
    -------------------
    1 1.596746 0.399186
    2 1.379002 0.743937
    3 0.843466 0.954803
    4 0.180787 1.000000

In other words, according to Table immediately above:

• slightly less than 40% of the variability in the data is accounted for by the first variable,
• 74%, by the first two, and
• 95% of the variability is accounted for by the first three variables.

When the data from the table above is plotted using a bars to show percent, the result is usually referred to as a scree plot.

Finally, i would not use regression analysis for this problem. To begin with, it's not necessary, and second the result from applying those techniques tell you the importance of variables in your regression model, not in the data itself (which is their purpose).

Answer 2

Yes you are going in the right direction.

This is a typical regression/correlation analysis case. There are multiple approaches to the problem, depending on the nature of the relation. The easiest is a linear dependency, the next would be functions that contain expressions like y = a * x1^n + b * x2^m... And, depending on the case, there are many more solution possibilities - all depending on the nature of the dependency.

There are many tools freely available to play around to find the best fitting function.

I always like to start with Excel fo simple problems - but there are som quite advanced special software packages available too. All depends on the complexity of the dependency and on the number of data points to verify/test

ps its real fun to look and find the best fitting function. Once you have it you can easily describe each parameters influence to the output.