1
$\begingroup$

I have been learning about Machine Learning (via Udacity) and Statistics (via Coursera) the past few months and trying to figure out a good way to combine them for a general approach to explaining data trends. I am aware so much of ML and stats rely on domain knowledge, and that without any domain knowledge we easily encounter the results of the "No Free Lunch" theorem. As such, I have incorporated domain knowledge in my approach accordingly.I walk through a simple example of stock market volumes below.

I know the answer to this question is "It depends on the data and a variety of factors", but it I wanted a general rule of thumb on how to approach a situation, does a general approach such as the below work?

Sample case study: The stock market is higher in year X compared to year Y, why?

Approach to analyzing the trend:

1) Exploratory Analysis: Using my domain knowledge of the stock market it is reasonable to look at features X_1, X_2,..X_n. I then perform exploratory data analysis on each of the features relative to the stock market value (ie, how has the volatility market place changed over time relative to the value of the Nasdaq, for example). I do this for each one of the features I thought was worth investigating to get a feel for the data.

2) Feature Selection : Use either information gain, or regression (or both) to see determine which features have the highest predictive power. Take the top n features and use them in the next step.

3) Hypothesis Tests: Run hypothesis tests on the data to try to see if each of the individual features had an impact on the market volume change.

4) Supervised Learning: With fewer features, we need less data to generalize our findings and are less likely to falling prey to the curse of dimensionality.So we can then can train our historical data on a decision tree, use cross validation, and generate a model. We can then use this model as a heuristic to see what feature had the largest impact on classifying an "up" stock market instead of a "down" stock market.

Is this a reasonable way to look to analyze events that have transpired (in this hypothetical example, the "events" being that the stock market went up)? I am trying to integrate all of the content from my introductory courses in a way that can be practical and aid in drawing insights from data. This is a general question having nothing to do with stock market volume, I just wanted to use an example for the sake of discussion if it made life easier.

$\endgroup$
0
$\begingroup$

Sure, that is pretty reasonable as a general approach. That said, you don't necessarily do all of those steps in that order for every single problem.

If you were an economist, you might start with the exploratory data analysis, skip the feature selection (or use your domain knowledge/research question to pick the features), and finish up with some hypothesis testing. If you were going to fit a model, you would probably use a regression model rather than crudely discretizing your data into "going up" and "going down."

On the other hand, for some engineering problems, one might not care about the hypothesis testing. If the system works (ideally on some held-out data), then it works, even if the data set isn't large enough to do some high-powered statistical inference. Natural language or image processing systems probably don't have enough data to measure per-lexeme or per-pixel significance--and no one would care even if they did.

A few other things

  • You may not want to consider features individually. In many cases, the information is carried by a combination of two or more features. The Fisher' Iris data set is a nice example of this. You can tell Iris setosa apart from the other species on the basis of a single feature (petal width), but you need combination of features to tell the other two species apart.

  • You need a way to measure performance/overfitting, whether it's using a held-out data set, using cross-validation, or a bootstrap. When you do so, you should think about this as measuring the performance of your entire pipeline (feature selection through prediction), so this should be wrapped around steps 1-4.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.