11
$\begingroup$

I'm a newcomer in Machine learning (also some statistics), been learning knowledge (supervised/unsupervised learning algorithms, relevant optimization methods, regularizations, some philosophies (such as bias-variance trade-off?)) for a while. I know that without any real practice, I would not gain deep understanding of those machine learning stuff.

So I begin with some classification problem with real data, say handwritten digit classification (MNIST). To my surprise, without any feature learning/engineering, the accuracy reaches 0.97 using random-forest classifier with raw pixel values as input. I also tried other learning algorithms, such as SVM, LR with parameters being tuned.

Then I got lost, would it be too easy or am I missing anything here? Just pick up a learning algorithm from the toolkit and tune some parameters?

If that would be all about machine learning in practice, then I would be losing my interest in this field. I thought and read some blogs for a few days, and I came to some conclusions:

  1. The most important part of machine learning in practice is feature engineering, that is, given the data, find out better representation of features.

  2. Which learning algorithm to use is also important, also the parameter tuning, but the final choice is more about experimentation.

I'm not sure I understand it correctly, hoping anyone can correct me and give me some suggestion about machine learning in practice.

$\endgroup$
  • 1
    $\begingroup$ I don't think I could answer your question in its entirety, but I'd like to reassure you that there are plenty of problems where the algorithms/tools are unsatisfactory in addition to poor feature quality (in my experience the social sciences are rife with both situations). I don't think that that's your only concern, but you seem discomforted by the ease of your projects so far. $\endgroup$ – user44764 May 21 '14 at 1:08
  • 1
    $\begingroup$ @Matthew, yes, I just got lost without knowing where to start when I'm facing one real problem, should I analyze the data to find out the nature so that I could extract useful features or just pick up a learning algorithm and run? $\endgroup$ – avocado May 21 '14 at 2:17
  • $\begingroup$ I would not board a plane that fly correctly 97 times out of 100. $\endgroup$ – lcrmorin May 21 '14 at 17:02
12
$\begingroup$

Machine learning (ML) in practice depends on what the goal of doing ML is. In some situations, solid pre-processing and applying a suite of out-of-the-box ML methods might be good enough. However, even in these situations, it is important to understand how the methods work in order to be able to troubleshoot when things go wrong. However, ML in practice can be much more than this, and MNIST is a good example of why.

It's deceptively easy to get 'good' performance on the MNIST dataset. For example, according to Yann Le Cun's website on MNIST performance, K nearest neighbours (K-NN) with the Euclidean distance metric (L2) also has an error rate of 3%, the same as your out-of-the-box random forest. L2 K-NN is about as simple as an ML algorithm gets. On the other hand, Yann, Yoshua, Leon & Patrick's best, first shot at this dataset, LeNet-4, has an error rate of 0.7%, 0.7% is less than a fourth of 3%, so if you put this system into practice reading handwritten digits, the naive algorithm requires four times as much human effort to fix its errors.

The convolutional neural network that Yann and colleagues used is matched to the task but I wouldn't call this 'feature engineering', so much as making an effort to understand the data and encode that understanding into the learning algorithm.

So, what are the lessons:

  1. It is easy to reach the naive performance baseline using an out-of-the-box method and good preprocessing. You should always do this, so that you know where the baseline is and whether or not this performance level is good enough for your requirements. Beware though, often out-of-the-box ML methods are 'brittle' i.e., surprisingly sensitive to the pre-processing. Once you've trained all the out-of-the-box methods, it's almost always a good idea to try bagging them.
  2. Hard problems require either domain-specific knowledge or a lot more data or both to solve. Feature engineering means using domain-specific knowledge to help the ML algorithm. However, if you have enough data, an algorithm (or approach) that can take advantage of that data to learn complex features, and an expert applying this algorithm then you can sometimes forego this knowledge (e.g. the Kaggle Merck challenge). Also, sometimes domain experts are wrong about what good features are; so more data and ML expertise is always helpful.
  3. Consider error rate not accuracy. An ML methods with 99% accuracy makes half the errors that one with 98% accuracy does; sometimes this is important.
$\endgroup$
  • 2
    $\begingroup$ +1, thanks so much. As I read your answer, if I don't have domain-specific knowledge to do feature engineering, then I should focus on inventing good methods to do feature learning using the given data, and try different learning algorithms. $\endgroup$ – avocado May 21 '14 at 23:44
  • $\begingroup$ @loganecolss Or you could also read what other people have done on similar data or talk to a domain-expert. $\endgroup$ – qdjm May 22 '14 at 4:43
7
$\begingroup$

I think that the examples that you find on blog or websites are examples where it is known that the common methods work well (even if, of course, they can be improved).

My specialization is in features engineering and I can tell you that often the standard algorithms do not work well at all. (I have not any knowledge of the field but often I work with people who have it.).

Here there is a real problem where I worked for 6 months : Given a matrix X with 100 samples and 10000 variables representing the genetic value of patients and an output y of size 100 x 1 that represents the density of the bones.

Can you tell me which genes influence the density of the bones?

Now I am working on another problem. I have a manufacturing production dataset with 2000 samples and 12000 variables. My boss would like to extract from this dataset non more than 30 variables in an unsupervised way.
I have tried some algorithms but I can not choose less then 600 variables because these are very very correlated between them. (I am still working on this...)

Another important think to consider is the speed performance of the various algorithms. In a lot of situations you can not wait 20 minutes waiting for a result. For example you need to know when to use NIPALS and when to use SVD to compute PCA.

Hope this can give you an idea of the problems that are common in ml.

$\endgroup$
  • $\begingroup$ Thanks for your advice, seems like I didn't approach $n<<p$ problems like that, interesting. $\endgroup$ – avocado May 24 '14 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.