3
$\begingroup$

For a long time now, I have been thinking about working with neural networks and genetic algorithms. I have never been able to decide whether it makes sense to start writing my own code, or to reuse the many, many options made by others.

In my experience, assuming the programmer knows what he or she is doing to a reasonable extent, there is very little that replaces C or C++ code crafted to the individual users preferences. For one thing, it empowers you to do almost anything with the code due to having complete knowledge of its workings. One can for instance incorporate whatever new advances that one happens to read about in the literature.

On the other hand, one person will often find it hard to do as much as a team of people. As an individual, it's always possible that one will miss various bugs. There are many other disadvantages to being a loner.

So, for those who have some experience in machine learning, I would like to hear what actually works in the real world. Is it more common to use a few well-known programming packages or do many people write their own custom implementations? I am specifically interested in the areas of RBM, neural networks, genetic programming and clustering.

Assume that I am willing to put a lot of time into learning about machine learning and that I will put at least as much time and effort into it as a graduate student would.

|cite|improve this question
$\endgroup$
  • $\begingroup$ I don't think this is off topic here, but I think this question may have more success on stackoverflow. $\endgroup$ – Bogdanovist Oct 29 '12 at 23:36
  • $\begingroup$ I did scan and read a few articles that covered the same question on stackoverflow, but I'm looking for a machine learning specific answer. I want to know what people who are successful in machine learning do. $\endgroup$ – Henry B. Oct 29 '12 at 23:50
  • 2
    $\begingroup$ Machine learning is a big field, and people are successful doing all sorts of things. You can work with datasets that easily fit in memory and happily make use of R and CRAN all day, or you can do deep learning neural networks using six months of time on a giant cluster at Google burning cycles as fast as possible with C++ or a similar language. I know of at least one well-known GP package written in Clojure. I think it depends on your goals. If you have a problem to solve, I'd stand on the shoulders of giants. For learning and fun or novel research, writing your own can be useful. $\endgroup$ – deong Oct 30 '12 at 0:52
  • 1
    $\begingroup$ Note that "cluster analysis" (actually, all of "unsupervised learning") is a step child to machine learning. I strongly recommend not approaching it from that point of view, which is very strongly "reproduce my labels" driven. Which doesn't really work for "unsupervised learning". Instead, clustering should be seen as an explorative method, from the KDD side, where the question is: "give me different labes than I already have". $\endgroup$ – Anony-Mousse Oct 30 '12 at 7:08
5
$\begingroup$

There are two sides to this medal.

On one hand, while there is a lot of code already around, there are also lots of bugs in this code. It may have been tested only on very small data sets, and once you run it on much larger data it becomes incredibly slow. Or it is closely tied to 2 dimensional data, to a particular data type, whatever. So when reusing existing code, you may end up discovering that it is incorrect, slow, or needs rewriting to work with your data.

On the other hand, you are likely to also make some errors or decisions that limit your codes performance. These issues may already have been fixed in the existing code. Using existing libraries may also give you functionality that you didn't deem necessary in the beginning, but find very valueable at the end. Plus, it may give you access to comparing it with other methods, visualization, debugging, evaluation etc.

My recommendation is as follows:

Do look for existing codes. But check them if they are actually written and designed for extensibility and reuse. There are some good examples: libSVM is the de facto standard for support vector machines, and it's used almost everywhere. There must be bindings for 20+ languages. Instead of rewriting SVM, this one is definitely code to reuse. Or ELKI, of which i'm quite a fan, is a Java framework for clustering and outlier detection (doesn't have machine learning yet, though). It takes a bit to get used to it, because of its mixture of object orientation and weaved-in optimization. But this makes it surprisingly fast (definitely outperforms R for me) and a lot of the stuff can be implemented in a few lines and then will still benefit from these optimizations. What makes this reusable is the modular architecture. I've written a customized distance function, and can use it it all the algorithms, and written an algorithm where you can plug in arbitrary distance functions. I've tried these things in R before, but either the module was limited to Euclidean distances and such, or it would need a distance matrix in memory, which takes $\mathcal{O}(n^2)$ memory and time. Plus, in the end I can do thousands of combinations distance+algorithm+index for comparison to see if my new stuff is good for anything. For example I learned this way that Canberra distance works surprisingly well across a number of scenarios, and would often be a much better choice than Euclidean distance.

On the other hand, there is a lot of code on the interwebs that is crap. I was looking for the OPTICS clustering algorithm. I found Matlab and Python versions: http://chemometria.us.edu.pl/download/optics.py but both of them were crap. Slow, and the result was essentially a DBSCAN result. OPTICS in Weka has a nice plot view, but doesn't really cluster, and it's incredibly slow. The ELKI version of OPTICS was a completely different league! I figure that someone just dumped whatever they had arrived with, but the code was never reviewed or even properly tested. The python version apparently is based on the Matlab version, which maybe was transscribed from the incomplete Weka version. Ouch!

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ (+1) I always check the number of downloads and the date of the last update ... if I do not have enough time to play, all small and abandoned projects are immediately dropped. Aside, reading the code of others is one step to become a better programmer. $\endgroup$ – steffen Nov 2 '12 at 14:15
1
$\begingroup$

Implementing a neural network is not really that technically challenging, especially if you are planning training them with a genetic algorithm instead of backpropagation or something. You can use this tutorial as a good starting place if you want to go that route.

In general I find that until I have implemented an algorithm I will not understand the details that I glossed over while reading about the paper. If your purpose is to really understand neural networks, I would whole heatedly recommend that you implement it. Once you have something in place, go implement backpropagation. Add in momentum for better training times. Use the implementation as a learning experience.

On the other hand, if you only want to run the algorithm on some data and see results, it is probably best to just take some off the shelf implementation and run with it. There are lots of packages in pretty much any language you want which can allow you to quickly get a neural network up and running.

In short, whether you implement an algorithm or use an off the shelf package depends on whether you are trying to dive deep into the details of the algorithm or just want to tinker.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

I absolutely recommend you to forget putting effort on C/C++ for this purpose, and convert to R programming language. In R, there are huge number of packages already developed and freely available for any purpose and you can reuse them easily.

Doing a hard-to-develop statistical test, running a Principal Component Analysis, learning a Neural Network or finding the Support Vector Machine are all done with SINGLE commands!

For the areas you have mentioned there is almost no need to develop the basic things in R. You should only develop your car on the base of available wheels!

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Maybe I should have said in my question, but I know fair number of languages:R, Matlab, Python, Mathematica $\endgroup$ – Henry B. Oct 30 '12 at 0:11
  • 1
    $\begingroup$ @HenryB. Answer updated $\endgroup$ – Ali Sharifi Oct 30 '12 at 0:15
  • $\begingroup$ I wouldn't want to do GAs or GP in R. Those types of programs are full of loops that can't be easily converted into operations that R won't be dog-slow at, and they're also areas where even a 2-3x speedup can be huge. You can easily spend hours on a single large problem, and if you want to do research in the field, you'll be running each test 30 or so times. $\endgroup$ – deong Oct 30 '12 at 0:46
  • $\begingroup$ It depends @deong. In many cases, when you run a 3rd party package 'in R' what you are really using is an R interface to a C/C++/FORTRAN code that will not suffer the performance limitations of a native R code that you speak of. Of course if you are playing around with your own implementation then a native R code will be horribly slow and horrible in all sorts of other ways since R is an awful tool for all purpose programming. It really depends on what you want, which is why this question has no real answer. $\endgroup$ – Bogdanovist Oct 30 '12 at 2:46
  • $\begingroup$ A lot of the R code is actually written in C, with just an R api. This has the consequence, that you can barely combine modules, except by feeding the output of one as input to another. And quite often, you may need to add some data transformations inbetween. I think R could be a lot better if the modules were better combine- and reusable. $\endgroup$ – Anony-Mousse Oct 30 '12 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.