Do Parameters optimization more feasible for industrial purpose than academic?

Question

We use hyperparameter optimization to increase the performance of the data miners, however most of the researchers don't use the parameters tuning as it needs a lot of time and effort.

In academic research, since we work with many datasets, in order to save time, we use default parameters and compromise on the accuracy. In industries, however, since there are few datasets and accuracy is required more than saving the time, so can we suggest that hyperparameter tuning is recommended for industrial purpose while for academic research, the default parameters are advisable?

Answer 1

I see what you're saying, but the answer is, not necessarily.

In fact the opposite might occur -- in an industrial application, you might be able to collect more data, or improve your sensors, so hyperparameter tuning is not the only way to improve the performance of your system. In academic research, since people generally compete on a set of standardized datasets which you're not allowed to modify, there's fewer of these "tricks" you can pull, and there's a lot more incentive to tune and tune until you've beat the competition.

You could even say that accuracy (or your metric of choice) matters more in academic research than industry, because in industry you probably just want to beat the previous implementation of the system (or if there was no previous implementation, then anything reasonable will do). In academic research, you might not have a publication unless your model beats the models of 100 other people working on the same problem, and everyone else is tuning parameters, so you have to as well.

Answer 2

Indeed hyperparameter tuning is very important in research papers most times.

The reason is that the research papers should be novel, which means that "default value" is hardly available since your method is a new one and has different parameters from the old methods, so you must tune your parameters carefully. Then the tuned parameters should be published to make a research reproducible, which is the default parameters if someone follows your work in the future.

I think what makes you think academic researches do not need much tuning is that they should be comparable to other papers. In this case, the datasets and the pre-trained model (e.g. vgg or resnet) should be exactly the same, as well as the hyper-parameters of the shared parts. This is mostly a requirement to make a fair comparison rather than a property of academic research.

Most papers do not provide any information on how hard they tune the parameters, but the parameters are indeed well-tuned unless they refer to another paper that provides these parameters. This will be very clear if you have tried to reproduce the results of several papers from scratch, it's almost impossible to get comparable performance if you use the default value from torch or tensorflow or choose a random one.

Edit: When it comes to industry, the performance is also crucial. However, this performance may be different from academic research. I'm still a student and only have some experience in collaborating with industry. I found that they only care about how much money you can earn or save for them using your technology, and your method must be directly usable (most research paper is not and will not be used). The novelty is not important and what you need to do is just hyperparameter optimization on a bunch of old methods, which makes tuning means more in industry.

When there is a private dataset in a company or the task is very hard, hyperparameter optimization is more feasible for industrial guys. However, they may not for industrial purposes, like alphago or alphastar. The Deepmind or FAIR often does not care about practicability because they are research departments, otherwise, this may still not possible. Datasets and computation resource are not free, they are very expensive, and industries live on the money.

If the question is: When does tuning more feasible for industrial purposes than academics?

I think there will be a clear answer: When we only care about the absolute performance (only compare performance, not methods) for an application problem (can make money, such as face recognition). The researchers can't but also don't need to do hyperparameter optimization on a private dataset because this is only related to absolute performance and makes their comparison unfair. The computation resource is similar and it's not about feasibility if we compare on easy problems.

In conclusion:

1. Hyperparameter optimization is more feasible in industries for a very large dataset (e.g. imagenet) or very hard problems (e.g. starcraft or go) because no one in college has such resources.

2. In industries, they may do hyperparameter optimization only for academic purposes. Deepmind or FAIR can spend a lot only to win a game.

3. Industry care about money at most time, which means that if you can't convince your boss that your work can save or earn money, you can't do anything. Collecting a dataset or running experiments on a large cluster is very costly, so you may still have no chance to do hyperparameter optimization.

4. Hyperparameter optimization is crucial in academic, in almost every paper. We just choose a smaller dataset or easier task to make it possible.

5. On absolute performance, the larger the dataset, the more the resource spent on tuning, the better the result. This is what the industry cares about because they compete in the wild world. On the contrary, researchers care about relative performance, this is the difference.