2
$\begingroup$

Hyperparameter optimizations and feature engineering can(in my understanding) both be used to create a machine learning model. But what is the difference? And what is done to the y = wx + b formula in both cases?

In my understanding, the w is a hyperparameter, that is "tuned"...

$\endgroup$
  • $\begingroup$ Hyperparameter optimization is changing model parameters to improve model performance. Feature Engineering is related to data and example of feature engineering would be feature reduction. $\endgroup$ – YatShan Feb 10 at 8:43
  • $\begingroup$ Yeah your understanding is correct on hyper parameter. but when comes feature tuning nothing but variables selection you may not select all variables for your model. based on variance and correlation you will use choose the variables and then you will apply ML algorithms. Feature engineering is come under (data engineering) while Hyperparameter optimizations will come under model tuning. $\endgroup$ – venkatadileep Feb 10 at 8:46
  • 2
    $\begingroup$ I think it would be better to take a coursera class on machine learning, which would answer all your questions here. But for correctness: w is not a hyperparameter, it is a model parameter. Hyperparameters are those that are not part of the final model, but can be tuned to affect the training process and the final result. $\endgroup$ – Alex A. Feb 10 at 8:50
  • 1
    $\begingroup$ Feature engineering on the other hand, means to modify x before the training process starts, by computing new, useful variables from the original variables. $\endgroup$ – Alex A. Feb 10 at 8:57
4
$\begingroup$

Feature engineering is about data and is the process of finding/creating features that might improve your model's performance. You sometimes engineer new features from raw data you have, use the existing ones and perform univariate/multivariate transformations. So, by engineering more features your linear regression model may for example become $y=w_1x_1+w_2x_2+b$, where $x_1,x_2$ are your engineered features.

Hyperparameter optimization (HPO) is related to the model you have, not the data. Many models have hyper-parameters to be tuned. In simple linear regression without regularization we don't have any. The $w$ coefficient is a parameter of the model, as well as $b$. They're not hyperparameters. A hyperparameter can be the regularization coeffcieint, $\lambda$ in your regularized linear regression model.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

Feature engineering: working with the available data in order to create/transform good predictors (your $X$). This can usually be done by transforming, averaging, combining, etc. the available columns of your database, in order to obtain the predictors that are the most meaningful (or just work better) for the problem at hand. This can also include feature selection - where we try to reduce the number of predictors to simplify our model.

Hyperparameter Optimization: the choice of the correct hyperparameters for your model. Hyperparameters are parameters that are specific to a statistical/ML model and that need to be set up before the learning process begins. These generally will dictate the behavior of your model, such as convergence speed, complexity etc. Examples are regularization coefficients (Lasso, Ridge), structural parameters (Number of layers of a Neural Net, number of neurons in each layer, Depth of a decision tree, etc.), loss/metrics (Optimizing L2/L1 loss, or Accuracy/LogLoss/AUC), and many more depending on the model you're using.
Hyperparameters are opposed to normal parameters, such as the $w$ of your $y=wx + b$ equation, that are instead learned during the training process, by looking at the data.

The two things are separate - the first one focus on the data and the variables you have, while the second one on the setup of your algorithm.

However, some feature engineering, and in particular feature selection, can be included in the Optimization of the Hyperparameters. Indeed, the number of features to be included in the model can often be treated as a hyperparameter (in the sense that it often needs to be set up before the training), and it can be optimized within the same optimization procedure (Random Search + Cross Validation, most often) jointly with the intrinsic hyperparameters of the algorithm.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy