# Multiple Regression and variable optimization

I'm seeking a means to find the optimal values for a set of variables in MLR. I've seen other posts which suggest using Linear Optimization. I have already fit a model and have the coefficients. Now, my goal is in order to achieve a specific DV value of Y, what is the best way to calibrate (select the optimal values) of each of the IV values? One might suggest to just increase the value of all IVs, but this comes at a cost. Hence the thought to use linear optimization.

EDIT: Let's say this is a marketing problem and let's say I have built a model that predicts a proprietary sentiment score from a set of IVs, including things like spend on advertising, number of advertisements, number of local markets in which to advertise, etc. - all continuous variables (no categorical). I have a data set consisting of 5000 data points collected over a period of time. (let's ignore any time series aspect for the time being). The MLR model says that each variable in the model is statistically significant, and I have beta weights / SE for each. for the sake of argument, let's say the model looks like this:

y^ = C + .876B1 + .435B2 + 1.23*B3, where: B1 = advertising spend B2 = num. advertisements B3 = num. local markets

Now, I want to know what is the optimal value of each variable (B1, B2, B3), such that is maximizes the sentiment score (Y^). It's not enough to know that the variables are statistically significant predictors, and one cannot simply just maximize all of them - because there is a cost to doing so, and funds are not unlimited.
Logically, one could just take the variable with the largest coefficient and "maximize that"; but in the real world, that's not how things are done - especially in marketing/advertising. I'm seeking options from what others have done to solve these kinds of problems.

I'm using Python for everything... No code to share yet, just thinking through options. If anyone has ideas to share or suggestions, I would appreciate it.

• It's not clear what you are trying to do. What kind of model have you already fitted ? What are you trying to optimize and why ? What is your research question / or the problem you are trying to solve. And please describe the data Commented Sep 27, 2020 at 17:39
• Hi Robert - I added some context to make the hypo more real. Commented Sep 27, 2020 at 18:24
• @RobertLong After reading the edit, does this sound to you like a Lagrange multiplier (maybe Karush-Kuhn-Tucker) problem where the regression equation is optimized subject to cost constraints? // To the OP: if that sounds about right, you might be interested in the operations research Stack for this kind of optimization problem. Optimizing a regression equation doesn’t quite make sense, as many are hyperplanes that shoot off to $\infty$. (We could find a maximum value if, for instance, there is a quadratic term, however.)
– Dave
Commented Sep 27, 2020 at 18:38
• I was inspired by this: stats.stackexchange.com/questions/475639/… However, I understand what you mean about the hyperplanes. I've not come across anyone else attempting to solve what seems like it should be a common problem. Commented Sep 27, 2020 at 18:52

Once you've finalized your MLR model, I would view this as the objective function within the context of an optimization problem.

I would then define appropriate constraints for B1, B2 and B3 and use linear programming (or simulation) to maximize the sentiment score accordingly.

There are quite a few Python packages to do this using linear programming - I've only just started using Python to do this and am currently experimenting with Gekko as it has easy support for using sparse matrices:

https://apmonitor.com/pdc/index.php/Main/LinearProgramming

And there are many ways to run a bounded simulation of the optimization solution space in base Python.

Would be interested to hear how you get on...

A regression model can be seen as a function: you input feature values, and the model outputs a fitted value (prediction). Like any other function, you can pick an output value and solve for the input feature value(s) that give such a value, if any exist.

Your function might be bounded from above, but it might not be. Consequently, it need not make sense to talk about a maximum. However, if there is a limit to the mix of feature values that are within your budget, then you have a constrained optimization problem where you want to find the feature value(s) maximizing the regression output, subject to the constraint that the cost of those feature values is within to it budget, and this makes sense whether your regression has an upper bound or not.

Formally, let $$p_i$$ be the price of feature $$i$$, and let $$B$$ be the budget.

$$\underset{(x_1,\dots,x_k)\in\mathbb R^k}{\arg\min} \left\{\hat\beta_0+ \hat\beta_1x_1+\dots\hat\beta_kx_k\Big\vert p_1x_1 +\dots + p_kx_k \le B \right\}$$

In words, this is saying that you want to find the feature values such that the regression output is maximized without those feature values exceeding the available budget. This is related to the Walrasian or Marshallian demand in economics.

The major techniques for solving a problem like this are the Karush-Kuhn-Tucker theorem and Lagrange multipliers. There must be Python packages that implement this kind of constrained optimization.

Note that the uncertainty in estimating the regression coefficients could impact your results, as is mentioned here. It might be that your coefficient estimates are so imprecise (wide confidence intervals) that, if you solve the constrained optimization problem for many reasonable combinations of regression coefficients, your optimal bundles of feature values bounce all over the place. I might be interested in how the constrained maximum varies upon bootstrapping, fitting a regression to the bootstrap sample, and solving the constrained optimization according to the bootstrap coefficients. If these values tend to give values that are acceptable, that would be reassuring. If the values are all over the place and often are too low, that is concerning.

In economics, a related idea to the optimization discussed above, utility maximization, is expenditure minimization: what is the least amount that must be spent in order to achieve a utility of at least the required amount. If you have a target in mind for the sentiment score, you can solve a similar (but different) constrained optimization problem to figure out the combination of features that minimizes the amount you must spend. (Can you figure out the set for which you find the $$\min$$ or $$\arg\min?$$) This is related to Hicksian demand in economics.