I am working on a problem where my objective is to predict y given some features x1,x2,x3,...x8,x9 I solved this problem using some statistical and machine learning techniques like regression, trees, random forests & svm. Now that I have a prediction for y, at a given x1,x2,x3..x6 I would like to achieve an optimal value of y, by changing some values of xn which are in my control. Let us say that y was predicted to be 5, however I need a value of 10. Can I put three features aside say x1,x2,x3 and get like a range or values for the aforementioned aside features such that the value of y is 10?

Basically, it is sort of like an inverse problem, where assuming I know the predictor I need to manipulate the features to increase the value of the predictor.

Reproducible example:

    y<- rnorm(100)
x1<- sin(rpois(100))
x2<- cos(rnorm(100))
x3<- sin(rnorm(100))+ rnorm(100)* 3cos(rnorm(100))
x4<- rnorm(100)
y.fit<- lm(y~x1+x2+x3+x4)
y.rf<- train(ROP~ .,data=training,method="rf",prox=TRUE)

So now that I have y.rf and y.fit, lets say i have control over the values of x1 & x2, hence I would like a given value of y say 0.5, and to achieve this value of y (0.5) at a fixed value of x3,x4 I would like a range for x1 and x2 or possible values for x1 & x2.

How should I proceed?


2 Answers 2


From what I understood, you would like to modify the features given the prediction label. I would say that this problem is pretty ill-posed. Why? Imagine you have three distinct data clusters where each cluster is a class. Obviously each data point belonging to its class carries a label from that class. The important point here that there are different points from one particular class carrying the same label but with different features. With no additional info you cannot go back from label to features. One straight forward way would be to map from label to class centroid. If instead of a label you had some measure (like a probability) belonging to that cluster and you had fit a Gaussian (for example) on that class, you could get the corresponding feature... But this is pretty hacky, not sure about applications.

  • $\begingroup$ Think about it from an engineering perspective, where you would like to get you set the speed of the wood slicer at 60 slices per minute, and one can change a few parameters like the speed of the chopper, sharpness of chipper teeth etc. , but some parameters are not changeable like the kind of wood. So after prediction of the speed of cutting the wood, I would like to achieve an optimal speed, so I would like to know what level I have to set the other parameters so that I can achieve this. Basically using the statistical model as a formula and using the inverse option! $\endgroup$ Mar 26, 2015 at 18:02

It is rather difficult to get your exact answer, but I can see a quick improvement to help you get closer to what you'd like.

First rather than trying to find how to tweak your features to get y from 5 to 10, you can calculate the gradient, or more simple the sensitivity of y to each of your features at 5. This way you can know 'locally only' how to move your factors to move slightly closer to 10 from 5. If your relationships of your factors to y is simple enough, then you may be able to iterate this to convergence.

This has several issues. For one, using your example, is it not known whether the factor values that set y=10 are unique or if they even exist.

Another issue, and implied by xeon above, is that your estimation of y is a a distribution and not a deterministic value. Because of this, when you are fitting you need to know what you are optimizing for. Do you want to find the factor levels where your mean estimate of y is 10, or median estimate, etc...?

  • $\begingroup$ That makes a lot of sense, I am not sure that my model is simple enough, but I would like to give the sensitivity thing a try. How would I go about calculating the sensitivity of my features at y=5 or at y=10? $\endgroup$ Mar 26, 2015 at 23:35
  • $\begingroup$ It will be slightly different for each model used. However on the simpler models like linear regression, the sensitivity is simply the coefficients (Betas) in the model y = beta*x. For more complicated models, it might be enough for you to just calculate the derivative of y with respect to the factor you care about. $\endgroup$
    – closedloop
    Mar 27, 2015 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.