I have 3 numeric variables $x$, $y$, and $z$. Specifically $x$ , $y$ are tachometer readings. I am given $x$ and $y$ and I need to predict $z$ (the velocity of a vehicle). I have built a model that uses standard methods (random forest) and this achieves a decent ($80\%$) accuracy. It achieves this despite the dependence of my $z$ values.

However I have since discovered that there is a function $f$ $st$ $f(z_{t-1}) = z_{t} $ which has a $85\%$ accuracy (higher than my model at current).

So is there a way I can use this function to improve the accuracy of my classifier. Baring in mind the fact that I never actually gather a value for velocity, I can only ever predict it.

I thought that maybe using lagged variable might work, but since $f$ in decidedly non-linear I am not sure if the model would be able to cope, and it seems slightly lacking in elegance.

My next thought would be to add a feature of $f(x)$ or $f(y)$ but again since $f$ is non linear this may not work.

My last idea would be to use something like the Kalman filter to join both the prediction based on $(x_{t}$,$y_{t})$ and the prediction based on $z_{t-1}$ but I feel this may lead to error propagation.

So my questions are: Does what I am doing have an actual name (i.e what could I type into google to get either papers or other questions about this topic)?

Would any of these approaches work?

Does anyone have any better ideas?

  • $\begingroup$ Could you provide additional information? What do you mean with 'non-time' data in the title of your question? And why are you interested in elegance when it comes to prediction, accuracy is the think you should care about... In general time-series prediction tasks I would consider doing research in the field of state-space models (e.g. Kalman Filter), you could also investigate Granger-Causalities, work with Neural Networks...there is simply too much do you give a precise answer without further specification. $\endgroup$ – muffin1974 Aug 23 '16 at 9:07
  • $\begingroup$ @muffin1974, By non-time data I meant that the variables I am training my models on are not dependent on time (specifically they are readings from an tachometer). And I am using these readings to predict the velocity of a vehicle (the z value). But separate from all of this, I have a function that gives an estimate of the velocity if it is given the previous velocity ($f$). I am trying to come up with a way of combining them without causing error propagation. I never actually gather the value of the velocity I can only ever predict it from x and y $\endgroup$ – Fiery Chicken Aug 23 '16 at 9:57
  • $\begingroup$ How did you come up with the function $f(z_t)$ ? Did you learn it from data? If so, why not fit a full model like $z_t = f(z_{t-1}, x_t, y_t) + \epsilon_t$ where you use something to find the best $f(..)$? $\endgroup$ – horaceT Aug 24 '16 at 21:17
  • $\begingroup$ @horaceT No I didn't learn it from the data although I have checked and the data I have in my training set does abide by the function $\endgroup$ – Fiery Chicken Aug 25 '16 at 5:19
  • $\begingroup$ @FieryChicken. I think the general term for what you're trying to do is NARX. This can also be done with a particular neural network architecture. $\endgroup$ – Alexander F. Aug 25 '16 at 16:17

Have you tried Gaussian Processes?

A random function $f$ is said to be a Gaussian Process if $$ f(x) \sim GP(m(x), k(x, x')), \quad x, x' \in \mathbb R^p. $$

The mean function $m$ is the location parameter and the kernel function represents the covariance between two outputs as function of the inputs: $$ k(x, x') = \mathrm{cov}[f(x), f(x')]. $$

You can encode your prior information about the dependence of $z$ on the previous value in the kernel function. This means that you have to assign a higher correlation value when the points are neighbouring. The Markovian kernel is the Exponential kernel (or Ornstein-Uhlenbeck). The GP will learn itself the nonlinear dependence in the output space.

If you are not familiar with the model, you need to have a look at Kevin Murphy, Machine Learning A Probabilistic Perspective, MIT - chapter 15.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.