Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

From what I understand, we can only build a regression function that lies within the interval of the training data.

For example (only one of the panels is necessary): enter image description here

How would I predict into the future using a KNN regressor? Again, it appears to only approximate a function that lies within the interval of the training data.

My question: What are the advantages of using a KNN regressor? I understand that it is a very powerful tool for classification, but it seems that it would perform poorly in a regression scenario.

share|improve this question
    
Can you clarify what you mean by "predict into the future"? Do you have time-series & you want to do forecasting, or are you trying to fit a relationship between 2 variables & want to use that in the future to guess a Y value from a known X value? –  gung Jun 21 at 20:51
    
For example, if I wanted to predict the value Y such that X=15 from the image above. A KNN-regressor wouldn't cut it right? –  user1008537 Jun 22 at 1:38
    
I would agree with you that if you trained on a set with $x \in [0,5]$ but expected that you may see values of $x$ far beyond what is in your data then non-parametric local methods might not be ideal. Instead you might want to use that domain knowledge and define a parametric model that includes your knowledge of how 'unobserved' $x$ is expected to behave. –  Meadowlark Bradsher Jun 22 at 2:45
1  
An example of KNN being used successfully for regression is Nate Silver's PECOTA baseball prediction thing. You can read about the pros and cons from the Wikipedia article on PECOTA or newspaper articles like this one: macleans.ca/authors/colby-cosh/… –  Flounderer Jun 22 at 2:48
2  
Also to make a more general point, as you become knowledgeable in statistics (or data mining/machine learning etc) you'll find that answers to very general questions such as yours will often be a paraphrased version of 'it depends'. Knowing what 'it depends' on and why is the knowledge. –  Meadowlark Bradsher Jun 22 at 2:51

2 Answers 2

up vote 6 down vote accepted

Local methods like K-NN make sense in some situations.

One example that I did in school work had to do with predicting the compressive strength of various mixtures of cement ingredients. All of these ingredients were relatively non-volatile with respect to the response or each other and KNN made reliable predictions on it. In other words none of the independent variables had disproportionately large variance to confer to the model either individually or possibly by mutual interaction.

Take this with a grain of salt because I don't know of a data investigation technique that conclusively shows this but intuitively it seems reasonable that if your features have some proportionate degree of variances, I don't know what proportion, you might have a KNN candidate. I'd certainly like to know if there were some studies and resulting techniques developed to this effect.

If you think about it from a generalized domain perspective there is a broad class of applications where similar 'recipes' yield similar outcomes. This certainly seemed to describe the situation of predicting outcomes of mixing cement. I would say if you had data that behaved according to this description and in addition your distance measure was also natural to the domain at hand and lastly that you had sufficient data, I would imagine that you should get useful results from KNN or another local method.

You are also getting the benefit of extremely low bias when you use local methods. Sometimes generalized additive models (GAM) balance bias and variance by fitting each individual variable using KNN such that:

$$\hat{y}=f_1(x_1) + f_2(x_2) + \dots + f_n(x_n) + \epsilon$$

The additive portion (the plus symbols) protect against high variance while the use of KNN in place of $f_n(x_n)$ protects against high bias.

I wouldn't write off KNN so quickly. It has its place.

share|improve this answer

I've never used it before, but I'd consider it in the future for the following reasons:

  1. If the data is bounded and I already have observations at those bounds
  2. It's 100% nonparametric
  3. It's local, so it can pick up on weird systematic but localized features of data without structural breaks or mixtures (maybe like spectroscopy data?)
  4. For the same reason, it works on arbitrarily nonlinear data
  5. It works in an arbitrary number of dimensions without substantive changes. Dimensionality is a real curse here, but since we're talking about bounded data anyway hopefully you can just get more data if you have more dimensions.

Yes, splines and LOESS can do similar stuff. But kNN is much simpler. And kernels might be "too smooth" depending on your use case. It's a specific use case, but I can see it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.