Exact Procedure for KNN classification - Cross Validated most recent 30 from stats.stackexchange.com 2019-07-16T02:23:02Z https://stats.stackexchange.com/feeds/question/175689 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/175689 0 Exact Procedure for KNN classification Turbo https://stats.stackexchange.com/users/78031 2015-10-06T11:53:16Z 2018-10-29T04:00:41Z <p>I want to know the exact procedure involved in KNN classification. I understand the bigger picture but I miss the details to implement.</p> <p>I have 3 pieces of data: Train, Validate and Test.</p> <p>1) Suppose we have training points $x_1, x_2,\dots,x_N$ each in $\Bbb R^D$ where $D$ is number of features and the labels are $y_1,\dots,y_N$ each in $\Bbb R$ where $N$ is number of training points.</p> <p>What does training involve? Do I need to pick nearest neighbors for each point $j\in\{1,\dots,N\}$ and relabel the points based on majority vote?</p> <p>2) What does validation involve?</p> <p>Given a validation point $x\in\Bbb R^D$ with label $y$ what should I do? </p> <p>3) Is testing same as validation?</p> https://stats.stackexchange.com/questions/175689/-/175698#175698 1 Answer by Erik for Exact Procedure for KNN classification Erik https://stats.stackexchange.com/users/10524 2015-10-06T12:44:22Z 2015-10-06T12:44:22Z <p>1 and 2) Training consists only of saving your training points. If you apply the model to a new data point, you find the neighest neighbours of the new point in the set of training points.</p> <p>3) The validation step of your procedure is often used for tuning hyperparameters, for example with nearest neighbours this could be the number $n$ of neighbours to consider. </p> <p>If you want to tune $n$ you can do the following:</p> <ol> <li>Save your training points</li> <li>For each $n$ in a reasonable range (e.g. 1-10, depends on dimension of data) classify all your validation points with $n$-Nearest Neighbours. Look at an appropiate measure of the performance of your model (application specific). Pick the $n$ which yields the best results.</li> <li>Save a set consisting of both the training points and the validation points.</li> <li>Use a single $n$-Nearest Neighbours model with the n to predict your test data using the data from step 3. The performance of that model on the test data is your final estimate of the performance of your model.</li> </ol> <p>If you are not tuning parameters you do not need a triple split. Note that considering which type of classifier performs best is also tuning a hyperparameter.</p> <p>In either case, iterated nested cross validation or bootstrapping gives more accurate estimates compared to this procedure which is rather inefficient. </p>