Questions tagged [k-nearest-neighbour]
A non-parametric method of classification and regression. The input consists of the $k$ closest training examples in the feature space. The output is either the mode of the neighbors (in classification) or their mean (in regression).
571 questions
0
votes
0
answers
25
views
Is attention a kind of K nearest neighbors regressor?
I came up with a strange KNN algorithm that I think is equivalent to self-attention:
Suppose we have a typical (features, label)-style dataset $\mathscr D = \{(x_1, Vx_1), (x_2, Vx_2),\dots, (x_N, ...
- 342
0
votes
0
answers
18
views
Is neighbour search better than approximate neighbour search?
If I have the following:
200k vectors
Find nearest neighbour among 100 vectors
Should I be just using NN and calculate distance at request time or use a vector database for ANN?
I can hold 200k ...
- 1
3
votes
1
answer
143
views
Modelling spatial autocorrelation with GAMs
the topic of spatial autocorrelation (SA) within the context of generalized additive models has already been discussed in several posts within this forum, see e.g.
Why does including latitude and ...
- 199
0
votes
0
answers
21
views
Weighting (rescaling) the dimensions (features) in $k$ nearest neighbors
When using $k$ nearest neighbors,one have to normalize the data set ( if X is the matrix of data such that every row represents a feature or dimension and every column represents a sample point, then ...
2
votes
1
answer
65
views
Are there strategies for measuring accuracy of Euclidean distance-based similarity without ground truthing?
I have subjects with about 200 features each. These feature vectors are stored in a vector database, where similarity searching with Euclidean distance is used to find subjects that are similar to a ...
- 23
1
vote
0
answers
31
views
I was working on this machine learning question and I dont seem to get how to solve it! [closed]
I have tried doing it so many ways, but none seem to work. I don't understand these LOOCV errors
- 11
1
vote
2
answers
70
views
KNN K = 1 Training on itself vs K > 1
When training a $KNN$ algorithm, why is that with $K = 1$, the model trains using the "1 nearest observation to each training point" and treats this as itself resulting in a training error ...
- 191
0
votes
0
answers
26
views
2
votes
0
answers
51
views
KNN-regression vs Linear regression
Is there any assumption on data or any number of k, that makes kNN-regression equivalent to linear regression?
- 97
0
votes
0
answers
10
views
Handling valid missing data in prediction model features
I'm developing an elastic net model using caret, with k-nearest neighbours to handle missing data in the features.
Some features are conditional on others such that a missing value is valid e.g., ...
- 3
1
vote
1
answer
75
views
The error-rate in "The elements of statistical learning"
This picture is from the book "the elements of statistical learning":
I am wondering how the test-error rate is calculated based on how the describe the simulation at the start? How do they ...
- 164
0
votes
0
answers
19
views
Is there some sort of multilevel KNN/ML model I can use to figure out which users will buy specific products?
I am wondering if there is some sort of multilevel model that I can use to identify likely buyers of specific products or create a lookalike audience.
The issue is that I have 1000s of products and ...
- 65
0
votes
0
answers
25
views
Feature Selection Before or After kNN Imputation?
From my understanding, kNN imputation is dependent on the variables where any two cases do not have missing values. Thus, would it be ideal to do feature selection before or after imputation?
- 1
0
votes
1
answer
22
views
Multivariate regression on metric spaces
I have a collection of neurons from different parts of the brain. Call this the "test" set.
To these neurons I can associate a morphology measurement, which is an element of a metric space $...
0
votes
0
answers
46
views
Edited nearest neighbor (ENN) under-sampling? When is that useful?
Tackling the problem of unequal group sizes, one would often intuitively think of increasing the number of observations in the minority group. But sometimes the opposite can also be useful, i.e. ...
- 677
2
votes
0
answers
59
views
K-nearest neighbors to estimate mutual information
I would like to use the mutual_info_regression object from scikit-learn to get a rough idea of how well any individual feature ...
- 1,256
0
votes
0
answers
27
views
Theoretical support for sample size exponential increase with dimensionality in nearest neighbour multi-class classification
In multi-class classification using nearest neighbour, I believe that as the dimension of the space increases, we need exponentially more samples to keep the classification error under a certain ...
- 211
0
votes
0
answers
23
views
Can I find the explicit feature map that generates exponent of a kernel?
Let's say I have a kernel $K$, and another kernel of the form :
$$
K' = e^K
$$
now I know how to prove K' is a kernel, I can do it using taylor expansion of $e^x$ around $0$,
but let's say if I want ...
- 123
0
votes
0
answers
35
views
Distance measure in KNN regression
I am currently reading up on KNN-regression. As I understand it the value of a given response variable $y_{i}$ at some point $(x_{1,i}, x_{2,i}, x_{3,i}...)$ is estimated as the average over the k-...
- 199
0
votes
0
answers
34
views
Is it valid to use unsupervised clustering to assign patients into larger groups before comparing them to controls?
I have very high-dimensional data (lipidomics) with two categories: patient/control and genetic mutation. These mutations have similar phenotypes, but they are considered different diseases. Their ...
0
votes
0
answers
20
views
Distance metric for dummy and continous variables
I'm trying to apply the KNN regression model to the data I have at my disposal which contains one dummy variable and two continuous variables (which I have normalized). I was wondering if it is okay ...
- 1
0
votes
0
answers
58
views
Independent features but PCA improves classifiers accuracy significantly. Why?
that's my first question on here :)
I am working with the kNN classifier on datasets from the multivariate normal distribution. I have to groups coming from ...
1
vote
1
answer
132
views
Elbow method for tuning DBSCAN when the minimum number of points per cluster is one
The elbow method calls for setting the number of nearest neighbors (let's call it $k$) to the minimum number of points for a cluster (let's call it $m$), but what do you do when $m\leftarrow1$? Is the ...
- 13
7
votes
1
answer
131
views
Why is the space complexity of KNN $O(dN)$?
If you Google search the space complextiy of KNN, virtually all answers are saying that it costs $O(dN)$, where $d$ is the dimension of our data and $N$ is the number of our data.
But why?
I ...
- 1,630
4
votes
1
answer
89
views
How is the "training error" of KNN plotted?
On https://www.cs.ubc.ca/~murphyk/Teaching/CS340-Fall07/L4_knn.pdf
Page 6
The author (Kevin Murphy) plots the training and test errors associated with kNN classifer.
I am not sure how training error ...
- 1,630
0
votes
1
answer
27
views
How do I select a KNN model?
I am considering methods of selecting an optimal K Nearest Neighbors model for classifying a pixel as representing a refugee's blue tarp or not. See https://www.kaggle.com/datasets/billbasener/pixel-...
- 121
1
vote
0
answers
58
views
Random Forest Variable Importance as a way to weight Nearest Neighbor Variables
I'm employing a nearest neighbor algorithm to find a real NFL game that is the most similar to my projected stats. Not all statistics have the same predictive-importance when projecting the outcome of ...
1
vote
1
answer
109
views
Is K-Nearest Neighbor and Nearest Neighbor algorithm the same?
Does anyone know if there is a difference between K nearest neighbor (KNN) and nearest neighbor algorithm (NN)? And if they are how are they different?
So far all I know is that NN is unsupervised ...
user379637
2
votes
1
answer
67
views
Multiple equidistant neighbours in 1-nearest-neighbour - how to break ties?
I am wondering what should happen when trying to compute leave-one-out cross-validation error when there are multiple 1-nearest-neighbours that is equidistant to the training point.
As an example, ...
- 21
1
vote
0
answers
58
views
Steps for condensed nearest-neighbor algorithm
I'm in the middle of writing my Master's thesis on undersampling techniques in imbalanced datasets, and I wanted to refer on this paper explaining the Condensed Nearest-Neighbor algorithm and what ...
- 11
2
votes
1
answer
44
views
what does the k-NN Algo do with equidistant training points from the test point?
This was my professor's interpretation but he didn’t provide an example:
there could be training points at the same distance from x such that more than k points are closest to x. In this case, we ...
- 21
0
votes
0
answers
23
views
My KNN result not as expected for all numerical data
I have developed the following KNN code and tested it against several datasets with >90% accuracy (the wdbc dataset from UCI for one, granted it was a categorical result), however when using the ...
- 1
7
votes
2
answers
4k
views
How to adjust the classification thresholds in a multiclass classification problem?
I am facing a multiclass classification problem where I have 4 classes and one of them dominates over the others. I use a KNN classification model and the majority of the instances are being ...
- 141
1
vote
0
answers
26
views
How to use Graph Edit Distances and the graph-level features at the same time
I have a distance matrix between paris of graphs computed by Graph Edit Distances. Besides, I have also a group or class label for each graph. Besides, each graph is assigned a target value in real ...
- 11
0
votes
1
answer
173
views
What is the pseudocode for the fastest possible k-nearest-neighbors (KNN) algorithm? [closed]
I have a BERT model that's fine-tuned so that given a sentence in my X column, the model gives a vector that approximates the corresponding sentence in my multidimensional Y array.
I'd like to use the ...
- 343
0
votes
1
answer
163
views
How to interpret NearestNeighbor results obtained using cosine similarity for tf-idf vectors
Why is the top result obtained using cosine similarity extremely close to 0 not the expected 1?
That implies complete orthogonality.
Data:
100k documents/rows with 2000 features(TF_IDF values of ...
- 111
0
votes
0
answers
633
views
Alternative to KNN
I have been working with a custom classification model with some success, and I was wondering if there is a package in Python or R does the same thing. Regular KNN chooses a K and does classification ...
- 149
4
votes
1
answer
61
views
KNN does not take into account the actual magnitude of the distance
Suppose I have 3 two dimensional points $(0, 0), (10000000, 0), (-1, 0)$ and $(0, 0)$ has label 1.0. If we were to use 1-NN to predict the label for $(10000000, 0), (-1, 0)$, then the answer for both ...
- 195
1
vote
1
answer
396
views
What's a Good $R^2$ Score in K Nearest Neighbors?
I'm fitting SciKit-Learn's KNeighborsRegressor on a 5 dimensional space and my model performance is peaking at a score of $\sim 0$.
In their documentation they say ...
- 667
0
votes
1
answer
65
views
How to find or choose the value of k in KNN?
I have a doubt regarding how to choose the value for k in KNN. I saw in many websites to take sqrt of samples. Is the sample here total number of rows or (number of rows x number of columns)? Can ...
- 1
2
votes
1
answer
259
views
Dealing with near duplicates of the same person in K Nearest Neighbor algorithm
For context, I am a beginner and this is my first time attempting to implement a machine-learning algorithm. This is for school. I am attempting to predict whether a 100-meter dash athlete wins a ...
2
votes
2
answers
3k
views
Accuracy score of my KNN model is constant as k increases?
This is my first time using a machine learning algorithm. It is for a school project. My model is attempting to predict from inputs: Weight, Height, and Age whether an Olympic athlete (100-meter dash) ...
1
vote
0
answers
24
views
why does SVM outperforms KNN in 1-gram but in 2,3,4, and 5 KNN outperforms SVM?
my project is authorship attribution which is a multiclass classification, the number of classes is 150, and the number of documents is 2798; it is also an unbalanced issue some classes have more ...
- 11
2
votes
1
answer
485
views
Clustering by using Locality sensitive hashing *after* Random projection
It is well known that Random Projection (RP) is tightly linked to Locality Sensitive Hashing (LSH). My goal is to cluster a large number of points lying in a $d$-dimensional Euclidean space, where $d$ ...
3
votes
0
answers
353
views
kNN Classifier Asymptotic Error Rate versus Bayes Error Rate
Suppose we are in the realm of $M$ class classification, $M \in \mathbb{N}$. I have seen the following result stated many times, but only proven for the case $k = 1$. I would like to prove it for ...
- 183
2
votes
1
answer
230
views
Example of KNN overfitting with k=1
I know that with k=1 a KNN lead to overfitting, this is because it follows the noisy data of the training sample and not generalize well on new input sample. But I am confused on how this happens, I ...
1
vote
0
answers
46
views
What is the best way (in terms of performance and accuracy) to perform value estimation/prediction?
How do I predict the value of an item given its features and attributes? What is the best approach to this regression problem, in terms of performance as well as accuracy? And how do I process this ...
- 111
2
votes
2
answers
679
views
KNN: Should we randomly pick "folds" in RandomizedSearchCV?
TL;DR
In KNN, K is the hyperparameter so we randomly pick it while performing RandomizedSearchCV. Should we also randomly pick the split [Cross-validation + Train] after k-folding? I am considering ...
- 123
2
votes
1
answer
163
views
What does it mean having 1 as best k parameter in K-NN?
I'm working with a large dataset (761 rows and about 57k-60k features) and after doing a feature selection to select the best 10 features I'm using different ML algorithms to classify some cases. In ...
- 23
2
votes
1
answer
340
views
What's the consquences of training a KNN model over the whole data?
I know it is a basic question, but is the only consequence not having a good evaluation of the data?
is the accuracy or precision of the model affected in any way if there is no test set? worse ...
- 553