This is a weird question, I know.
I'm just a noob and trying to learn about different classifier options and how they work. So I'm asking the question:
Given a dataset of n1-dimensions and n2-observations where each observation can be classified into n3-buckets, which algorithm most efficiently (ideally with just one training iteration) produces a model (classification boundary) that would perfectly classify every observation in the dataset (completely overfit)?
In other words, how does one most easily overfit?
(Please don't lecture me on 'not overfitting'. This is just for theoretical educational purposes.)
I have a suspicion that the answer something like this: "Well, if your number of dimensions is greater than your number of observations, use X algorithm to draw the boundary(ies), otherwise use Y algorithm."
I also have a suspicion that the answer will say: "You can draw a smooth boundary, but that more computationally expensive than drawing straight lines between all differing classified observations."
But that's as far as my intuition will guide me. Can you help?
I have a hand-drawn example of what I think I'm talking about in 2D with binary classification.
Basically, just split the difference, right? What algorithm does this efficiently for n-dimensions?