What is this clustering technique called? I have a scatterplot of (x,y). The data point does not show clusters, it's just a mass of points all over the screen.
Each point is given a certain colour based on whether the response $z$ (which depends on $(x,y)$) has some property or not.
Then the scatterplot does end up looking like a cluster: the same-coloured points are bundled together, but there is still significant overlap.
I would like to add straight lines (or something like that?) to my scatterplot that separates these differently coloured points as best as possible, such that for any given $(x,y)$, I can check what colour I should reasonably expect it to belong to.
What can I use or this?
 A: It seems like your problem is actually supervised problem; to be more precise, a two-class classification problem: either the point has some property, or it does not. You are coloring the data points according to this property. If that is the case, you could use any classification technique , for instance logistics regression (which is actually a classification technique, despite its name) and then draw the decision boundary on your $(x,y)$ scatterplot afterwards to visually represent what group (has property/has not property) new data points would belong to.
Clustering is an unsupervised technique; that is to say, you don't know the response. The visual representation of coloring by cluster label or coloring by response is very similar, however the ideas behind it are different.
A: You may want to look at Support Vector Machines. Linear SVMs attempt to find the "best" straight line to separate two populations of points:

Nonlinear SVMs allow "curvy" decision boundaries, but then you need to be careful not to overfit to the noise in your dataset.
SVMs are implemented in most machine learning tools. Wikipedia has some pointers. We also have an svm tag, earlier questions in the tag may be helpful.
(Incidentally, this is not clustering, so looking for this keyword won't help you much.)
