0
$\begingroup$

Summary

I have a dataset of 2D points that exhibit a distinct pattern. My goal is to create a boundary that encompasses the main cluster of points while excluding outliers. In other words, if a point isn't within this boundary, then the point is considered to be unrelated to the observed points. What is the best method, or model to derive this boundary?

Example approach

One approach I've considered is defining a grid of negative clusters or a background mask, followed by applying a clustering algorithm. However, I'm interested in exploring alternative methods that might provide more accurate results.

Main question

What are the most suitable methods or models for automatically deriving a boundary that encompasses the main cluster of 2D data while ignoring outliers?

Notes

In the figure below I used some generated data and drew the boundary by hand. I'd prefer the model to be able to derive a similar boundary. enter image description here

$\endgroup$

1 Answer 1

0
$\begingroup$

If anyone's interested, here's my approach to the problem and how I solved it. It more or less includes 5 steps. If you think a more straightforward approach can be used, please let me know.

# Inputs
frontier_data = pd.read_csv(frontier_url)
feature_meshgrid = np.meshgrid(*[np.arange(frontier_data[f].min(), frontier_data[f].max(), 0.01) for f in frontier_data])
mesh_table = pd.DataFrame(dict(zip(frontier_data.columns, [f.ravel() for f in feature_meshgrid])))

# 1) Fit an outlier detection model to the data
lof = LocalOutlierFactor(n_neighbors=20, algorithm='auto', leaf_size=30,
                         contamination='auto', metric='minkowski', p=2,
                         novelty=True).fit(frontier_data.to_numpy())
mesh_pred = lof.predict(mesh_table).reshape(feature_meshgrid[0].shape)
cluster = pd.Series(lof.predict(frontier_data), name='cluster')

# 2) Use a smoothing function to smooth out the decision boundary
mesh_threshold_smoothed_pred = threshold_local(mesh_pred, block_size=11)

# 3) Derive the boundary points
bounds = pd.Series((1-np.abs(mesh_threshold_smoothed_pred)).ravel(), name='bounds')
mesh_boundary = mesh_table[bounds >= 0.6].reset_index(drop=True).copy()

# 4) Fit a DBSCAN (clustering) model to separate the points for each line
dbscan = DBSCAN(eps=0.1, min_samples=5).fit(mesh_boundary)
mesh_boundary['clusters'] = dbscan.labels_

# 5) Fit a polynomial to each set of points
equations = mesh_boundary.groupby('clusters').apply(lambda x: polynomial_equation(poly_fit(x['y'], x['x'])[1].coef_[::-1]))
equations = equations.rename('equations')
boundary_pred = mesh_boundary.groupby('clusters').apply(lambda x: pd.Series(poly_fit(x['y'], x['x'])[0],
                                                                            name='pred_x', index=x['y']))
boundary_pred = boundary_pred.drop_duplicates()
boundary_pred = pd.merge(mesh_boundary, boundary_pred, on=['clusters', 'y'], how='left')
boundary_pred = pd.merge(boundary_pred, equations, how='left', on='clusters')

You can see the end result in the figure below.enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.