2
$\begingroup$

I'm trying Sklearn's RANSAC algorithm implementation to produce a simple linear regression fit with built-in outlier detection/rejection (code below). This is what my raw data looks like:

enter image description here

Even using max_trials=10000000 (the Maximum number of iterations for random sample selection) the estimated coefficients (slope, intercept) and the R^2 & RMSE statistics vary quite a bit with each new run of the same exact raw data.

I would've assumed the results converged if one gave it enough iterations, but apparently this is not true. Are 10 million iterations not enough? Is my data too noisy? Am I missing some parameter I should fix in RANSACRegressor or is this just the nature of the RANSAC algorithm?

import numpy as np
from sklearn import linear_model
from sklearn.metrics import r2_score, mean_squared_error

# Data
x = [-0.099000000000000005, 0.094, 1.3129999999999999, 0.096000000000000002, 0.088999999999999996, -0.002, 0.11, 2.1440000000000001, -0.0040000000000000001, 0.245, 0.66600000000000004, 1.0029999999999999, 0.71899999999999997, 1.9359999999999999, 0.098000000000000004, 0.39400000000000002, 0.35699999999999998, 1.284, -0.027, 0.123, 1.837, 0.85299999999999998, 1.0669999999999999, 0.126, 0.078, -0.001, 0.318, 0.39900000000000002, 1.4259999999999999, 1.4970000000000001, 1.0860000000000001, -0.10000000000000001, 1.296, 0.44700000000000001, 0.23599999999999999, 0.26300000000000001, 1.5589999999999999, 1.1200000000000001, 1.2390000000000001, 1.0920000000000001, 1.1779999999999999, 1.915, -0.10299999999999999, 0.156, 0.53200000000000003, 1.871, -1.2170000000000001, 0.38]
y = [-4.5064700000000002, -3.8593700000000002, -3.5898699999999977, -3.7799699999999987, -3.7595699999999983, -3.8206699999999998, -3.7413699999999981, -3.0731699999999975, -3.7439699999999974, -3.6922700000000006, -3.7251700000000003, -3.6609700000000007, -3.8754699999999982, -3.1616699999999973, -3.8635699999999993, -3.8146700000000013, -3.6913699999999992, -3.5147699999999986, -3.852369999999997, -3.8125699999999991, -0.58567000000000036, -3.81447, -3.6871699999999983, -3.6793700000000005, -3.8035699999999988, -3.8506699999999991, -3.6703700000000019, -3.7025699999999979, -2.6628699999999981, -3.5626700000000007, -3.0997699999999959, 1.2492300000000007, -3.7071510000000014, -3.9486509999999981, -3.7250509999999988, -3.9179509999999986, -3.5861509999999974, -3.8082509999999985, -3.7271509999999992, -3.780851000000002, -3.6115509999999986, -3.4124880000000033, -3.8527880000000003, -3.7375880000000006, -3.8264880000000012, -3.4370879999999993, -4.0668880000000005, -3.8614880000000014]
# Proper shape
x, y = np.array(x).reshape(-1, 1), np.array(y).reshape(-1, 1)

# Fit linear model with RANSAC algorithm
ransac = linear_model.RANSACRegressor(min_samples=2, max_trials=10000000)

# Run regression 10 times with the same raw data
for _ in range(10):
    ransac.fit(x, y)

    # Estimated coefficients
    m, c = float(ransac.estimator_.coef_), float(ransac.estimator_.intercept_)
    print("\nm={:.3f}, c={:.3f}".format(m, c))

    # R^2 & RMSE statistics
    inlier_mask = ransac.inlier_mask_
    x_accpt, y_accpt = x[inlier_mask], y[inlier_mask]
    y_predict = m * x_accpt + c
    R2 = r2_score(y_accpt, y_predict)
    RMSE = np.sqrt(mean_squared_error(y_accpt, y_predict))
    print("R^2: {:.3f}, RMSE: {:.3f}".format(R2, RMSE))
Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

Here are some observations:

Increasing min_sample makes the model much more stable. I believe the model will find it difficult to decide on what is an outlier when only sampling from from a minimum of two different samples.

I dropped this into a for-loop to iterate over some min_samples and this real steadied the model. Here it is:

import numpy as np
from sklearn import linear_model
from sklearn.metrics import r2_score, mean_squared_error

# Data
x = [-0.099000000000000005, 0.094, 1.3129999999999999, 0.096000000000000002, 0.088999999999999996, -0.002, 0.11, 2.1440000000000001, -0.0040000000000000001, 0.245, 0.66600000000000004, 1.0029999999999999, 0.71899999999999997, 1.9359999999999999, 0.098000000000000004, 0.39400000000000002, 0.35699999999999998, 1.284, -0.027, 0.123, 1.837, 0.85299999999999998, 1.0669999999999999, 0.126, 0.078, -0.001, 0.318, 0.39900000000000002, 1.4259999999999999, 1.4970000000000001, 1.0860000000000001, -0.10000000000000001, 1.296, 0.44700000000000001, 0.23599999999999999, 0.26300000000000001, 1.5589999999999999, 1.1200000000000001, 1.2390000000000001, 1.0920000000000001, 1.1779999999999999, 1.915, -0.10299999999999999, 0.156, 0.53200000000000003, 1.871, -1.2170000000000001, 0.38] * 10
y = [-4.5064700000000002, -3.8593700000000002, -3.5898699999999977, -3.7799699999999987, -3.7595699999999983, -3.8206699999999998, -3.7413699999999981, -3.0731699999999975, -3.7439699999999974, -3.6922700000000006, -3.7251700000000003, -3.6609700000000007, -3.8754699999999982, -3.1616699999999973, -3.8635699999999993, -3.8146700000000013, -3.6913699999999992, -3.5147699999999986, -3.852369999999997, -3.8125699999999991, -0.58567000000000036, -3.81447, -3.6871699999999983, -3.6793700000000005, -3.8035699999999988, -3.8506699999999991, -3.6703700000000019, -3.7025699999999979, -2.6628699999999981, -3.5626700000000007, -3.0997699999999959, 1.2492300000000007, -3.7071510000000014, -3.9486509999999981, -3.7250509999999988, -3.9179509999999986, -3.5861509999999974, -3.8082509999999985, -3.7271509999999992, -3.780851000000002, -3.6115509999999986, -3.4124880000000033, -3.8527880000000003, -3.7375880000000006, -3.8264880000000012, -3.4370879999999993, -4.0668880000000005, -3.8614880000000014]*10
# Proper shape
x, y = np.array(x).reshape(-1, 1), np.array(y).reshape(-1, 1)

# Fit linear model with RANSAC algorithm

# Run regression 10 times with the same raw data
for n in range(2, 25):
    RMSE = []

    ransac = linear_model.RANSACRegressor(min_samples=n, max_trials=10000000)


    for _ in range(10):
        ransac.fit(x, y)

        # Estimated coefficients
        m, c = float(ransac.estimator_.coef_), float(ransac.estimator_.intercept_)
#         print("\nm={:.3f}, c={:.3f}".format(m, c))

        # R^2 & RMSE statistics
        inlier_mask = ransac.inlier_mask_
        x_accpt, y_accpt = x[inlier_mask], y[inlier_mask]
        y_predict = m * x_accpt + c
        R2 = r2_score(y_accpt, y_predict)
        RMSE.append(np.sqrt(mean_squared_error(y_accpt, y_predict)))
#         print("R^2: {:.3f}, RMSE: {:.3f}".format(R2, RMSE))
    print('min_samples: %d' % n, 'std: %f' % np.std(RMSE))

and the output,

min_samples: 2 std: 0.003821
min_samples: 3 std: 0.003895
min_samples: 4 std: 0.002477
min_samples: 5 std: 0.002746
min_samples: 6 std: 0.003013
min_samples: 7 std: 0.002673
min_samples: 8 std: 0.002763
min_samples: 9 std: 0.001543
min_samples: 10 std: 0.001367
min_samples: 11 std: 0.001390
min_samples: 12 std: 0.001390
min_samples: 13 std: 0.001486
min_samples: 14 std: 0.001390
min_samples: 15 std: 0.001214
min_samples: 16 std: 0.001390
min_samples: 17 std: 0.000000
min_samples: 18 std: 0.000910
min_samples: 19 std: 0.000000

and a graph of the trend, x is min_samples and y the standard deviation of the RMSE.

Model stability vs min_sample

Improve this answer
$\endgroup$
2
  • $\begingroup$ I too realized this was perhaps a better question for CV and I flagged it so that a moderator would move it over there, it never happened though. I run a few tests setting the min_samples parameter to a range of values but it doesn't seem to help much. The values themselves keep varying even when I use up to 90% of the data to produce the fit. Thank you anyway! $\endgroup$
    – Gabriel
    Commented Jan 25, 2018 at 11:14
  • 1
    $\begingroup$ I have flagged it also - we will see. I would like to see a more stats/technical answer to your question! $\endgroup$
    – josh
    Commented Jan 25, 2018 at 11:30
1
$\begingroup$

You are missing the random seed parameter - RANSAC uses random numbers to select the samples to use for the iterations.

So what you are looking for is: 

ransac = linear_model.RANSACRegressor(min_samples=n, max_trials=10000000, random_state= num)

Where num is an integer of your choosing, you can trial as many as you like in a loop and pick the best one as well.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If I understand correctly that would fix the output result, but then I'd have to pick some number to initialize the regressor and I'm back to square one: how do I pick this number? $\endgroup$
    – Gabriel
    Commented Jul 13, 2018 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.