1
$\begingroup$

I was running some test to see the benefits of using robust regression over ols. For that purposes, I created synthetic data in the following manner, A is the random normal matrix(100k*500), x0 random normal vector(generated using np.random.normal). Then I generate the response variable b as

b=Ax0
noise=np.ramdom.normal(0,1,A.shape[0]);
scaled_noise=0.25*norm(b)/norm(noise)*noise

To add the outliers I choose random 1% entries(i's) of scaled_noise and make them scaled_noise[i]=1000*scaled_noise[i] and call this vector outlier_noise and set bn=b+outlier_noise.

With the data above I run ols and robust regression(using weighted least square algo with huber loss ) with A,bn and compare the two by the relative error $\frac{||b_{pred}-b||}{||b||}$, where $b_{pred}=A\hat{x}$ and $\hat{x}$ is the optimal solution retured by the algorithms. I observe that my relative error in ols is $1.00014$ and and robust regression its $ 0.99996$, which seems pretty close. So I was wondering if there is a better way of generating synthetic data so that we can make the case for need for robust regression strong? I tried changing my noise to laplace in the above code(noise=np.random.laplace(0,1,A.shape[0])) but not much difference. Any hints and suggestions are greatly appreciated. Thanks a lot.

$\endgroup$

1 Answer 1

1
$\begingroup$

A more dramatic illustration can be made by adding a cluster of distinct outliers.

Adapting from the scikit-learn documentation, try this:

from sklearn import datasets
import numpy as np

# Generate data with outliers: 1000 samples total, of which 50 are outliers
n_outliers = 50
data_with_outliers_X, data_with_outliers_y, data_with_outliers_coef = datasets.make_regression(n_samples=1000, n_features=1, n_informative=1, noise=10, coef=True, random_state=0)
np.random.seed(0)
data_with_outliers_X[:n_outliers] = 3 + 0.5 * np.random.normal(size=(n_outliers, 1))
data_with_outliers_y[:n_outliers] = -3 + 10 * np.random.normal(size=n_outliers)

Then you can regress the response data_with_outliers_y on covariates data_with_outliers_X.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.