5
$\begingroup$

Here is a demo set of data points that are drawn from a larger sample. I fit a Weibull distribution in R using the {fitdistrplus} package, and get back reasonable results for shape and scale parameters.

# in R: 
library(fitdistrplus)
x <- c(4836.6, 823.6, 3131.7, 1343.4, 709.7, 610.6, 
       3034.2, 1973, 7358.5, 265, 4590.5, 5440.4, 4613.7, 4763.1, 
       115.3, 5385.1, 6398.1, 8444.6, 2397.1, 3259.7, 307.5, 4607.4, 
       6523.7, 600.3, 2813.5, 6119.8, 6438.8, 2799.1, 2849.8, 5309.6, 
       3182.4, 705.5, 5673.3, 2939.9, 2631.8, 5002.1, 1967.3, 2810.4,
       2948, 6904.8)

fitdist(x, "weibull")

Result:

Fitting of the distribution ' weibull ' by maximum likelihood 
Parameters:
         estimate  Std. Error
shape    1.501077   0.2003799
scale 3912.816005 430.4170971

Then I try to do the same thing using scipy.stats. I use the weibull_min function. (I've seen recommendations to use exponweib with constraint a=1 and can confirm results are the same.)

# in python 
import numpy as np
import pandas as pd
from scipy import stats

x = [4836.6, 823.6, 3131.7, 1343.4, 709.7, 610.6, 
     3034.2, 1973, 7358.5, 265, 4590.5, 5440.4, 4613.7, 4763.1, 
     115.3, 5385.1, 6398.1, 8444.6, 2397.1, 3259.7, 307.5, 4607.4, 
     6523.7, 600.3, 2813.5, 6119.8, 6438.8, 2799.1, 2849.8, 5309.6, 
     3182.4, 705.5, 5673.3, 2939.9, 2631.8, 5002.1, 1967.3, 2810.4,
     2948, 6904.8]

stats.weibull_min.fit(x)

Here are the results:

shape, loc, scale = (0.1102610560437356, 115.29999999999998, 3.428664764594809)

This is clearly a terrible fit to the data, as I can see if I just sample from this fitted distribution:

import matplotlib.pyplot as plt
import seaborn as sns

c, loc, scale = stats.weibull_min.fit(x)
x = stats.weibull_min.rvs(c, loc, scale, size=1000)
sns.distplot(x)

Why is the fit so bad here?

I am aware that by constraining the loc parameter, I can recreate the results from {fitdistrplus}, but why should this be necessary? Shouldn't the unconstrained fit be more likely to overfit the data than to dramatically, and ridiculously under-fit it?

# recreate results from R's {fitdistrplus}
stats.weibull_min.fit(x, floc=0)
$\endgroup$
1
$\begingroup$

This was addressed in https://github.com/scipy/scipy/issues/11806. We discussed that the optimizer wasn't finding a good local minimum. A better fit can be found by providing a better initial guess for the location loc=0 (note: this is different from fixing the location with floc=0) or using a different optimizer.

$\endgroup$
0
$\begingroup$

I am aware that by constraining the loc parameter, I can recreate the results from {fitdistrplus}, but why should this be necessary?

This is more a problem with scipy deciding to parameterize their density functions by location and scale rather than my something more sensible (in my opinion, I'm sure someone is going to comment and say location/scale is the superior parameterization. I'm not interested in having those conversations, this is more a matter of personal preference than anything. I digress).

The location parameter, as per here, is essentially just a measure of a shift in the distribution. A scale of 1 means the distribution is moved right a total of 1 units. When you say floc=0 you're telling scipy "hey, please don't estimate the location for me. I already know where it is and it is 0".

You can verify this by looking at your output. When you don't pass floc = 0, then loc is 115.29999. Scipy's best estimate of the location is min(data) which just so happens to be 115.3 (the difference between what scipy estimates and the actual min is likely a numerical artifact).

$\endgroup$
5
  • $\begingroup$ Thanks! It definitely seems to make sense to use min(data) as a starting estimate for the location parameter, but is this estimate not updated during the maximum likelihood optimization process? And does this explain why the other two parameter estimates are so bad? I mean, a shift from 0 to 115 isn't really that big a deal on the scale of this data, and if that were the only discrepancy I really wouldn't mind. But the shape and scale estimates are fairly ridiculous. $\endgroup$
    – Nayef
    Apr 5 '20 at 23:22
  • 1
    $\begingroup$ @Nayef I have no clue what scipy does under the hood. That is something you can easily look at though. $\endgroup$ Apr 5 '20 at 23:23
  • $\begingroup$ Update: opened an issue here, and will wait to hear back. Looks like distribution fitting is being actively worked on right now. github.com/scipy/scipy/issues/11806 $\endgroup$
    – Nayef
    Apr 6 '20 at 0:50
  • $\begingroup$ scipy has a more general distribution. If you want the two parameter distribution, then just fix the third parameter. But I don't see why you need to complain that scipy uses the 3 parameter distribution in the loc-scale family given that it allows the use of the 2-parameter distribution as a special case. $\endgroup$
    – Josef
    Apr 10 '20 at 18:07
  • $\begingroup$ @josef I see that you're a former scipy.stats maintainer, and as I've said in my post I'm not interested in arguing over why I feel like scipy not intuitive when it comes to distirbutions. I'm sure the choices were made for good reasons, but the location/scale mixup has felt like a gotcha since I learned scipy. $\endgroup$ Apr 10 '20 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.