# Manually Deriving the Maximum Likelihood Estimates for Less Common Probability Distributions

I have a question relating to Manually Deriving the Maximum Likelihood Estimates for Less Common Probability Distributions.

Suppose we generate 200 random numbers from a Normal Distribution ~ (1,2):

 [1]  1.397769643 -1.349810123  1.409096115  1.166556052  2.462052443  3.028535770 -1.857377329  0.065942000  0.351794046  2.147062096  2.363010032 -1.189474247
[13]  3.442713676  0.351788119 -0.309373535  1.837829950  3.101517512  1.039759334 -0.301201509  1.535874305 -0.728008267  2.143456207  4.348591039  2.820767778
[25]  3.496333171 -2.338135209  0.746765242  1.746013026  0.944348016  1.970963351  2.354741787  2.387481721  1.205882610  0.402545739  0.902755998 -0.642025338
[37]  0.087375328  1.806437453  3.791417174 -0.589403950 -0.239511885  1.137691201  2.380651204  1.850905237  2.962715996  3.959523099  3.621970972 -1.058406304
[49] -1.507255542 -1.885716470  1.135787149 -2.674447419  3.554089580  2.186803656  2.177498440 -1.036666961  3.498291156 -0.800947746  1.288569947 -2.301217797
[61]  4.714108879  0.515299873  2.078148740  2.275190922  2.422285866 -0.149438329 -0.049109475  1.827983868 -0.896601107  0.440313103  1.334843025 -0.775672031
[73] -0.837142576 -1.704267065 -2.688634934  1.996045412  2.808195823  1.854402259  0.052777805  2.900927434 -3.109949576  2.196366889  2.739805512  2.299826580
[85]  1.057179217  0.814933668  0.420696242  0.850065781  2.249043234  1.600343804  0.479352527  1.666304984  0.473000452  0.999406254  3.067888101  3.128883912
[97] -0.714565878  1.387855402  2.827717839  4.909632706 -0.008479906 -1.445391996  1.150020382  2.197519540  0.542254705  5.978928354  0.446459107  1.177704991
[109]  1.638146128  0.776432261  1.421044468 -2.076820462  1.350773328 -4.431283287  2.329737776  1.996587842  1.628355787  0.023008381 -1.408107937  2.005475065
[121] -0.860002975  0.859310023 -0.926337294  0.104039654 -0.661428856  0.031098359  1.686128381 -0.166502427 -0.548287576 -0.328115104  3.030054374  3.473628834
[133] -2.560485018  2.740471686  1.525781872  1.154106972  3.330453097  0.399179112  1.990016298 -0.940498588  1.683962236 -0.435178631 -0.025076808  0.822784836
[145]  0.010967310  0.827452539 -1.517881301  4.446102114  0.798512538  1.829764053  0.576889237  1.286887132  3.239158791  2.684281081  2.582538018  4.313381936
[157] -2.324854734  0.434557666  0.682587948  3.193361284  0.907390976  2.247828419 -0.919286073  0.135801307 -1.673057204  1.044632060  2.068368158  1.248012441
[169] -0.718772118  1.760703820 -2.243362849  3.569998000  2.246035684 -2.093363452  2.127768101  2.526914676 -0.181808756  1.633249166 -0.062230338  3.412009734
[181]  1.606922725  3.702113831  0.601343928  2.156066225  1.332592964 -0.035667929 -1.249321565  2.676814894  3.825397261  1.502243284 -1.436908125  1.173751825
[193]  2.495839812  2.137135555  0.491835980 -0.324536947  0.128536831 -0.478446746  2.658782246  4.427213921


By plotting the distribution of this data (I called this data "a"), it's reasonable to expect that this data might have come from a Normal Distribution (I did this using the R programming language):

plot(hist(a))
plot(density(a))


If we assume a Normal Distribution, Maximum Likelihood Estimation theory show us how to derive the estimators for the "mean" and "variance" of the Normal Distribution:

As a result, if we use the formulas for these estimators, our estimates will be pretty close to the actual values used to generate these points:

mean = sum(a)/200
[1] 1.057149

standard_deviation = sqrt(sum((a - mean(a))^2) / (200 - 1))
[1] 1.771531


My Question: How can we derive similar formulas for the Maximum Likelihood Estimates for less common distributions, such as the Tweedie Probability Distributions?

In R, random numbers from a Tweedie Distribution can be generated according to the following parameters ("power", "mu", "phi"):

library(tweedie)
y = rtweedie( 200, power = 1.2, mu = 1.5, phi = 1.2)
hist(y)

y1 = rtweedie( 200, power = 2, mu = 1, phi = 1.5)
hist(y1)


If we look at some of these randomly generated numbers from the Tweedie Distribution (power = 2, mu = 1, phi = 1.5):

  [1] 4.835229e-01 3.273596e-01 7.250924e-01 2.420644e+00 4.385017e-01 8.516473e-01 9.575440e-01 3.623502e-01 6.725927e-01 3.359006e-02 2.171428e+00 2.043827e+00
[13] 7.253516e-02 4.601185e-01 1.381129e+00 3.434772e-01 3.124828e-01 2.210105e-01 5.395479e-01 7.862751e-02 2.240753e+00 1.200838e-01 5.442454e-01 6.421638e-01
[25] 3.014555e+00 4.403903e-01 5.410043e-02 6.235082e+00 2.149887e-01 1.036470e+00 9.106634e-01 3.148187e-01 1.558772e-02 5.553067e-01 5.049492e-02 1.119339e+01
[37] 1.034588e+00 2.013105e-01 1.766221e-01 4.711094e-01 5.075948e-01 1.982064e-01 2.532509e-01 7.562298e-01 1.856084e+00 4.363536e-01 1.787625e+00 1.198450e+00
[49] 4.674783e-01 9.785719e-01 4.146315e-01 1.365006e+00 2.022401e-01 2.258816e-02 1.438350e+00 2.975052e-01 1.310374e+00 2.574818e-01 1.530184e+00 5.716808e-01
[61] 1.006795e+00 2.789478e-02 2.110384e+00 6.499831e-01 2.293844e-01 8.956054e-03 5.910401e-01 2.473032e+00 8.723009e-01 4.730672e+00 1.072683e+00 2.961126e-02
[73] 1.540705e-01 4.643479e-02 1.048683e-01 6.655147e-01 4.578233e-01 8.636943e-01 3.723237e-01 1.108970e+00 1.091554e+00 6.452105e-02 2.967971e+00 3.087451e+00
[85] 2.542959e+00 3.112244e-02 7.727888e-01 9.219587e-01 1.239732e-01 4.568944e-01 2.543266e+00 2.355232e+00 1.806799e-01 1.652674e-01 8.144687e-02 1.190040e+00
[97] 2.117846e+00 9.787840e-01 9.352342e-02 3.874447e-01 4.245093e+00 1.898894e+00 9.890710e-01 8.805150e-01 8.246694e-02 3.846567e+00 5.313029e-01 1.493798e+00
[109] 1.582339e+00 2.194297e+00 5.971556e-02 9.142690e+00 5.213884e-04 7.181976e-02 1.593938e+00 8.947677e-01 3.631769e-02 1.022450e-02 2.591273e-01 3.802100e-01
[121] 1.509313e+00 1.412009e+00 1.342824e-01 2.137578e+00 1.022353e-01 1.178305e-01 1.008145e+00 5.294744e-02 6.757453e-02 1.240903e+00 4.604285e-01 3.069427e+00
[133] 6.520134e-01 4.636159e-01 2.093750e+00 1.804378e-01 3.066245e-01 1.611355e+00 8.780143e-02 2.801444e-01 1.074960e+00 2.617296e-01 1.509953e+00 2.612716e-02
[145] 8.929552e-03 1.681465e+00 8.246581e-01 2.288786e+00 2.032069e+00 8.485531e-01 9.085607e-01 5.273483e-01 6.169230e-02 1.814618e-01 1.481157e+00 2.420769e+00
[157] 1.078106e-01 1.582527e+00 1.264192e-01 1.481463e+00 2.316226e-01 7.622064e-01 4.965801e-01 2.422014e+00 1.153327e+00 2.873479e+00 2.011737e-02 6.827516e-01
[169] 6.406967e-01 5.710539e-01 4.386122e-01 6.174785e-01 7.832803e-02 5.669017e-03 8.192740e-01 1.158842e-01 8.627114e-01 3.967166e-01 2.668195e+00 3.451398e+00
[181] 1.994125e-01 4.742352e-01 1.986717e+00 1.184821e+00 4.672017e-01 4.829571e+00 7.589721e-02 7.405997e-02 6.214423e-01 5.717092e+00 3.910497e-01 1.104365e+00
[193] 8.593129e-01 1.996427e+00 4.901073e-01 1.831476e-01 9.505221e-02 3.102266e+00 1.617453e-03 1.186391e+00


Can we also use MLE to derive the formulas for the "power", "mu" and "phi" parameters? Suppose we know the probability distribution formula for the Tweedie Distribution (linked above) - can someone please show me how the derivatives of the log-likelihood function for the different model parameters of the Tweedie Distribution can be written, optimized and calculated? Given the 200 randomly generated numbers from the Tweedie Distribution - if we used the MLE formulas, would they return values of "power", "mu" and "phi" close to 2, 1 and 1.5?

Thanks!

• The question as stated for "less common distributions" is far too broad (and vague--what might the "common" distributions be?) for this site, but interpreting it literally as for Tweedie distributions only makes it sufficiently focused to be answerable.
– whuber
Nov 25 at 16:18
• Find the likelihood function, take the logarithm to convert products into sums, take the derivative, find where the derivative equals zero, and determine which of those points is the highest. It’s the usual from calculus, just with nasty functions that might not have decent derivatives for by-hand calculations.
– Dave
Nov 25 at 16:22
• In general you can't solve MLEs in closed form. Typically you have to use numerical optimization methods to find the argmax of the (log-)likelihood and further numerical calculation to obtain approximate standard errors and covariances if required. In the case of the Tweedie, if you fix the power parameter, $p$, you have a GLM. This is particularly convenient as you can use that to compute the profile likelihood at any particular $p$, and hence, by iterating calls to GLM, obtain a numerical ML estimate of the parameter $p$, and then obtain the ML estimates of the remaining ... ctd Nov 27 at 5:14
• ... parameters from the GLM at that value of $p$. This is similar to the method used by MASS (which comes with R but is one of a number of packages in the distribution of R but which are not loaded by default) to compute the theta parameter in a negative binomial regression (see glm.nb in MASS and the corresponding book by Venables and Ripley). GLMs, being exponential family, have nice sufficient statistics, which makes them relatively simple to fit compared to the general ML case and optimization of a single parameter is simpler. There's an R package for Tweedie models Nov 27 at 5:16
• Thank you everyone for your replies! I realize that the estimates for "power", "mu" and "phi" will require some numerical optimization method (compared to the normal distribution in which the maximum likelihood estimates for "mu" and "sigma" have a closed form solution). I was just curious to find a reference that explains how to set up the equations for "mu", "phi" and "power" so that a computer can solve them. For instance, there is a "summation over infinity" in the likelihood equation: how can we handle this in the mle process? Nov 27 at 5:37