1
$\begingroup$

I have 6 sets of Volume(v) & Duration(d) data. I have fitted a quite few distributions to the data such as Weibull, Gamma, Log-Normal, Exponential, GEV, Pareto, Log Logistic, Poisson, and GP. This is one of the data set:

                    d            v
  [1,]              4         48.0
  [2,]             16         73.6
  [3,]              4         52.4
  [4,]              2         62.0
  [5,]             10         48.5
  [6,]             28         99.3
  [7,]              6         49.5
  [8,]             15         61.0
  [9,]              8         56.5
 [10,]             11         52.5
 [11,]             11         55.5
 [12,]              8         89.4
 [13,]             18         54.5
 [14,]              5         56.5
 [15,]              3         67.6
 [16,]              6         51.1
 [17,]              5        112.0
 [18,]             10         51.0
 [19,]             10         50.6
 [20,]             10         52.0
 [21,]              2         77.5
 [22,]              2         53.0
 [23,]              3         56.0
 [24,]              9         51.6
 [25,]              2         50.0
 [26,]              7        103.9
 [27,]              4         50.1
 [28,]              4         51.5
 [29,]              5         55.1
 [30,]             17         64.4
 [31,]             11         54.9
 [32,]              7         89.5
 [33,]              9         50.0
 [34,]             10         50.9
 [35,]              3         56.5
 [36,]              6         54.0
 [37,]              5         49.0
 [38,]              8         50.0
 [39,]              2         51.0
 [40,]              9         66.0
 [41,]              5         57.9
 [42,]              9         57.5
 [43,]             15         48.0
 [44,]              8         64.0
 [45,]              4         52.0
 [46,]              4         54.5
 [47,]              4         70.5
 [48,]              4         51.4
 [49,]              4         86.0
 [50,]              5         70.5
 [51,]              2         61.5
 [52,]             11         76.9
 [53,]             12         69.6
 [54,]              6         47.9
 [55,]              4         64.5
 [56,]              4         62.5
 [57,]              8         72.9
 [58,]              4         53.5
 [59,]              9         81.4
 [60,]             23         53.5
 [61,]              8         77.0
 [62,]              8         71.5
 [63,]              5         87.5
 [64,]             13         67.5
 [65,]              9         66.0
 [66,]              8        139.0
 [67,]              5         54.0
 [68,]             15         61.5
 [69,]              9         59.5
 [70,]              7         77.7
 [71,]             13         50.5
 [72,]             22         48.4
 [73,]              6         68.9
 [74,]              4         53.5
 [75,]              2         49.5
 [76,]              5         49.6
 [77,]              6         51.1
 [78,]             15         67.0
 [79,]              6         58.0
 [80,]              7         51.0
 [81,]             10         64.0
 [82,]              8         58.8
 [83,]             16        102.9
 [84,]              3         61.0
 [85,]             35         54.6
 [86,]             39        107.1
 [87,]              3         49.0
 [88,]              8         53.0
 [89,]             20         52.1
 [90,]             22         65.5
 [91,]             18         50.9
 [92,]             13         51.7
 [93,]             17         77.4
 [94,]             11         75.9
 [95,]              3         63.5
 [96,]             38        120.3
 [97,]              4         69.0
 [98,]              3         68.5
 [99,]             47         63.8
[100,]             72         91.2
[101,]             72         84.0
[102,]              9         57.5
[103,]              5         68.5
[104,]             48         88.8
[105,]              8         54.5
[106,]              3         74.5
[107,]             11         62.2
[108,]              3         65.5
[109,]             55         50.8
[110,]             48         96.0
[111,]             96         62.4
[112,]             54        111.4
[113,]             18         52.0
[114,]             48         79.2
[115,]             48         79.2
[116,]             72        144.0
[117,]              6         54.0
[118,]              5         78.0
[119,]              5         77.0
[120,]             16         51.3
[121,]              3         65.0
[122,]              8         64.5
[123,]              7         79.6
[124,]              4         48.9
[125,]              8         76.6
[126,]              6         50.5
[127,]              4         52.6
[128,]              3         81.1
[129,]              6         65.5
[130,]              7         61.0
[131,]              6         54.9
[132,]              2         57.5
[133,]              9         60.0
[134,]             10         54.0
[135,]              2         50.0
[136,]              5         57.5
[137,]              9         65.0
[138,]             10         50.6
[139,]              5         63.5
[140,]              7         62.6
[141,]              5        100.0
[142,]              2         49.5
[143,]              6         72.0
[144,]              5         81.5
[145,]              6         48.3
[146,]              4         49.0
[147,]             11         69.0
[148,]              7         49.0
[149,]             19         49.1
[150,]             11         75.5
[151,]              2         63.0
[152,]              5         74.5
[153,]              3         58.6
[154,]              5         49.4
[155,]             11         52.0
[156,]              2         50.0
[157,]              3        101.0
[158,]              8         72.5
[159,]              7         48.1
[160,]              2         51.0
[161,]             11         60.5
[162,]             11         50.1
[163,]              2         62.0
[164,]             10         51.6
[165,]              9         49.6
[166,]              3         56.1
[167,]             16         80.1
[168,]              6         81.4
[169,]              2         48.0
[170,]              4         52.5
[171,]              4         49.9
[172,]             19         63.1
[173,]             40         81.9
[174,]             12        105.5
[175,]              5         85.0
[176,]              6         56.4
[177,]              6         49.6
[178,]              5         64.1
[179,]             13         48.6
[180,]              8         54.5
[181,]              7         75.0
[182,]              7         64.5
[183,]              3         64.9
[184,]              3         54.6
[185,]              5         86.5
[186,]              2         51.0
[187,]              5         52.4
[188,]              3         55.0
[189,]              9         50.5
[190,]              9         96.0
[191,]              7         50.5
[192,]              2         49.5
[193,]              3         55.9
[194,]             13         65.0
[195,]              5         60.9
[196,]              6         49.0
[197,]             10         49.6
[198,]              2         60.5
[199,]              8         55.4
[200,]              4        107.5
[201,]              3         60.1
[202,]              8         64.5
[203,]              5         51.6
[204,]              3         54.0
[205,]              6         76.0
[206,]              3         64.5
[207,]              3         63.0
[208,]              6         73.0
[209,]             12         90.0
[210,]              5         62.0
[211,]              3         70.5
[212,]              3         95.0
[213,]             11         77.5
[214,]              5         61.1
[215,]              2         60.0
[216,]              2         48.0
[217,]              7         94.5
[218,]              7         68.0
[219,]              8         79.5
[220,]              4         60.4
[221,]              8         75.0
[222,]              5         55.0
[223,]             18         55.0
[224,]              2         67.0
[225,]              8        158.0
[226,]              7         91.5
[227,]              9         61.5
[228,]              4         73.0
[229,]              7         79.0
[230,]              2         67.5
[231,]              3         58.0
[232,]              6        102.5
[233,]              8         87.0
[234,]              8         74.5
[235,]              4         55.5
[236,]             18        112.5
[237,]             12         75.5
[238,]              3         57.5
[239,]              4         48.5
[240,]              5         55.0
[241,]             14         61.0
[242,]              8         85.4
[243,]              7         79.5
[244,]              5         59.5
[245,]              4         48.0
[246,]              3         72.0
[247,]              7         61.0
[248,]             13         50.0
[249,]              4         55.5
[250,]              2         48.0
[251,]              3         88.0
[252,]              9         55.5
[253,]              4        108.0
[254,]              7         52.6
[255,]              1         99.5
[256,]              2         60.0
[257,]             10        100.0
[258,]              2         53.5
[259,]              4         83.5
[260,]             12         83.0
[261,]              9         56.8
[262,]             15         68.1
[263,]              7        126.6
[264,]              6         54.5
[265,]              7         59.4
[266,]              9         59.1
[267,]              6         50.0
[268,]              6         52.5
[269,]              7         67.0
[270,]              4        129.0
[271,]             20         81.5
[272,]             19         57.5
[273,]              9         54.5
[274,]              6         55.5
[275,]              5         65.0
[276,]              4         53.0
[277,]              9         77.1
[278,]              7         81.5
[279,]              6         72.6
[280,]              6         61.4
[281,]              3         58.0
[282,]              3         59.5
[283,]              4         56.5
[284,]              4        126.1
[285,]              3         77.5
[286,]              3         84.5
[287,]             11         56.0
[288,]              2         62.0
[289,]              3         74.5
[290,]              5         82.0
[291,]              5         52.5
[292,]              8         52.5
[293,]             11         78.0
[294,]              2         57.5
[295,]             14         55.0
[296,]             14         59.5
[297,]              3         51.0
[298,]              2         52.5
[299,]              6         60.0
[300,]              6         88.5
[301,]              4         52.0
[302,]              3         56.0
[303,]              4         59.0
[304,]              3         87.0
[305,]              3         65.5
[306,]              6        108.5
[307,]              6         57.0
[308,]             17         52.0
[309,]              9         62.0
[310,]              7         56.0
[311,]             12         64.0
[312,]              7         54.0
[313,]             31         92.5
[314,]              8         73.0
[315,]              7         55.0
[316,]             26         73.5
[317,]             63         76.5
[318,]            315        117.5
[319,]             12         73.5
[320,]              5         54.0
[321,]              2         58.5
[322,]              7         83.0
[323,]              3         53.0
[324,]              3         48.0
[325,]             10         78.5
[326,]              3         72.5
[327,]              2         52.0
[328,]              4         57.0
[329,]              6         55.5
[330,]              7         57.0
[331,]              6         53.0
[332,]             13         52.5
[333,]              9         59.5
[334,]              8         79.0
[335,]              4         67.0
[336,]              8         73.0
[337,]              7         62.5
[338,]              4         80.5
[339,]              3         54.0
[340,]              6         58.0
[341,]              6         98.0
[342,]              2         49.0
[343,]              4         52.5
[344,]              2         55.0
[345,]             17         58.0
[346,]             13         80.0
[347,]             11         60.0
[348,]              3         83.5
[349,]              8         75.5
[350,]              4         67.0

I'm using fevd function in extRemes package to fit GEV and GP and
fitdist function in fitdistrplus package to fit other distribution. The coding basically like this

fw1 <- fitdist(d, "weibull")
fw2 <- fitdist(v, "weibull")

fit1 <- fevd(d, type="GEV")
fit5 <- fevd(v, type="GEV")

but none of the distributions can fit my data. Anyone can help me with the coding/ R? what distributions suitable for my data? what other distributions that I can try? I also try this code. This is the first time I've done this and I'm not familiar with the distributions. Thank you for your help!

EDIT:

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Why do you need to fit a distribution? What are you doing with it? $\endgroup$
    – Glen_b
    Commented Jul 1, 2020 at 3:20
  • $\begingroup$ because later I want to estimate Copula under a Parametric assumption for my study. Really need help @Glen_b $\endgroup$
    – Mia
    Commented Jul 1, 2020 at 6:14
  • $\begingroup$ Can you provide histograms of the data? Somebody might be able to suggest a suitable family with help from a histogram. $\endgroup$
    – jcken
    Commented Jul 1, 2020 at 7:30
  • $\begingroup$ I just realize that I give different data. I just edit it and add histograms @jcken $\endgroup$
    – Mia
    Commented Jul 1, 2020 at 9:14
  • $\begingroup$ From the histograms, I would check log-normal and Gamma for "volume" and negative binomial for "duration". $\endgroup$
    – user289381
    Commented Jul 1, 2020 at 10:07

1 Answer 1

Reset to default
4
$\begingroup$

EDIT: an important warning about choosing the likelihood from the density plot of the data. In these cases, there is the risk of overfitting the data. Here a good answer about how to reduce this risk https://stats.stackexchange.com/a/20738/289381 (Gelman's advise is to use a leave-one-out cross-validation).

From the density plot of the "volume" variable, its distribution is a mixture of two log-normal (notice the right shoulder "peak"):

enter image description here

You can find some details about a mixture of two Gaussians here https://stats.stackexchange.com/a/474775/289381

This is how I fitted its parameters using a Bayesian approach using the R package brms

library(brms)

dat <- read.csv("data.csv")
colnames(dat) <- c("d", "v")

mix <- mixture("lognormal", "lognormal")
mdl_1 <- brm(v ~ 1, data=dat, family=mix)  # Using the default priors

This is the plot of the posteriors (the names of the parameters are quite clear, $\theta$ represents the probability to belong to the first or the second component of the mixture):

plot(mdl_1, N=6)

enter image description here

This is the posterior predictive check that shows that the fitted distribution capture the data reasonably well:

pp_check(mdl_1, nsamples = 50)

enter image description here

duration

The data looks highly skewed to the right. I am not sure if the outliers are expected or are the result of something that went wrong.

The log-normal distribution seems to fit well the data as you can see here from the posterior predictive distribution

enter image description here

These are the posterior for the mean and st.dev. of the log-normal distribution:

enter image description here

This is the code (using brms):

mdl_ln <- brm(d ~ 1, data=dat, family="lognormal")
plot(mdl_ln)
pp_check(mdl_ln, nsamples = 50)
Improve this answer
$\endgroup$
9
  • $\begingroup$ Hi, thank you for your answer! I've tried to fit a negative binomial using the fitdist function but I get very low p-value for the chi-square test. I never try the Bayesian approach and mixture models. How do I generate data from the model? $\endgroup$
    – Mia
    Commented Jul 1, 2020 at 16:23
  • $\begingroup$ and how to read the plot? is theta1 is for the first peak? the value is between 0.3 and 0.6? $\endgroup$
    – Mia
    Commented Jul 1, 2020 at 16:37
  • $\begingroup$ You have a couple of strong outliers in "d". Depending on what you are trying to achieve with the data, there are multiple approaches. It depends on the information you have about the process that has generated the data. Are these extreme values expected? About $\theta$, the distribution for "volume" is a mixture of two log-normal distributions with mean $\mu_1$, $\mu_2$, and variance $\sigma_1^2$, $\sigma_2^2$. $\theta$ is the mixing weight, such that $f(y) = \theta_1 * f_1(y|\mu_1, \sigma_1^2) + \theta_2 * f_2(y|\mu_2, \sigma_2^2)$. $\endgroup$
    – user289381
    Commented Jul 1, 2020 at 17:11
  • $\begingroup$ Although I am not a fan of data transformations, you can try to fit $log(d)$. A normal distribution would work, even though you still have another peak to the right (check with plot(density(log(dat$d))). Another option is fitting a log-normal distribution (without transforming the data). This seems to work fine. Remember that log-normal and normal of log are different things. $\endgroup$
    – user289381
    Commented Jul 1, 2020 at 19:23
  • $\begingroup$ thanks for the explanation! ya, actually this is extreme values at the 95th percentile of volume threshold. I was thinking about data transformation but didn't sure about that. How do I know if it is okay to transform data? I've tried to fit a log-normal but it doesn't fit and it fit just fine to a Pareto, I'll check again. $\endgroup$
    – Mia
    Commented Jul 2, 2020 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.