7
$\begingroup$

I have a distribution, which I initially assumed to be a Rayleigh, but it almost certainly isn't. Before I consider convolutions of various distributions, e.g. Rayleigh convolved with Boltzmann, Rayleigh convolved with Gaussian and so on, I was hoping someone with a good eye might be able to identify it:

enter image description here

I have plotted the data with a Rayleigh on top of it to illustrate that it is somewhat similar but clearly this isn't the distribution.

I've been asked to provide a little more information about the data. The data itself are fit residuals from a freqeuncy spectrum. The units of the residuals are in $\rm{dBV_{pk}}$, the definition of which is $\rm{dBV_{pk}} = 10\log_{10}(V^{2}_{pk})$.

I have converted the residuals from $\rm{dBV_{pk}}$ to $V^{2}_{pk}$ by $V^{2}_{pk} = 10^{\rm{dBV_{pk}} / 10}$ and this is what is shown in the histogram.

I initially assumed a Rayleigh as the original spectrum is an FFT, which transforms a signal with real and imaginary parts (both of which are Gaussian distributed) and the absolute value of the FFT is taken, which is exactly how a Rayleigh is produced.

I again will add some further details outlining my motivation.

I have some FFT spectra, which I know the general lineshape of. I want to get an understanding on the noise that is on top of the lineshape, so I look at the fit residuals. The idea being that if I know how the residuals of a spectra are distributed, I can then add it to the lineshape model for simulation purposes. I don't want to add my noise in logorithmic units, i.e. $\rm{dBV_{pk}}$, it is preferable to do this in $V_{pk}^{2}$.

The data I have provided are the residuals from 64 spectra, each having 801 residual points.

I can of course just perform a KDE of this and use this for simulation but it is nice to understand where this profile comes from. For example if one has flat white noise in the frequency domain, and convert this to linear units this is absolutely a Rayleigh distribution -- emerging because the real and imaginary parts of the signal are Gaussian distributed and one always takes the absolute magnitude of a resultant FFT -- Rayleigh!!

I would like to find a similar argument flow for this case.

Data available here: https://filebin.net/17y3un9vs1kh5cq0

Improve this question
$\endgroup$
21
  • 6
    $\begingroup$ Is your goal a parsimonious approximate description of the data or a distribution based on first principles from physical laws? $\endgroup$
    – COOLSerdash
    Sep 4, 2019 at 18:21
  • 2
    $\begingroup$ FWIW, the original residuals are distributed remarkably like a Weibull distribution with shape parameter 5.56, appropriately shifted and scaled. They are a tiny bit heavier in the left tail. $\endgroup$
    – whuber ♦
    Sep 4, 2019 at 18:23
  • 1
    $\begingroup$ @Irish Since you asked for the data, it behooves you to look at them. Although there is serial correlation out to lag 2, there is no "remarkable dependency." Q.P. These details are important and useful. Please edit your post so that it describes your data accurately. $\endgroup$
    – whuber ♦
    Sep 4, 2019 at 18:33
  • 1
    $\begingroup$ You can easily convert any description of the original residuals into an equivalent description of their exponentials. There is much more to be seen in the original values, so I recommend sticking with them for the time being. It would also help to see one of those spectra and to learn how you processed them: I would expect (1) the residuals to be correlated with the detectable peaks and (2) their distribution to depend on the method used to fit the peaks. $\endgroup$
    – whuber ♦
    Sep 4, 2019 at 18:38
  • 1
    $\begingroup$ @whuber You are indeed correct a Weibul distribution fitted to the logarithmic residuals is a perfect fit!!!! $\endgroup$
    – user27119
    Sep 4, 2019 at 19:07

2 Answers 2

Reset to default
14
$\begingroup$

For simulation purposes, a Weibull distribution may work well. Allow me to explain why and to say something about the limitations.

A plot of the original (unexponentiated) residuals immediately suggested a Weibull distribution to me. (One reason this family comes to mind is that it includes Rayleigh distributions, which are Weibull with shape parameter $2.$) The formula will depend on three parameters: a shape parameter plus a scale and location. A standard exploratory technique to test such a distributional hypothesis is the (quantile-quantile) probability plot: one draws a scatterplot of quantiles of the data against the same quantiles of a reference distribution. When this scatterplot is nearly linear, the data differ from the reference distribution only by a change of units--the scaling and recentering.

One exploratory way to find a good shape parameter is to adjust it until the probability plot looks as linear as possible. To avoid too much work, I used various approaches: only data from the first spectrum (optimal shape is $6.3$); equally spaced centiles of all data (optimum is $5.63$); and a variance-weighted version of the latter (optimum is $4.99$). There's little to choose from among those (they all fit the data pretty well). Taking the middle value produces the probability plot at the left:

Figure

The probability plot is exceptionally straight throughout its range, indicating a good fit.

The middle plot shows the corresponding Weibull frequency graph superimposed on the histogram. It tracks the peaks of the bars well, also suggesting a good fit. However, the corresponding chi-squared test indicates a little lack of fit ($\chi^2=334.6,$ $p=2\times 10^{-15}$ with $154$ degrees of freedom based on length-$0.1$ bins from $-8.5$ to $7.0$). To analyze the lack of fit I created a "rootogram" as invented by John Tukey. This displays the square roots of the histogram densities relative to the fitted distribution, thereby greatly magnifying the deviations of the data distribution above and below the fit. This is the right plot in the figure.

To interpret the rootogram, bear in mind that the square root of a count will, on average, be less than one unit from its expected value. You can see that's the case with most of the bars in the rootogram, confirming the previous good fits. In this plot, however, it is apparent that relative to the Weibull fit, the data are a little more numerous at the extremes and the center (the red positive bars) compared to the middle values (the blue negative bars), and this is a systematic, nearly symmetric pattern.

In this sense the Weibull description is not entirely adequate: we should not conclude there is some underlying physical law to explain a Weibull distribution of residuals. The Weibull shape is merely a mathematical convenience that succinctly describes these data very well. (There are other issues, such as the possibility of serial correlation of the residuals within each spectrum. There is some correlation, but it extends only for a couple of lags and therefore is unlikely to suggest any meaningful modification of the foregoing description.)

Ultimately, then, whether you use a Weibull distribution to simulate residuals (which you can exponentiate if you wish) depends on whether these small but systematic departures are important to capture in the simulation.

For the record, the Weibull distribution shown here has shape parameter $5.63,$ scale parameter $11.85,$ and is shifted by $-10.95.$ Because Weibull distributions are just power transformations of Exponential (that is, Gamma$(1)$) distributions, and Exponential random variates are easily obtained as the negative logarithms of the Uniform$(0,1)$ variates supplied by standard pseudorandom number generators in computing systems, it is easy and computationally cheap to generate Weibull variates. Specifically, letting $U$ have this Uniform distribution, simulate the (raw) residuals as

$$X = (-\log(U))^{1/5.63} * 11.85 - 10.95.$$

To illustrate this process, and to serve as a reference for interpreting the preceding data plots, I created a random sample in this manner of the same size as the original dataset ($801\times 64$ values) and drew its histogram, the same Weibull frequency curve, and the corresponding rootogram.

Figure 2

The typical bar is between 0 and 1 in height--but this time, the bar heights appear to vary randomly and independently, rather than following the systematic pattern in the data rootogram.

Improve this answer
$\endgroup$
5
  • 4
    $\begingroup$ Thanks for this extremely thorough answer, I especially like how you give reasons that we cannot infer anything specific about physical interpretation. This is slightly disappointing but you answered the original question quite perfectly! $\endgroup$
    – user27119
    Sep 4, 2019 at 20:40
  • $\begingroup$ (+1) What's the rationale for the $\pm 1$ rule concerning the residuals in a rootogram? Tukey writes in his paper about rootograms: "Most of us know that square roots of counts are likely to be almost uniformly perturbed." Why is that? Has it something to do with variance stabilizing transformations for counts? $\endgroup$
    – COOLSerdash
    Sep 4, 2019 at 20:46
  • 3
    $\begingroup$ @COOL Poisson distributions. Recalling that a Poisson$(\lambda)$ variate has mean and variance equal to $\lambda,$ carrying out the variance-stabilizing analysis I sketched at stats.stackexchange.com/a/251661/919 indicates you should take the root of the variable. The variance ranges from about $0.402173$ (at $\lambda=1$) down to $1/4$ for large $\lambda.$ The 68-95-99.7 rule suggests most values will lie within $2\times\sqrt{1/4}=1$ of the mean. You can see this with quick simulations in R: for(lambda in c(1,3,10)) hist(sqrt(rpois(1e3, lambda))). $\endgroup$
    – whuber ♦
    Sep 4, 2019 at 21:02
  • $\begingroup$ @whuber Great, thanks for the explanation! $\endgroup$
    – COOLSerdash
    Sep 4, 2019 at 21:10
  • 1
    $\begingroup$ Just as a small update for @whuber, the Weibull distribution served perfectly for my simulation purposes. I compared residual distributions from simulation and real data -- they were identical within error bars. Really thanks again for such a great answer. $\endgroup$
    – user27119
    Sep 5, 2019 at 18:09
2
$\begingroup$

Weibull Shape = 2, Scale = 1, Location = 0.2

I just did a hand-fit, but Weibull looks better than your Rayleigh.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Weibull is a good guess--but fit it to the logarithms of these values, not the values themselves. $\endgroup$
    – whuber ♦
    Sep 4, 2019 at 18:24
  • $\begingroup$ @whuber, are you suggesting Weibull should only fit logarithmic data? Also, he added that he's using dB, which is inherently a logarithmic scale. $\endgroup$
    – Ron Jensen
    Sep 4, 2019 at 18:30
  • 2
    $\begingroup$ The Weibull is a poor fit to the exponentiated residuals--you can see that in the systematic and relatively large departures between your black curve and the orange bars--but a different Weibull distribution is a remarkably good fit to the original residuals. $\endgroup$
    – whuber ♦
    Sep 4, 2019 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.