I want to propose 3 very related questions:

  1. I know that in the case of a exponential probability distribution, the maximum likelihood estimate of its parameter $\lambda$ is: $\lambda=(\frac{1}{N}\sum_{i=1}^{N} x_i)^{−1}$. I wonder what is the best way to estimate this parameter in the case that we have a series of observations that are not independent. For example, a vector $V=[v_0,v_1,…,v_N]$ where vi are observations of a variable varying in time (say a voltage in a device). And we want to find a probability distribution for the voltage.

  2. More importantly. I wonder if one can assume independence in the observations under some assumptions. For example, if we take many observations from the voltage and reshuffle them, can we consider that they are independent?

  3. An additional question that bothers me is the dependence between two different random variables. If for example in a real life scenario, where the dependence between two variables $X$ and $Y$ is not clear, and we need to compute the probability of $X>Y$ or some other relationship between the variables: Is there any way to assume independence? (proceeding with the analogy of the voltage, imagine two voltage measurements in a device where it is too complicated to assess how they are related -- would it be all right to assume they are independent?.

Thanks in advance for your help!


Let's take your example of measuring voltage $V$ in a device, say a simple RC circuit, at certain points in time $T$.

  1. It seems what you are actually interested in is estimating the exponential decay of voltage over time, i.e. as a function of time.

    $$V \approx \alpha\exp(-\beta T)$$

    To estimate the relationship with time, you can use regression. In this case, you can take logs and estimate with linear regression

    $$\log(V) \approx a + b T$$

    and recover the parameters through back transformation. Here you are explicit about the dependency of observations on time.

    There is a important difference between estimating an exponential function versus fitting an exponential distribution. Note that we don't want to set internal constraints on $\alpha$ and $\beta$ which would have to be the case to make this a distribution function ($\alpha$ is our initial voltage, $\beta$ is the reciprocal of resistance times capacitance). Of course we could turn this into a probability distribution function by asking, e.g. what's the probability of seeing a voltage $V$ less than or equal to a particular value $v$ across some time interval, but then you've thrown out an important part of the structure -- the relationship to time.

  1. If you simply shuffle the observations once, which is equivalent to sampling without replacement, you will not get independence. Consider that if the first half after shuffling have more values greater than the median, then the second half after shuffling must have more values less than the median. Of course you could do this across (a sample of) all permutations, then you have a permutation test. To get independence, you may sample multiple times with replacement. This is the bootstrap. Again though, you've tossed out the structural relationship to time.

  2. When you are building a model for real world phenomena, you always make assumptions. Sometimes we even make assumptions we know aren't completely true because they make the modeling more convenient/feasible/easier to conceptualize. (Take the Black-Scholes model, which is widely used in the financial industry for options. The assumptions it makes about the market are idealized, and no doubt false, however, the model is still convenient and allows a ususally "good-enough" system of comparison between different financial products.). Even in a model of a simple RC circuit, you might routinely ignore parasitic elements, e.g. the fact that the wiring itself acts like minor capacitance.

    The important thing to do is to validate your model back to the real world

    • Does it make sense? What are its theoretical limitations? What's its operating range? What happens if you are wrong? Certainly assuming independence of voltage closely related parts of the same device is going to be theoretically wrong. However, treating some components as modular is routine. After all, one designs a modular component assuming they receive the correct and fairly steady voltage within some tolerance and to play nicely by not causing undue interference with other components.

    • Monitoring. Of course, before you go plugging in components into expensive equipment, you had better make sure that everything is actually operating within stated tolerances. Is everything performing as intended under your model? How about while you are running a microwave oven and a cell phone nearby?

    • Outcomes analysis. And sometimes you are in for surprises. Perhaps those simplifying assumptions you made come back to haunt you when the control dial is set to 3 on the fizz component and 5 on the buzz component and some minor noise starts oscillating in a positive feedback loop that should be impossible under your model (and here I must admit that I'm not an electrical engineer and just making up these hypothetical examples). Maybe now that simplification of independence you made is critically wrong and you have to revise your model. Good thing you thoroughly tested your assumptions in a lab first before sending that satellite all the way to Mars.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.