Questions tagged [empirical-cumulative-distr-fn]
Empirical cumulative distribution function: a step function increasing by $1/n$ at each unique $X$-value that occurred in the sample.
128
questions
0
votes
0answers
15 views
Probabilities from ecdf and normalised data - is this acceptable difference? (using R)
I have a dataset of 500 observations, definitely not normally distributed: The minimum is around 50 000 in the data. If I use ecdf to predict e.g. 100 000, I get:
...
0
votes
1answer
63 views
What is a "bootstrap from the Empirical CDF"?
I am given the following set of questions.
In all the questions the basic background is a sample X_1, ... , X_n.
If n=3 and you observe the numbers 1, 2, and 4, what is the sample median.
If you ...
0
votes
1answer
21 views
Determine a product's rating given a known 3-sigma tolerance
Consider a 100Ω resistor with a 10% tolerance. We can assume this is the 3-sigma value since this is typical in manufacturing. Thus, we can expect ≈99.73% of such resistors to range from 90Ω to 110Ω.
...
1
vote
1answer
36 views
0
votes
1answer
87 views
Smoothed CDF to calculate asymptotic normality
If we have the following estimator: $\hat{F_Z}(z)=\frac{1}{N}\sum_{i=1}^N1\{Z_i\leq z\}$. The CDF of $Z$ is defined as $F_Z(z)=Pr(Z\leq z)$. $Z_1, ..., Z_N$ is i.i.d. data.
What would be the steps to ...
0
votes
0answers
44 views
counts under unequally spaced intervals - stackoverflow points table
I am looking for a function similar to ecdf
which help me to asses the quantile for certain number of stackoverflow points across all users.
We have a table with ...
2
votes
0answers
58 views
Plug-in principle with kernel density estimate
The plug-in principle says that to estimate a statistical functional of the form
$$
T(\mu) = \int f(x)\ d\mu(x)
$$
we can replace $\mu$ with the empirical distribution $\mu_n$ depending on data $X_1,\...
0
votes
0answers
9 views
Standard Errors of ECDF with Potential Model Fit Error
I have data and a proposed likelihood. I used MLE to fit the likelihood, but it is a very complicated likelihood and there is some estimation error of the model. The likelihood may be correct, but ...
6
votes
1answer
152 views
What are the simplest examples of nonlinear statistical functionals?
I am reading Wasserman's book "All of Statistics" in which he defines a statistical functional as any function $T(F)$ of the cumulative distribution function $F(x)$ that outputs a real ...
2
votes
1answer
28 views
Picking a specific estimated CDF from a set of CDFs provided by an ECDF
Let $F_X$ be a CDF of an unknown random variable $X$. If we have independent samples $x_1, x_2, \ldots, x_n$ of $X$ then we can estimate $F_X$ non-parametrically using an ECDF $\hat{F}_n$. By Central ...
2
votes
2answers
76 views
Notation for ECDF
I'm reviewing statistical functionals and U-statistics, trying to make notes, and I am tripping on notation. From my understanding, $X$ is used to denote a random variable and $x$ is used to denote an ...
0
votes
0answers
20 views
Trying to interpret the comparison between two ECDF curves
I am trying to understand ECDFs to compare two models "a" and "b". My initial interpretation from the plot is that judging by the difference between the two curves model "b&...
2
votes
1answer
130 views
Density estimation from ECDF - numerical derivatives and scaled domains
Suppose we want to get a density estimate of some data X. One way is to compute the empirical CDF,
...
0
votes
0answers
25 views
Bootstrap bias-corrected percentiles
Following [1] p. 185-186 the $BC_a$ (bootstrap corrected and accelerated) confidence intervals are given by:
$$
(\hat{\theta}^{*(\alpha_1)},\hat{\theta}^{*(\alpha_2)}),
$$
where $\hat{\theta}^{*(\...
2
votes
0answers
21 views
Nonparameteric Empirical Estimator For Stochastic Process
Motivation: If $X$ is a random-variable defined on some probability space $(\Omega,\Sigma,\mathbb{P})$ then Glivenko-Cantelli lemma guarantees that the empirical distribution $\frac1{N}\sum_{n=1}^N \...
0
votes
0answers
8 views
Understanding of scale loss with Gaussian prior
I am reading a paper called PHOSA wherein a scale loss is used. I had the following questions regarding this scale loss in equation 8:
What part of this loss function is actually incorporating a ...
0
votes
0answers
53 views
How to deal with a PDF that takes CDF data as input? (Metalog distributions)
I am trying to utilise the Metalog distribution in a Machine Learning project.
For this project, I need to be able to obtain likelihoods using the PDF of the distribution.
https://en.wikipedia.org/...
0
votes
0answers
19 views
Convergence of the first moment of empirical distribution
Imagine I have a sequence of random variables $\{x_n\}_{n=1}^{\infty}$. These are not i.i.d. random variables, but an arbitrary sequence.
For any $n$, I can define the empirical CDF function $F_n(t) = ...
0
votes
0answers
17 views
Reverse Cumulative Frequency explanation
I've been seeing a lot of Reverse Cumulative Frequency (RCMF) plots lately and trying to figure out how to understand the way reverse cumulative frequencies are calculated, and the interpretation of ...
1
vote
0answers
36 views
Is it possible to convert a pdf obtained with density() to an ecdf?
Consider the following code that gives us (an estimate of) the pdf of a random variable $X$:
X = rnorm(100,10,1)
XDensity = density(X)
I want to obtain the ecdf of ...
2
votes
0answers
47 views
Fitting a function to ECDF [closed]
I have some ECDFs. I would like to summaries the ECDFs with functional approximations. I was thinking that a polynomial, spline, or other line fitting procedure would generate a nice parsimonious ...
0
votes
0answers
20 views
Computing the most representative sample of a random variable
Let $X$ be a real-valued random variable and $n > 0$. Using numerical methods, how can we find the vector $\vec v$ of $n$ real numbers that is most characteristic of $X$, in the sense that the ...
3
votes
1answer
79 views
Median from two unmergable datasets
I am trying to calculate the "overall" median of a variable that is spread across two datasets. I have access to the raw data in each dataset but can't bring their raw data together. What ...
2
votes
0answers
171 views
Why are people using KS statistic to choose cutoff for binary predictions?
I've seen that people are sometimes using Kolmogorov-Smirnov statistic to determine a cutoff in a binary classification models, such as logistic regression. However, i do not fully understand why and ...
3
votes
1answer
49 views
Obtaining an expression for empirical mean from empirical CDF definition
This is my first post so I will try to be as clear and concise as possible. I am doing a course in statistics and we define the true mean of a random gaussian variable to be as follows:
$\mu$ = $\int_{...
1
vote
1answer
267 views
Confidence Interval of p-Quantile from Empirical CDF
I am trying to provide an interval estimate for the 0.8-quantile of some numeric data, which is assumed to be an IID sample from some unknown, continuous distribution. I constructed an Empirical CDF ...
0
votes
0answers
52 views
Calculating the quantile of random forest test cases
I want to calculate the quantile of the observed value of a test case with respect to the prediction interval generated from a random forest, so for each test case I want the proportion of the ...
0
votes
0answers
27 views
Goodness of fit methods for density estimation
If we want to estimate the probability distribution function (pdf) of finite-sampled real continuous data using one of the following approaches:
Parametric density estimation: fit a well-known ...
0
votes
0answers
54 views
Empirical estimation of conditional distribution $Y|X$ at the boundaries of X
I want to estimate conditional distributions of Y | X. Where X contains several continuous covariates.
I'm coding in R. I tried several methods so far, but none gives me entirely satisfactory results ...
1
vote
1answer
55 views
Does the Glivenko-Cantelli theorem work back and forth?
If we have a sample $X_1, X_2, \ldots, X_n \sim F$ then $\hspace{1mm}sup_x|F_n(x) -F(x)|\xrightarrow{a.s./p}0$.
Now, if I can come up with a theoretical cdf $F$ such that $\hspace{1mm}sup_x|F_n(x) -F(...
0
votes
0answers
148 views
Is it accurate to take the maximum distance between CDF and ECDF only at the edges? (Kolmogorov-Smirnov Test)
I have two samples, one obtained empirically and the other is the result of a simulation. I want to tune the simulation so that the result resembles the reality, for that I will minimize the KS ...
0
votes
1answer
196 views
Example of cumulative distribution function and the empirical distribution function [closed]
A random of 100 rolls of the die. The outcomes 1, 2, 3, 4, 5, 6, occurred 13, 19, 10, 17, 14, 27 times, respectively. Calculate the cumulative distribution function and the empirical distribution ...
0
votes
1answer
295 views
How can i find empirical survival function using survival function in R?
The survival function is given by:
S(y; α, λ) = (α/α−1)* (1 − α^(−e^(−λy ))), if α is not equal 1
= e^(−λy) if α =1
y = 1 4 4 7 11 13 15 15 17 18 19 19 20 20 ...
36
votes
4answers
3k views
Intuitive explanation of Kolmogorov Smirnov Test
What is the cleanest, easiest way to explain someone the concept of Kolmogorov Smirnov Test? What does it intuitively mean?
It's a concept that I have difficulty in articulating - especially when ...
3
votes
0answers
144 views
Comparing empirical distributions with an interaction effect
My experiment includes two subject groups: group1 and group2. Both groups undergo a behavioral test, the result of which is called “movement celerity” (this is my dependent variable). Both groups are ...
0
votes
1answer
38 views
Fit cdf and pdf from points on empirical cdf
I have cumulative counts with respect to a variable x, which looks like:
...
1
vote
1answer
60 views
Can we apply the Probability Integral Transform to Dependent Random Variables?
Let's suppose we deal with a non-homogeneous Poisson process having intensity function λ(t), t ≥ 0. The event times X1, X2, … of ...
0
votes
1answer
42 views
How to perform van Zyl's (2018) ecf-based normality test in R [closed]
I have been reading quite a bit on empirical characteristic function (ecf) based normality tests, but cannot find any functions to perform such a test in R and lack the mathematical ability to figure ...
2
votes
0answers
36 views
What are “absolute ECDFs” called (if anybody uses them)?
By absolute ECDF, I am referring to an ECDF that shows the absolute number of samples above some value as opposed to the fraction of such samples.
For example, the bottom plot below displays such an ...
1
vote
0answers
22 views
Quantile-calculation example: Why the term $c$?
I need help to interpret a solution to the following example:
Based on historical claim amounts $x_1,...,x_{47}$, which are assumed to be outcomes from an unknown claim distribution, you are ...
0
votes
2answers
425 views
Simulate data from ECDF in R [closed]
Consider the following 11 continuous non-negative observations:
0, 0.3, 0.31, 0.33, 0.37, 0.49, 0.51, 0.53, 0.59, 0.6
Obtain the empirical cumulative distribution function for these observations.
...
2
votes
1answer
82 views
How to quantify distribution concentration in cdf? [closed]
For example, let's say I have a list of users, each user has its revenue.
I can plot the cdf of both user and revenue, to see if there is some concentration, for example, may be 40% user contribute ...
2
votes
0answers
93 views
Measuring distance between cumulative distribution and empirical distribution
What is an easy to understand step by step procedure on how to compute a distance between a cumulative distribution function and an empirical distribution function given a random sample using ...
8
votes
3answers
1k views
Confidence Interval of CDF
I am trying to determine if there is a statistically meaningful distinction between the cumulative probability density curves shown in the figure below.
It's simple enough to do a $t$-test on the ...
0
votes
0answers
30 views
How can I identify an unfamiliar cumulative distribution function?
I have 116 Bessel-corrected sample variances (average of squared distances from sample mean), each from a sample of three measurements. All measurements were done using the same method.
I had ...
2
votes
0answers
41 views
What linear transformation minimizes the Kolmogorov–Smirnov distance between two sets of points?
Suppose we have two sets of real-valued points $X_1, \dots, X_m$ and $Y_1, \dots, Y_n$. We want to find the linear transformation that makes the distributions look alike. If we want them to have the ...
4
votes
1answer
453 views
MLE as an expectation over the empirical distribution
I am reading Ian Goodfellow "Deep Learning" book. At page 128, it writes the maximum log-likelihood estimator and then says it is equivalent to the expectation over the empirical distribution
To ...
5
votes
1answer
359 views
Maximum likelihood as minimizing the dissimilarity between the empirical distriution and the model distribution
I am reading Ian Goodfellow "Deep Learning" book. At page 128 it says
One way to interpret maximum likelihood estimation is to view it as
minimizing the dissimilarity between the empirical ...
1
vote
1answer
37 views
Determine significance of an observed value
I have an ECDF of values that do not follow a particular distribution (thought they are slightly normal, they are not). And I wish to determine if a new observed value is significant or an outlier or ...
1
vote
0answers
56 views
Empirical 1-sigma confidence interval
I would like to construct a 1-sigma confidence interval for my 1D data. I don't know the underlying distribution, and it is strongly skewed, so standard deviation will not suffice. I see that people ...
