Questions tagged [empirical-cumulative-distr-fn]
Empirical cumulative distribution function: a step function increasing by $1/n$ at each unique $X$-value that occurred in the sample.
146
questions
0
votes
0
answers
12
views
Random variables implicit in Glivenko-Cantelli theorem
Theorem 1 here states $\lim_{n\to\infty} \sup_x |\mathbb{F}_n(x) - F(x)| = 0$ with probability $1$, where $\mathbb{F}_n$ is the empirical distribution function of the first $n$ $X$s, which are i.i.d. ...
- 289
0
votes
1
answer
39
views
+50
Compare CCDF of datasets with different sample size
I have 3 dataframes with the same structure (each dataframe includes a different type of tweet). Here are the columns of dataframes: id, ...
- 101
0
votes
0
answers
25
views
Estimation of Distribution using multiple ECDFs
Every day, I keep track of the processing times for each input to my CPU and create empirical cumulative distribution functions (ECDFs) based on this data. Let's assume I have 100 observations per day ...
- 33
1
vote
2
answers
73
views
Quantize a continuous random variable
Suppose we have a continuous random variable $X$. We do not know its distribution function, but have $n$ i.i.d. samples.
I am looking for methods that quantize (discretize) $X$ into a categorical ...
- 361
1
vote
0
answers
14
views
How to apply a seasonal index into an eCDF?
I am doing a Before/After analysis, which is aiming at evaluating the effect of a change. Assume I have made a change at the beginning of June-2022, and I want to evaluate the effect of the change ...
- 111
0
votes
0
answers
28
views
How to simulate critical values for a two sample KS test?
I believe I managed to simulate critical values for a single sample with the following code in R.
...
- 11
0
votes
0
answers
62
views
Averaging ECDF vertically : Proof of convergence
Suppose we have a set A , we split into multiple disjoint subsets ai
We only have access to the ai sets , is there a way to compute the ECDF for the set A without looking at it ?
If for example we ...
0
votes
0
answers
51
views
Doubt in empirical distribution, sample and random variable
I have a basic doubt in sample and random variables. I have read related posts on this site but still some doubt is still left.
Suppose we have a population and we are drawing some entries from it ...
- 115
0
votes
0
answers
40
views
How to compare two empirical CDFs obtained from current status data
I have failure time information of a product in the form of current status data, which looks something like this.
Observed Time($t_i$)
failed ? ($y_i$)
5 hrs
1
6 hrs
0
5 hrs
0
7 hrs
1
...
0
votes
1
answer
55
views
How to plot a confidence band around the ecdf in the form (ecdf-b, ecdf+b) for a certain b in R?
I need to plot a confidence band around my ecdf. I calculated a value b and I basically just need to plot ecdf+b and ecdf-b but R doesnt let me do that. Does anyone know how this can be done?
- 1
0
votes
1
answer
37
views
Empirical distribution function by sampling from a m.v. distribution
I have mathematically rewritten my problem as a function of multiple iid variables:
$$
f(X_1, X_2, ..., X_n),
$$
where $$X_i \in \mathcal{N}(0,1)$$
I now want to determine the empirical distribution ...
9
votes
1
answer
451
views
When was the earliest appearance of Empirical Cumulative Distribution Plots?
I would be surprised if we actually had a date here. I am curious who, if anyone, created the ecdf plot. When did the ecdf make its first appearance? If we do not know when the first ecdf plot was ...
- 1,801
4
votes
1
answer
89
views
Does this statistic comparing two samples using EDFs have a name?
So, I wanted to cook up a statistic that was similar to the Wasserstein metric for finite sized samples from distributions on a continuous support that is also invariant to reparameterization of the 1-...
- 171
0
votes
0
answers
40
views
Substitution principle - Sample mean estimator
We can read in Keith Knight - Mathematical Statistics Example 4.22 the following.
EXAMPLE 4.22: Suppose that $\theta(F)=\int_{-\infty}^{\infty} h(x) d F(x)$. Substituting $\widehat{F}$ for $F$, we get
...
- 147
1
vote
2
answers
31
views
How to estimate correlations between the standard errors on empirical quantile estimates from two correlated series?
I have two very long series of samples from a pair of correlated random variables, $x$ and $y$.
(they are fat tailed distributions, roughly Pareto, if that helps).
(if you're curious...$x$ and $y$ are ...
0
votes
0
answers
11
views
Reference request: Almost sure weak convergence
(Reposting from Math StackExchange)
I've encountered the term "almost sure weak convergence" of empirical measures in several places but haven't been able to find a textbook reference. Could ...
- 163
6
votes
1
answer
275
views
Is there an equivalent to an ECDF with a "<" sign?
The empirical cumulative distribution function is defined as
$$
F(x)=\frac{1}{n} \sum_{i=1}^{n} \mathbb I_{x_i \leq x}
$$
Is there an equivalent interpretation of this function as:
$$
F(x)=\frac{1}{n} ...
- 63
0
votes
0
answers
265
views
Estimating density function at specific percentile using empirical cdf
Summary
I'm running an experiment where I'm using the empirical CDF of a known random variable to approximate the density of the random variable at a specific percentile. In general I'm interested in ...
- 1
0
votes
0
answers
19
views
Simplifying empirical distribution functions
I have an empirical distribution function, that is I have say 2000 points $x_i$ each with a probability $w_i>0$ and $\sum w_i=1$ .
Now, I want to reduce this empirical probability distribution by ...
0
votes
0
answers
103
views
Convergence of standarized empirical distribution functions to distribution of standarized sample mean
Suppose we have a family of $i.i.d.$ random variables $\{X_n\}_{n=1}^\infty$, with distribution function $F(\cdot;\mu,\sigma^2)$, with mean $\mu\in\mathbb{R}$ and variance $\sigma^2>0$. By virtue ...
- 312
2
votes
1
answer
64
views
Covariance of the empirical probability mass function
Suppose a discrete random variable $Y$ takes $k$ levels of different values $y_1,y_2,...,y_k$. Let $P(Y=y_k):=p_k$.
Suppose we have $n$ i.i.d. samples of $Y$, my question is:
How can we compute the ...
- 361
2
votes
0
answers
20
views
Simulating New Observations from a Empirical Distribution Function [duplicate]
Recently, I found out that it is possible to simulate new observations from an Empirical Distribution Function (EDF) . Suppose I collected some data (e.g. heights of baksetball players): $x_1, x_2,\...
- 7,142
1
vote
0
answers
82
views
Empirical distribution for feature binning
In paper "A simple yet effective baseline for non-attributed graph classification" (https://arxiv.org/pdf/1811.03508.pdf) authors use empirical distribution for feature binning. Precisely, ...
- 229
2
votes
1
answer
118
views
symmetrization in glivenko-cantelli proof
In this proof of the Glivenko-Cantelli theorem, page 2 of these notes, two types of symmetrization are used. The first transforms the sup of the centered empirical cdf $$P(\sup_{z\in\mathbb{R}}|(1/n)\...
- 21
1
vote
1
answer
35
views
rank to quantile estimate?
R> x=c(92, 3, 1, 4, 15, 4)
R> rank(x)
[1] 6.0 2.0 1.0 3.5 5.0 3.5
Given the rank results of an input vector of sampled data, I can estimate the quantiles ...
- 542
1
vote
1
answer
161
views
How to average several posteriors distributions from a Monte Carlo Simulation
Say you produce several posteriors distributions from different runs of the same model under different seeds. That is to say you have something like the following:
...
- 219
2
votes
1
answer
688
views
Distance between two categorical distributions
I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same....
- 67
0
votes
0
answers
81
views
Probabilities from ecdf and normalised data - is this acceptable difference? (using R)
I have a dataset of 500 observations, definitely not normally distributed: The minimum is around 50 000 in the data. If I use ecdf to predict e.g. 100 000, I get:
...
- 677
2
votes
1
answer
410
views
What is a "bootstrap from the Empirical CDF"?
I am given the following set of questions.
In all the questions the basic background is a sample X_1, ... , X_n.
If n=3 and you observe the numbers 1, 2, and 4, what is the sample median.
If you ...
- 571
0
votes
1
answer
320
views
Determine a product's rating given a known 3-sigma tolerance
Consider a 100Ω resistor with a 10% tolerance. We can assume this is the 3-sigma value since this is typical in manufacturing. Thus, we can expect ≈99.73% of such resistors to range from 90Ω to 110Ω.
...
- 103
1
vote
1
answer
56
views
0
votes
1
answer
132
views
Smoothed CDF to calculate asymptotic normality
If we have the following estimator: $\hat{F_Z}(z)=\frac{1}{N}\sum_{i=1}^N1\{Z_i\leq z\}$. The CDF of $Z$ is defined as $F_Z(z)=Pr(Z\leq z)$. $Z_1, ..., Z_N$ is i.i.d. data.
What would be the steps to ...
- 233
2
votes
0
answers
124
views
Plug-in principle with kernel density estimate
The plug-in principle says that to estimate a statistical functional of the form
$$
T(\mu) = \int f(x)\ d\mu(x)
$$
we can replace $\mu$ with the empirical distribution $\mu_n$ depending on data $X_1,\...
- 196
7
votes
1
answer
236
views
What are the simplest examples of nonlinear statistical functionals?
I am reading Wasserman's book "All of Statistics" in which he defines a statistical functional as any function $T(F)$ of the cumulative distribution function $F(x)$ that outputs a real ...
- 603
2
votes
1
answer
52
views
Picking a specific estimated CDF from a set of CDFs provided by an ECDF
Let $F_X$ be a CDF of an unknown random variable $X$. If we have independent samples $x_1, x_2, \ldots, x_n$ of $X$ then we can estimate $F_X$ non-parametrically using an ECDF $\hat{F}_n$. By Central ...
- 23
2
votes
2
answers
153
views
Notation for ECDF
I'm reviewing statistical functionals and U-statistics, trying to make notes, and I am tripping on notation. From my understanding, $X$ is used to denote a random variable and $x$ is used to denote an ...
- 31
3
votes
1
answer
369
views
Density estimation from ECDF - numerical derivatives and scaled domains
Suppose we want to get a density estimate of some data X. One way is to compute the empirical CDF,
...
- 53
1
vote
0
answers
28
views
Nonparameteric Empirical Estimator For Stochastic Process
Motivation: If $X$ is a random-variable defined on some probability space $(\Omega,\Sigma,\mathbb{P})$ then Glivenko-Cantelli lemma guarantees that the empirical distribution $\frac1{N}\sum_{n=1}^N \...
- 544
0
votes
0
answers
148
views
How to deal with a PDF that takes CDF data as input? (Metalog distributions)
I am trying to utilise the Metalog distribution in a Machine Learning project.
For this project, I need to be able to obtain likelihoods using the PDF of the distribution.
https://en.wikipedia.org/...
- 101
1
vote
0
answers
134
views
Is it possible to convert a pdf obtained with density() to an ecdf?
Consider the following code that gives us (an estimate of) the pdf of a random variable $X$:
X = rnorm(100,10,1)
XDensity = density(X)
I want to obtain the ecdf of ...
- 900
2
votes
0
answers
237
views
Fitting a function to ECDF [closed]
I have some ECDFs. I would like to summaries the ECDFs with functional approximations. I was thinking that a polynomial, spline, or other line fitting procedure would generate a nice parsimonious ...
- 1,801
0
votes
0
answers
29
views
Computing the most representative sample of a random variable
Let $X$ be a real-valued random variable and $n > 0$. Using numerical methods, how can we find the vector $\vec v$ of $n$ real numbers that is most characteristic of $X$, in the sense that the ...
- 19.5k
3
votes
1
answer
367
views
Median from two unmergable datasets
I am trying to calculate the "overall" median of a variable that is spread across two datasets. I have access to the raw data in each dataset but can't bring their raw data together. What ...
- 133
3
votes
0
answers
834
views
Why are people using KS statistic to choose cutoff for binary predictions?
I've seen that people are sometimes using Kolmogorov-Smirnov statistic to determine a cutoff in a binary classification models, such as logistic regression. However, i do not fully understand why and ...
- 441
3
votes
1
answer
209
views
Obtaining an expression for empirical mean from empirical CDF definition
This is my first post so I will try to be as clear and concise as possible. I am doing a course in statistics and we define the true mean of a random gaussian variable to be as follows:
$\mu$ = $\int_{...
- 31
1
vote
1
answer
1k
views
Confidence Interval of p-Quantile from Empirical CDF
I am trying to provide an interval estimate for the 0.8-quantile of some numeric data, which is assumed to be an IID sample from some unknown, continuous distribution. I constructed an Empirical CDF ...
- 124
0
votes
0
answers
223
views
Empirical estimation of conditional distribution $Y|X$ at the boundaries of X
I want to estimate conditional distributions of Y | X. Where X contains several continuous covariates.
I'm coding in R. I tried several methods so far, but none gives me entirely satisfactory results ...
- 157
1
vote
1
answer
104
views
Does the Glivenko-Cantelli theorem work back and forth?
If we have a sample $X_1, X_2, \ldots, X_n \sim F$ then $\hspace{1mm}sup_x|F_n(x) -F(x)|\xrightarrow{a.s./p}0$.
Now, if I can come up with a theoretical cdf $F$ such that $\hspace{1mm}sup_x|F_n(x) -F(...
- 43
0
votes
0
answers
455
views
Is it accurate to take the maximum distance between CDF and ECDF only at the edges? (Kolmogorov-Smirnov Test)
I have two samples, one obtained empirically and the other is the result of a simulation. I want to tune the simulation so that the result resembles the reality, for that I will minimize the KS ...
- 197
0
votes
1
answer
1k
views
Example of cumulative distribution function and the empirical distribution function [closed]
A random of 100 rolls of the die. The outcomes 1, 2, 3, 4, 5, 6, occurred 13, 19, 10, 17, 14, 27 times, respectively. Calculate the cumulative distribution function and the empirical distribution ...
- 81