Questions tagged [empirical-cumulative-distr-fn]
Empirical cumulative distribution function: a step function increasing by $1/n$ at each unique $X$-value that occurred in the sample.
153
questions
3
votes
1
answer
87
views
Finding Cumulative Distribution Function of weighted data
I have census tract data where each row holds the population size and a variable value (e.g., income). I want to plot the cumulative distribution function (CDF) of the TRUE population, i.e., ...
0
votes
0
answers
36
views
Why not use eCDF to choose the model of survivability?
I have been reading a little bit about survivability and I am a little confused about choosing models. Let me set an experiment:
You run 50 toasters until all of them fail, noting down their lifespan $...
1
vote
0
answers
45
views
How do you test if two discrete ECDFs are drawn from the same population?
Background
I have two Empirical Cumulative Distribution Functions (ECDFs) based on two samples of very different sizes.
Sample 1:
1020 data points, Power-Law-like distribution, discrete data in the ...
0
votes
0
answers
18
views
Resample random variable to fit different variance
Suppose I have samples drawn from a random variable, and I want to multiply that random variable with a scalar constant. How should I transform the samples such that they would have been drawn from ...
1
vote
0
answers
37
views
Proof of empirical expression of Cramer-Von Mises Statistics
I would like to know how to derive the empirical expression of the test statistics below:
$$
\omega^2 = \sum_{i=1}^{n}(U_{(i)}-\frac{2i-1}{2n})^2 + \frac{1}{12n}
$$
from
$$
\omega^2 = n \int_\Omega(...
1
vote
0
answers
58
views
Confidence interval for sum of product of scaled binomial random variables
I have discrete, independent, but not necessarily identically distributed random variables $X_1,\dots,X_n$ that take on non-negative integer values. Each random variable has unknown distribution ...
0
votes
0
answers
144
views
Comparing the output distribution of two ML models
Consider a regression task (e.g. predicting house prices) with a given train and test sets.
We start with constructing a linear regression model, in which we assume $y_i=X^T\beta+\epsilon$ with $E[\...
1
vote
1
answer
43
views
Transforming data with a fitted distribution function
I have a bivariate dataset on $[0,1]^2$ in which I am interested in fitting a joint distribution. I fit a Gaussian copula but am unsure how to judge if it's a good fit. I tried transforming my data ...
1
vote
1
answer
55
views
Adjustment needed for multivariate Dvoretzky–Kiefer–Wolfowitz inequality on MCMC samples?
I was thinking about studying bounds on the multivariate empirical cumulative distribution function for samples from an MCMC chains. The multivariate Dvoretzky–Kiefer–Wolfowitz inequality would seem ...
2
votes
1
answer
98
views
Covariance between two binomial random variables or expectation of product of binomial random variables
I have an empirical distribution $S_n(x)$ (= proportion of samples less than equal to x) from a random sample $X_1, X_2, ..., X_n$ for a random variable $X \sim F_X$. Consider the random variable $T_n(...
4
votes
1
answer
173
views
Kolmogorov-Smirnov test with median ranks instead of traditional empirical distribution function
I know that Kolmogorov-Smirnov test uses the empirical distribution function of the sample studied $\widehat{F}(X_i) = \frac{i}{n}$ and then measures the adequacy of function $\widehat{F}$ to function ...
2
votes
1
answer
85
views
Convergence of histograms
Good morning,
I'm interested in the convergence of empirical distribution functions. In particular, let us suppose to have a population with unknown distribution function $ F(x) $ and that I can ...
0
votes
0
answers
22
views
Random variables implicit in Glivenko-Cantelli theorem
Theorem 1 here states $\lim_{n\to\infty} \sup_x |\mathbb{F}_n(x) - F(x)| = 0$ with probability $1$, where $\mathbb{F}_n$ is the empirical distribution function of the first $n$ $X$s, which are i.i.d. ...
0
votes
1
answer
94
views
Compare CCDF of datasets with different sample size
I have 3 dataframes with the same structure (each dataframe includes a different type of tweet). Here are the columns of dataframes: id, ...
0
votes
0
answers
51
views
Estimation of Distribution using multiple ECDFs
Every day, I keep track of the processing times for each input to my CPU and create empirical cumulative distribution functions (ECDFs) based on this data. Let's assume I have 100 observations per day ...
2
votes
2
answers
185
views
Quantize a continuous random variable
Suppose we have a continuous random variable $X$. We do not know its distribution function, but have $n$ i.i.d. samples.
I am looking for methods that quantize (discretize) $X$ into a categorical ...
1
vote
0
answers
16
views
How to apply a seasonal index into an eCDF?
I am doing a Before/After analysis, which is aiming at evaluating the effect of a change. Assume I have made a change at the beginning of June-2022, and I want to evaluate the effect of the change ...
0
votes
0
answers
80
views
Averaging ECDF vertically : Proof of convergence
Suppose we have a set A , we split into multiple disjoint subsets ai
We only have access to the ai sets , is there a way to compute the ECDF for the set A without looking at it ?
If for example we ...
0
votes
0
answers
64
views
Doubt in empirical distribution, sample and random variable
I have a basic doubt in sample and random variables. I have read related posts on this site but still some doubt is still left.
Suppose we have a population and we are drawing some entries from it ...
0
votes
0
answers
86
views
How to compare two empirical CDFs obtained from current status data
I have failure time information of a product in the form of current status data, which looks something like this.
Observed Time($t_i$)
failed ? ($y_i$)
5 hrs
1
6 hrs
0
5 hrs
0
7 hrs
1
...
0
votes
1
answer
138
views
How to plot a confidence band around the ecdf in the form (ecdf-b, ecdf+b) for a certain b in R?
I need to plot a confidence band around my ecdf. I calculated a value b and I basically just need to plot ecdf+b and ecdf-b but R doesnt let me do that. Does anyone know how this can be done?
0
votes
1
answer
73
views
Empirical distribution function by sampling from a m.v. distribution
I have mathematically rewritten my problem as a function of multiple iid variables:
$$
f(X_1, X_2, ..., X_n),
$$
where $$X_i \in \mathcal{N}(0,1)$$
I now want to determine the empirical distribution ...
9
votes
1
answer
490
views
When was the earliest appearance of Empirical Cumulative Distribution Plots?
I would be surprised if we actually had a date here. I am curious who, if anyone, created the ecdf plot. When did the ecdf make its first appearance? If we do not know when the first ecdf plot was ...
5
votes
1
answer
98
views
Does this statistic comparing two samples using EDFs have a name?
So, I wanted to cook up a statistic that was similar to the Wasserstein metric for finite sized samples from distributions on a continuous support that is also invariant to reparameterization of the 1-...
2
votes
2
answers
43
views
How to estimate correlations between the standard errors on empirical quantile estimates from two correlated series?
I have two very long series of samples from a pair of correlated random variables, $x$ and $y$.
(they are fat tailed distributions, roughly Pareto, if that helps).
(if you're curious...$x$ and $y$ are ...
6
votes
1
answer
307
views
Is there an equivalent to an ECDF with a "<" sign?
The empirical cumulative distribution function is defined as
$$
F(x)=\frac{1}{n} \sum_{i=1}^{n} \mathbb I_{x_i \leq x}
$$
Is there an equivalent interpretation of this function as:
$$
F(x)=\frac{1}{n} ...
3
votes
1
answer
81
views
Covariance of the empirical probability mass function
Suppose a discrete random variable $Y$ takes $k$ levels of different values $y_1,y_2,...,y_k$. Let $P(Y=y_k):=p_k$.
Suppose we have $n$ i.i.d. samples of $Y$, my question is:
How can we compute the ...
2
votes
0
answers
20
views
Simulating New Observations from a Empirical Distribution Function [duplicate]
Recently, I found out that it is possible to simulate new observations from an Empirical Distribution Function (EDF) . Suppose I collected some data (e.g. heights of baksetball players): $x_1, x_2,\...
1
vote
0
answers
105
views
Empirical distribution for feature binning
In paper "A simple yet effective baseline for non-attributed graph classification" (https://arxiv.org/pdf/1811.03508.pdf) authors use empirical distribution for feature binning. Precisely, ...
2
votes
1
answer
141
views
symmetrization in glivenko-cantelli proof
In this proof of the Glivenko-Cantelli theorem, page 2 of these notes, two types of symmetrization are used. The first transforms the sup of the centered empirical cdf $$P(\sup_{z\in\mathbb{R}}|(1/n)\...
1
vote
1
answer
53
views
rank to quantile estimate?
R> x=c(92, 3, 1, 4, 15, 4)
R> rank(x)
[1] 6.0 2.0 1.0 3.5 5.0 3.5
Given the rank results of an input vector of sampled data, I can estimate the quantiles ...
1
vote
1
answer
313
views
How to average several posteriors distributions from a Monte Carlo Simulation
Say you produce several posteriors distributions from different runs of the same model under different seeds. That is to say you have something like the following:
...
2
votes
1
answer
1k
views
Distance between two categorical distributions
I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same....
0
votes
0
answers
153
views
Probabilities from ecdf and normalised data - is this acceptable difference? (using R)
I have a dataset of 500 observations, definitely not normally distributed: The minimum is around 50 000 in the data. If I use ecdf to predict e.g. 100 000, I get:
...
2
votes
1
answer
660
views
What is a "bootstrap from the Empirical CDF"?
I am given the following set of questions.
In all the questions the basic background is a sample X_1, ... , X_n.
If n=3 and you observe the numbers 1, 2, and 4, what is the sample median.
If you ...
0
votes
1
answer
618
views
Determine a product's rating given a known 3-sigma tolerance
Consider a 100Ω resistor with a 10% tolerance. We can assume this is the 3-sigma value since this is typical in manufacturing. Thus, we can expect ≈99.73% of such resistors to range from 90Ω to 110Ω.
...
1
vote
1
answer
57
views
Difference in means
For a df that looks something like the following
...
0
votes
1
answer
144
views
Smoothed CDF to calculate asymptotic normality
If we have the following estimator: $\hat{F_Z}(z)=\frac{1}{N}\sum_{i=1}^N1\{Z_i\leq z\}$. The CDF of $Z$ is defined as $F_Z(z)=Pr(Z\leq z)$. $Z_1, ..., Z_N$ is i.i.d. data.
What would be the steps to ...
2
votes
0
answers
135
views
Plug-in principle with kernel density estimate
The plug-in principle says that to estimate a statistical functional of the form
$$
T(\mu) = \int f(x)\ d\mu(x)
$$
we can replace $\mu$ with the empirical distribution $\mu_n$ depending on data $X_1,\...
7
votes
1
answer
270
views
What are the simplest examples of nonlinear statistical functionals?
I am reading Wasserman's book "All of Statistics" in which he defines a statistical functional as any function $T(F)$ of the cumulative distribution function $F(x)$ that outputs a real ...
2
votes
1
answer
61
views
Picking a specific estimated CDF from a set of CDFs provided by an ECDF
Let $F_X$ be a CDF of an unknown random variable $X$. If we have independent samples $x_1, x_2, \ldots, x_n$ of $X$ then we can estimate $F_X$ non-parametrically using an ECDF $\hat{F}_n$. By Central ...
2
votes
2
answers
220
views
Notation for ECDF
I'm reviewing statistical functionals and U-statistics, trying to make notes, and I am tripping on notation. From my understanding, $X$ is used to denote a random variable and $x$ is used to denote an ...
3
votes
1
answer
547
views
Density estimation from ECDF - numerical derivatives and scaled domains
Suppose we want to get a density estimate of some data X. One way is to compute the empirical CDF,
...
1
vote
0
answers
31
views
Nonparameteric Empirical Estimator For Stochastic Process
Motivation: If $X$ is a random-variable defined on some probability space $(\Omega,\Sigma,\mathbb{P})$ then Glivenko-Cantelli lemma guarantees that the empirical distribution $\frac1{N}\sum_{n=1}^N \...
0
votes
0
answers
172
views
How to deal with a PDF that takes CDF data as input? (Metalog distributions)
I am trying to utilise the Metalog distribution in a Machine Learning project.
For this project, I need to be able to obtain likelihoods using the PDF of the distribution.
https://en.wikipedia.org/...
1
vote
0
answers
173
views
Is it possible to convert a pdf obtained with density() to an ecdf?
Consider the following code that gives us (an estimate of) the pdf of a random variable $X$:
X = rnorm(100,10,1)
XDensity = density(X)
I want to obtain the ecdf of ...
2
votes
0
answers
301
views
Fitting a function to ECDF [closed]
I have some ECDFs. I would like to summaries the ECDFs with functional approximations. I was thinking that a polynomial, spline, or other line fitting procedure would generate a nice parsimonious ...
0
votes
0
answers
36
views
Computing the most representative sample of a random variable
Let $X$ be a real-valued random variable and $n > 0$. Using numerical methods, how can we find the vector $\vec v$ of $n$ real numbers that is most characteristic of $X$, in the sense that the ...
3
votes
1
answer
635
views
Median from two unmergable datasets
I am trying to calculate the "overall" median of a variable that is spread across two datasets. I have access to the raw data in each dataset but can't bring their raw data together. What ...
3
votes
0
answers
1k
views
Why are people using KS statistic to choose cutoff for binary predictions?
I've seen that people are sometimes using Kolmogorov-Smirnov statistic to determine a cutoff in a binary classification models, such as logistic regression. However, i do not fully understand why and ...