Questions tagged [empirical-cumulative-distr-fn]
Empirical cumulative distribution function: a step function increasing by $1/n$ at each unique $X$-value that occurred in the sample.
131
questions
0
votes
0
answers
12
views
Distance between ecdf and cdf in R [closed]
I want to calculate the distance between ecdf and cdf with kolmogorov–Smirnov and plot the distance as function of the number of samples.
This is my code so far:
How I created "exp_v":
...
- 1
0
votes
0
answers
12
views
Measure for similarity of two distribution ECDF with sensitivity to the tails
Currently, I am looking for a measure to quantify the overall dissimilarity or similarity of two sample distributions (possibly of different size). I would like to compare observed data with model ...
- 11
0
votes
0
answers
11
views
Is the empirical distribution the only unbiased distribution estimator?
Given $n$ samples, if $\hat{p}$ is the empirical distribution of $p \in \Delta_{\mathcal{X}}$ where $\mathcal{X}$ is a finite domain, we know that $\mathbb{E} \hat{p} - p = 0$.
Is the empirical ...
- 41
1
vote
0
answers
13
views
Empirical distribution for feature binning
In paper "A simple yet effective baseline for non-attributed graph classification" (https://arxiv.org/pdf/1811.03508.pdf) authors use empirical distribution for feature binning. Precisely, ...
- 179
2
votes
1
answer
68
views
symmetrization in glivenko-cantelli proof
In this proof of the Glivenko-Cantelli theorem, page 2 of these notes, two types of symmetrization are used. The first transforms the sup of the centered empirical cdf $$P(\sup_{z\in\mathbb{R}}|(1/n)\...
- 21
1
vote
1
answer
24
views
rank to quantile estimate?
R> x=c(92, 3, 1, 4, 15, 4)
R> rank(x)
[1] 6.0 2.0 1.0 3.5 5.0 3.5
Given the rank results of an input vector of sampled data, I can estimate the quantiles ...
- 512
0
votes
0
answers
24
views
Does empirical cumulative distribution function (ECDF) has its Akaike information criterion (AIC)?
Working on multivariate distribution fitting, and right now I have marginal univariate transform models and a copula model. Was thinking if I pick ECDF for marginals, do I still have meaningful AIC?
...
1
vote
1
answer
44
views
How to average several posteriors distributions from a Monte Carlo Simulation
Say you produce several posteriors distributions from different runs of the same model under different seeds. That is to say you have something like the following:
...
- 239
0
votes
0
answers
14
views
Doubt about distribution function of continuous distribution with (Generalized) Pareto tails
In this paper:
https://www.sciencedirect.com/science/article/pii/S0167947315003163
They proposed an estimation method for the parameters of the Generalized Pareto Distribution. Defining:
$$
F_n(x) \...
- 67
0
votes
0
answers
13
views
Expressions for entropy of an observed set of points (continuous variables) [duplicate]
The entropy of a discrete variable can be defined as
$$H(x) = - \sum_{i=1}^n p(x) \log p(x) $$
or for continuous distributions and a density $f(x)$ we could compute
$$h[f] = \int_{\mathbb{X}} f(x) \...
- 46.2k
1
vote
1
answer
66
views
Distance between two categorical distributions
I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same....
- 57
0
votes
0
answers
30
views
Probabilities from ecdf and normalised data - is this acceptable difference? (using R)
I have a dataset of 500 observations, definitely not normally distributed: The minimum is around 50 000 in the data. If I use ecdf to predict e.g. 100 000, I get:
...
- 541
1
vote
1
answer
94
views
What is a "bootstrap from the Empirical CDF"?
I am given the following set of questions.
In all the questions the basic background is a sample X_1, ... , X_n.
If n=3 and you observe the numbers 1, 2, and 4, what is the sample median.
If you ...
- 531
0
votes
1
answer
61
views
Determine a product's rating given a known 3-sigma tolerance
Consider a 100Ω resistor with a 10% tolerance. We can assume this is the 3-sigma value since this is typical in manufacturing. Thus, we can expect ≈99.73% of such resistors to range from 90Ω to 110Ω.
...
- 103
1
vote
1
answer
45
views
0
votes
1
answer
105
views
Smoothed CDF to calculate asymptotic normality
If we have the following estimator: $\hat{F_Z}(z)=\frac{1}{N}\sum_{i=1}^N1\{Z_i\leq z\}$. The CDF of $Z$ is defined as $F_Z(z)=Pr(Z\leq z)$. $Z_1, ..., Z_N$ is i.i.d. data.
What would be the steps to ...
- 233
0
votes
0
answers
62
views
counts under unequally spaced intervals - stackoverflow points table
I am looking for a function similar to ecdf
which help me to asses the quantile for certain number of stackoverflow points across all users.
We have a table with ...
- 93
2
votes
0
answers
85
views
Plug-in principle with kernel density estimate
The plug-in principle says that to estimate a statistical functional of the form
$$
T(\mu) = \int f(x)\ d\mu(x)
$$
we can replace $\mu$ with the empirical distribution $\mu_n$ depending on data $X_1,\...
- 166
7
votes
1
answer
198
views
What are the simplest examples of nonlinear statistical functionals?
I am reading Wasserman's book "All of Statistics" in which he defines a statistical functional as any function $T(F)$ of the cumulative distribution function $F(x)$ that outputs a real ...
- 603
2
votes
1
answer
38
views
Picking a specific estimated CDF from a set of CDFs provided by an ECDF
Let $F_X$ be a CDF of an unknown random variable $X$. If we have independent samples $x_1, x_2, \ldots, x_n$ of $X$ then we can estimate $F_X$ non-parametrically using an ECDF $\hat{F}_n$. By Central ...
- 23
2
votes
2
answers
93
views
Notation for ECDF
I'm reviewing statistical functionals and U-statistics, trying to make notes, and I am tripping on notation. From my understanding, $X$ is used to denote a random variable and $x$ is used to denote an ...
- 31
2
votes
1
answer
174
views
Density estimation from ECDF - numerical derivatives and scaled domains
Suppose we want to get a density estimate of some data X. One way is to compute the empirical CDF,
...
- 43
1
vote
0
answers
26
views
Nonparameteric Empirical Estimator For Stochastic Process
Motivation: If $X$ is a random-variable defined on some probability space $(\Omega,\Sigma,\mathbb{P})$ then Glivenko-Cantelli lemma guarantees that the empirical distribution $\frac1{N}\sum_{n=1}^N \...
- 1
0
votes
0
answers
94
views
How to deal with a PDF that takes CDF data as input? (Metalog distributions)
I am trying to utilise the Metalog distribution in a Machine Learning project.
For this project, I need to be able to obtain likelihoods using the PDF of the distribution.
https://en.wikipedia.org/...
- 61
1
vote
0
answers
59
views
Is it possible to convert a pdf obtained with density() to an ecdf?
Consider the following code that gives us (an estimate of) the pdf of a random variable $X$:
X = rnorm(100,10,1)
XDensity = density(X)
I want to obtain the ecdf of ...
- 830
2
votes
0
answers
90
views
Fitting a function to ECDF [closed]
I have some ECDFs. I would like to summaries the ECDFs with functional approximations. I was thinking that a polynomial, spline, or other line fitting procedure would generate a nice parsimonious ...
- 943
0
votes
0
answers
24
views
Computing the most representative sample of a random variable
Let $X$ be a real-valued random variable and $n > 0$. Using numerical methods, how can we find the vector $\vec v$ of $n$ real numbers that is most characteristic of $X$, in the sense that the ...
- 19.1k
3
votes
1
answer
107
views
Median from two unmergable datasets
I am trying to calculate the "overall" median of a variable that is spread across two datasets. I have access to the raw data in each dataset but can't bring their raw data together. What ...
- 133
2
votes
0
answers
427
views
Why are people using KS statistic to choose cutoff for binary predictions?
I've seen that people are sometimes using Kolmogorov-Smirnov statistic to determine a cutoff in a binary classification models, such as logistic regression. However, i do not fully understand why and ...
- 399
3
votes
1
answer
95
views
Obtaining an expression for empirical mean from empirical CDF definition
This is my first post so I will try to be as clear and concise as possible. I am doing a course in statistics and we define the true mean of a random gaussian variable to be as follows:
$\mu$ = $\int_{...
- 31
1
vote
1
answer
532
views
Confidence Interval of p-Quantile from Empirical CDF
I am trying to provide an interval estimate for the 0.8-quantile of some numeric data, which is assumed to be an IID sample from some unknown, continuous distribution. I constructed an Empirical CDF ...
- 124
0
votes
0
answers
107
views
Empirical estimation of conditional distribution $Y|X$ at the boundaries of X
I want to estimate conditional distributions of Y | X. Where X contains several continuous covariates.
I'm coding in R. I tried several methods so far, but none gives me entirely satisfactory results ...
- 157
1
vote
1
answer
70
views
Does the Glivenko-Cantelli theorem work back and forth?
If we have a sample $X_1, X_2, \ldots, X_n \sim F$ then $\hspace{1mm}sup_x|F_n(x) -F(x)|\xrightarrow{a.s./p}0$.
Now, if I can come up with a theoretical cdf $F$ such that $\hspace{1mm}sup_x|F_n(x) -F(...
- 43
0
votes
0
answers
268
views
Is it accurate to take the maximum distance between CDF and ECDF only at the edges? (Kolmogorov-Smirnov Test)
I have two samples, one obtained empirically and the other is the result of a simulation. I want to tune the simulation so that the result resembles the reality, for that I will minimize the KS ...
- 177
0
votes
1
answer
629
views
Example of cumulative distribution function and the empirical distribution function [closed]
A random of 100 rolls of the die. The outcomes 1, 2, 3, 4, 5, 6, occurred 13, 19, 10, 17, 14, 27 times, respectively. Calculate the cumulative distribution function and the empirical distribution ...
- 81
0
votes
1
answer
609
views
How can i find empirical survival function using survival function in R?
The survival function is given by:
S(y; α, λ) = (α/α−1)* (1 − α^(−e^(−λy ))), if α is not equal 1
= e^(−λy) if α =1
y = 1 4 4 7 11 13 15 15 17 18 19 19 20 20 ...
- 25
37
votes
4
answers
4k
views
Intuitive explanation of Kolmogorov Smirnov Test
What is the cleanest, easiest way to explain someone the concept of Kolmogorov Smirnov Test? What does it intuitively mean?
It's a concept that I have difficulty in articulating - especially when ...
- 2,391
3
votes
0
answers
154
views
Comparing empirical distributions with an interaction effect
My experiment includes two subject groups: group1 and group2. Both groups undergo a behavioral test, the result of which is called “movement celerity” (this is my dependent variable). Both groups are ...
- 71
0
votes
1
answer
92
views
Fit cdf and pdf from points on empirical cdf
I have cumulative counts with respect to a variable x, which looks like:
...
- 1,052
1
vote
1
answer
108
views
Can we apply the Probability Integral Transform to Dependent Random Variables?
Let's suppose we deal with a non-homogeneous Poisson process having intensity function λ(t), t ≥ 0. The event times X1, X2, … of ...
- 163
0
votes
1
answer
50
views
How to perform van Zyl's (2018) ecf-based normality test in R [closed]
I have been reading quite a bit on empirical characteristic function (ecf) based normality tests, but cannot find any functions to perform such a test in R and lack the mathematical ability to figure ...
- 45
2
votes
0
answers
43
views
What are “absolute ECDFs” called (if anybody uses them)?
By absolute ECDF, I am referring to an ECDF that shows the absolute number of samples above some value as opposed to the fraction of such samples.
For example, the bottom plot below displays such an ...
- 2,297
1
vote
0
answers
23
views
Quantile-calculation example: Why the term $c$?
I need help to interpret a solution to the following example:
Based on historical claim amounts $x_1,...,x_{47}$, which are assumed to be outcomes from an unknown claim distribution, you are ...
- 139
0
votes
2
answers
767
views
Simulate data from ECDF in R [closed]
Consider the following 11 continuous non-negative observations:
0, 0.3, 0.31, 0.33, 0.37, 0.49, 0.51, 0.53, 0.59, 0.6
Obtain the empirical cumulative distribution function for these observations.
...
2
votes
1
answer
141
views
How to quantify distribution concentration in cdf? [closed]
For example, let's say I have a list of users, each user has its revenue.
I can plot the cdf of both user and revenue, to see if there is some concentration, for example, may be 40% user contribute ...
- 1,145
2
votes
0
answers
106
views
Measuring distance between cumulative distribution and empirical distribution
What is an easy to understand step by step procedure on how to compute a distance between a cumulative distribution function and an empirical distribution function given a random sample using ...
- 21
8
votes
3
answers
2k
views
Confidence Interval of CDF
I am trying to determine if there is a statistically meaningful distinction between the cumulative probability density curves shown in the figure below.
It's simple enough to do a $t$-test on the ...
- 3,252
0
votes
0
answers
30
views
How can I identify an unfamiliar cumulative distribution function?
I have 116 Bessel-corrected sample variances (average of squared distances from sample mean), each from a sample of three measurements. All measurements were done using the same method.
I had ...
- 67
2
votes
0
answers
59
views
What linear transformation minimizes the Kolmogorov–Smirnov distance between two sets of points?
Suppose we have two sets of real-valued points $X_1, \dots, X_m$ and $Y_1, \dots, Y_n$. We want to find the linear transformation that makes the distributions look alike. If we want them to have the ...
- 219
4
votes
1
answer
752
views
MLE as an expectation over the empirical distribution
I am reading Ian Goodfellow "Deep Learning" book. At page 128, it writes the maximum log-likelihood estimator and then says it is equivalent to the expectation over the empirical distribution
To ...
- 1,370