Finding Cumulative Distribution Function of weighted data

Question

I have census tract data where each row holds the population size and a variable value (e.g., income). I want to plot the cumulative distribution function (CDF) of the TRUE population, i.e., accounting for the population size in each tract. So if my data has ~3000 census tracts, the entire population is ~ 8 milion.

I tried converting the data to a long form, but the result is a very large DF that is hard to plot and analyse (my real data have ten years of data, so it creates a DF with 80 milion rows).

Here is a sample data and the long form version I tried:

df = data.frame(PopSize = rnorm(20,1000,300),
                Income  = rnorm(20,100,30))
df_long = df[rep(seq_len(nrow(df)), times=df$PopSize),]

This converts the 20 rows with a sample size of ~1000 in each, to a DF with ~20,000 rows, making it impracticle to work with in my real data of ~8 milion.

Is there a way to create a CDF plot that will account for tha sample size in each row? Thanks

Answer 1

The empirical cumulative distribution function (ECDF), describes at every $x$ point (income in this case), the proportion of the population that is less or equal to that $x$ value. It doesn't matter whether your data has one row per observation, or each row groups multiple observations. It can still be calculated.

Basing the equation off the Wikipedia page notation the ECDF is:

$$ \hat F_n(x) = \frac{\text{number of elements}<x}{n} $$

• $n$ is constant, and in your example is sum(df$PopSize)
• the number of elements less than $x$ is the cumulative sum of df$PopSize less than $x$

In R language:

set.seed(123)
df = data.frame(PopSize = rnorm(20,1000,300),
                Income  = rnorm(20,100,30))
df <- df[order(df$Income),]
plot(
  x = c(0, cumsum(df$Income)) # 0 because your x variable is a population
  , y = c(0, cumsum(df$PopSize)/sum(df$PopSize))
  , type = "s"
  , xlim = c(0, sum(df$Income))
  , ylim = c(0, 1)
)