2
$\begingroup$

This is related to a previous question I have asked, but I am not after visualization but rather a meaningful summary statistic.

Situation: I have many (150k) customers. Each generates his own distribution of shopping times (say, hour of the 24-hour day). Each customer thus has his own empirical distribution of shopping times.

If I want to get a sense of the average within-person variation (distribution, really) in trip times, how might I go about doing this?

Since time is cyclical, what would a meaningful way to measure within-person variance in trip times? 23 should be closer to 0, rather than 20, for example. If I could get something here, then I could plot the distribution of those variances, where someone who shops within 1-2 hours always has very low variance, but someone who shops equally likely at any time has the most variation.

Improve this question
$\endgroup$
3
  • $\begingroup$ For clarification to other readers: the title was edited after my answer was posted. $\endgroup$
    – Andy
    Commented Dec 7, 2014 at 17:43
  • $\begingroup$ Sorry, my bad, the title wasn't as informative as it should have been. Either way I think I'm own my own now, this question is dead to activity. $\endgroup$
    – wolfsatthedoor
    Commented Dec 7, 2014 at 17:53
  • 1
    $\begingroup$ Don't give up on it so easily. It's Sunday so activity is generally lower. Push it again on Monday and maybe it'll get you a better answer. $\endgroup$
    – Andy
    Commented Dec 7, 2014 at 17:59

1 Answer 1

Reset to default
2
$\begingroup$

You can calculate the within mean and within standard deviation by hand. Some statistical packages like Stata can provide such summary statistics. For Stata this would be the xtsum command which displays the overall mean as well as the overall, between, and within standard deviations, minimum, and maximum.

If you have Stata you can try

webuse nlswork
xtset idcode year
xtsum ln_wage

which gives the above mentioned panel data statistics for log earnings of each panel (here individuals).

Alternatively, you can calculate all these statistics by hand. The within mean is $$x_{it} - \overline{x}_i + \overline{\overline{x}}$$ where the overall mean $\overline{\overline{x}}$ is added to make the results comparable across individuals.

If you want the within standard deviation, you need to calculate the within sum of squares as $$WSS = \sum(x_{it} - \overline{x}_i)^2$$ and divide it by $N\overline{T}-1$ degrees of freedom, where $N$ is the number of panels and $\overline{T}$ is the average length of each panel, and take the square root: $$\sigma_{within} = \sqrt{\frac{WSS}{N\overline{T}-1}}$$

I'm not quite clear what your time variable is, i.e. whether it is a 24-hour day or is it several days. Either way, you can also calculate the within standard deviation for each person in a given day and treat hours of the day as the time variable. People whose shopping is more dispersed over the day will have larger values than people who shop within a short time-span.

Again, if you have Stata you can cross check the xtsum results by following the above steps:

// generate the within panel mean
egen yi_mean = mean(ln_wage), by(idcode)
// get the within panel deviations from the mean
gen yi_dev = ln_wage - yi_mean
// square those
gen yi_dev_sq = yi_dev^2
// and sum the up to get WSS
egen wss = sum(yi_dev_sq)

// calculate the number of individuals
egen group = group(idcode)
sum group
local n = r(max)

// calculate the average panel length (probably there is a more elegant way than what I am doing but it works)
bysort idcode: gen N = _N
bysort idcode: gen n = _n
replace N = . if N!=n
sum N
local N_bar = r(mean)

// calculate the within standard deviation
di sqrt(wss/(`N_bar'*`n'-1))
Improve this answer
$\endgroup$
13
  • $\begingroup$ Suppose my variable is just hour of day. You see the problem with calculating the within sum of squares the normal way, right? $\endgroup$
    – wolfsatthedoor
    Commented Dec 7, 2014 at 14:25
  • $\begingroup$ Not really, but you can tell me :) $\endgroup$
    – Andy
    Commented Dec 7, 2014 at 14:32
  • $\begingroup$ Alternatively, you could consider each individual-day combination and take the time distance of each purchase relative to the first purchase a person made. So if I shop first at 8am and then at 9am the distance is only 1 compared to an 8am purchase vs. 8pm purchase (distance = 12) and then take the average of these distances within day. Then take the average of those for each person over all the days in the sample. $\endgroup$
    – Andy
    Commented Dec 7, 2014 at 15:07
  • $\begingroup$ If someone shops only at 11pm and midnight, then they will look like they have a lot if variance in their trip times because 11pm is 23 and midnight is 0. $\endgroup$
    – wolfsatthedoor
    Commented Dec 7, 2014 at 16:39
  • $\begingroup$ Andy the problem with that is that if the first purchase was at an unusual time, and all other trips are made at some other far off hour, then the variance by that metric will look large. $\endgroup$
    – wolfsatthedoor
    Commented Dec 7, 2014 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.