7
$\begingroup$

(I am not strong in statistics so please forgive me if I don't use the correct terms here)

Let's say I have a book with many (1000+) sections, and each section varies significantly in the number of words it contains. My goal is to divide the sections across a period of time (let's say 150 days) where the sections are read in order, but there is as little variance as possible in the number of words read per day.

For example, I need to read every word of a giant tax code book in three months. It's broken into sections that vary from a few sentences to multiple pages. I only want to read complete sections, and I want to spend as close to the same amount of time reading every day as I can.

I've written a script that works like this:
1. I divide the sections evenly across the number of days without regard to words per day. For the remaining sections, I select an evenly spaced collection of days and add a section to each one.
2. I then calculate the actual number of words that ended up in each day, calculate the average, and then select the day that is furthest from the average.
3. Based on that day's word count, I evaluate the standard deviation of the word counts based on 3 scenarios: a) the first section on that day is moved to the previous day, b) the last section on that day is moved to the next day, and c) no change. If a or b results in a lower standard deviation, I keep that change and go back to step 2. If not, I move to the day with the next highest difference from average in words per day and repeat step 3.
4. Eventually the algorithm will be unable to find a change that improves the standard deviation after running step 3 on each day in the set. At this point the script is finished.

I've tested this, it works and provides a set that is smoother than the result of step 1. But I'm not finding the optimal distribution of sections this way ... I still have big peaks that can be fixed manually.

My next thought was to allow for more complex scenarios in step 3 ... looking back/forward more days or moving more sections. But I'm not sure this is going to work.

Is there a name for what I'm trying to do? Has this problem already been solved? Is there a better approach than mine? (Practically speaking my script takes too long to run in real time on a website) I had no luck with searching, but I'm not even sure what I'm searching for. Has someone already created an algorithm to do this? Any better ideas? I'm not sure how to tag the question because I'm just not familiar enough with the terms.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ This is an interesting problem but not really a statistics problem, although your algorithm uses some statistical ideas; it's more of an algorithms problem. You might try asking it at cs.stackexchange.com or programmers.stackexchange.com . $\endgroup$
    – Michael Lugo
    Commented May 4, 2015 at 18:20
  • $\begingroup$ @MichaelLugo I started to wonder that after I'd read my own post a few times. I'm not sure though ... I don't feel like it's something that can only be done with scripting. $\endgroup$
    – EyeVandy
    Commented May 4, 2015 at 21:17
  • $\begingroup$ Can you show an example (as small as possible) that has the features you describe (i.e. that your algorithm leaves peaks you can fix by hand) and stats how much improvement in s.d. you get when yo do the improvement? $\endgroup$
    – Glen_b
    Commented Apr 26, 2016 at 23:07
  • $\begingroup$ Have you thought of using the A* algorithm? I'm not sure there is a deterministic way to solve you problem, but a heuristic algorithm that uses standard deviation as the scoring-metric seems like a good way to go $\endgroup$
    – anguyen
    Commented Apr 27, 2016 at 5:12

1 Answer 1

Reset to default
-1
$\begingroup$

Try looking into MA(moving average) Times series. This evaluates the significance of a average based on data points spread over a specified amount of time(each day for example) and can determine if there are outliers in you data for any specific day and will determine the trending average for your data.

For the step 3 of your process look into exponential smoothing.

Here is a link that may help: http://www.itl.nist.gov/div898/handbook/pmc/section4/pmc4.htm Also wiki: time series MA

Improve this answer
$\endgroup$
3
  • $\begingroup$ Seems like these techniques are for forecasting. I'm not looking to make any predictions here; I want to modify a schedule to reduce the spread among the size of each day's contents. In other words ... given that the standard deviation of my statistical population (word count of each day's reading) is X, how can I algorithmically modify the population to provide a lower standard deviation? $\endgroup$
    – EyeVandy
    Commented May 4, 2015 at 15:44
  • $\begingroup$ okay, i originally misunderstood so to make your SD smaller you have to get your data closer to the mean(which i believe you mentioned) to do so i suggest taking a boxplot of your information and distributing the pages read in a day from the high points on your boxplot to your low points. Essentially bringing your points closer to the mean. Then after you distribute out the pages read re-evaluate your boxplot and all your data points should be inside your box giving you a smaller SD. (Of course this takes some manual intervention but it will get the job done) $\endgroup$
    – DaveRowan
    Commented May 4, 2015 at 15:57
  • $\begingroup$ That's essentially what my algorithm does, but it's limited in how far it can distribute the sections (right now one day forward or backward) so it doesn't smooth out the peaks as much as it could. Extending that idea might make it impractical to calculate. $\endgroup$
    – EyeVandy
    Commented May 4, 2015 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.