How should I account for small samples in a larger data set? Should I remove them?

Question

We're trying to develop a simple tool that will help our teams optimise their clients' Facebook posting strategies. In our experience, time of post can have a big impact on audience response; but those times will differ from audience to audience based on their behaviours (can they access Facebook from work? What times of day are they online? When are they at a loose end, and when are other activities taking up their concentration?) We expect responsiveness to vary throughout the day for these (and other known & unknown) reasons. Furthermore, of course, time of day isn't the only reason that response rates vary; the content of the post is one very significant reason.

I don't know if this background is useful; whether it makes the following question clearer.

The following R script (I lack sufficient reputation to post charts) contains data that cover posting activity and audience response by hour of post. You'll notice that the 6 am point (for which we have two observations) massively outperforms the rest of the day. We often see these very high response rates for hours that have fewer observations.

post.hour <- c(0:23)
posts <- c(0,0,0,0,0,0,2,15,16,10,17,13,29,21,23,18,29,24,34,42,51,48,49,17)
response <- c(0,0,0,0,0,0,5282,8627,6080,2716,2831,3258,6291,7756,4008,4614,11838,2611,10527,14706,5416,10970,19098,9505)
mean.response <- response/posts

d <- data.frame(post.hour,posts,response,mean.response)

library(ggplot2)

response.chart <- ggplot(d,aes(post.hour,mean.response)) +
 geom_point() + geom_line() +
 ylab("mean response") +
 opts (title="mean audience response by post hour")
response.chart

I've tried manually removing hours that have fewer than a certain number of observations:

This seems unsatisfactory, and (furthermore) is hard to automate across widely differing data sets.

1. Should I remove these hours? If so, what statistical tools should I use to perform this repeatedly and automatically across multiple data sets?

2. If the answer is "no" -- and I suspect that it might be -- what should I do to account for these tiny sample sizes?

Conscious that this is possibly Statistics 101 stuff. It should be clear from this that I have not - in fact - had any statistical training. I'd greatly appreciate you taking this into account if and when you respond to this question. I'm not even sure that the hourly data represent samples...