1
$\begingroup$

For a project, I've collected some basic information on weather, and some basic information on births. Both data sets are focused on a single state, Massachusetts. The weather set contains very basic information: the weather event (hail, winter storm, etc.), and the month it occurred in. Not very granular. Similarly, the birth records are merely total births in a given month.

The birth data set contains records from 2007-2011. The winter weather set contains records from 1996-2013. So I'll only be using some of the winter weather records.

The goal of this project is somewhat tongue-in-cheek, but serious in nature. I hope to see if there is a correlation between serious winter weather events (in a given month), and a spike in births nine months later; i.e., do bad storms keep people indoors, and does that lead to more births. I would expect to see an increase from whatever the baseline is, when there is a similar spike in serious winter weather events.

I'm not looking for anybody to do this project for me, but hints and suggestions are very welcome. I was planning to use associative rules mining (and the "arules" package) to accomplish this goal, but I haven't had any success. I guess my data isn't organized as "transaction data", so I'm not sure what type of data mining technique to use here (I've only had limited experience with arules). Any suggestions?

Using rbind, I've already combined the two data sets into two distinct files, which contain all the relevant information.

combined.weather1 <- includes three variables (YEAR, MONTH, EVENT_TYPE), and 2750 observations. It contains info from 1996-2013. The YEAR variable is integer, while the MONTH and EVENT_TYPE variables are factors.

birth.data <- includes three variable (YEAR, MONTH, BIRTH_TOTAL), and sixty observations. It contains data from 2007-2011. The YEAR variable is integer, while the MONTH and BIRTH_TOTAL variables are factors.

To further complicate things, I'd like to rule out some of the less serious winter weather events, such as "Hail", "Frost/Freeze", and "Winter Weather".

Worth noting, there are no missing or NA values in any of these records. The data is clean and complete (and took forever to prepare!).

So I need to select a data mining technique, first and foremost. From there, I would very much appreciate if anybody can offer help on how to set it up to compare these two simple data sets, while factoring in a nine month difference between certain winter weather events and (possible) spikes in births.

EDIT: adding examples, per request.

dput(combined.weather1[1:10],)

    structure(list(YEAR = c(1996L, 1996L, 1996L, 1996L, 1996L, 1996L, 
    1996L, 1996L, 1996L, 1996L), MONTH_NAME = structure(c(5L, 5L, 
    5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L), .Label = c("April    ", "August   ", 
    "December ", "February ", "January  ", "July     ", "June     ", 
    "March    ", "May      ", "November ", "September", "October  ", 
    "April", "August", "December", "February", "January", "July", 
    "June", "March", "May", "November", "October"), class = "factor"), 
        EVENT_TYPE = structure(c(2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
        2L, 2L), .Label = c("Hail", "Heavy Snow", "Winter Storm", 
        "Winter Weather", "Ice Storm", "Frost/Freeze", "WINTER WEATHER", 
        "Blizzard"), class = "factor")), .Names = c("YEAR", "MONTH_NAME", 
    "EVENT_TYPE"), row.names = c(NA, 10L), class = "data.frame")

dput(birth.data[1:10,])

    structure(list(YEAR = c(2007L, 2007L, 2007L, 2007L, 2007L, 2007L, 
    2007L, 2007L, 2007L, 2007L), MONTH = structure(c(5L, 4L, 8L, 
    1L, 9L, 7L, 6L, 2L, 12L, 11L), .Label = c("April", "August", 
    "December", "February", "January", "July", "June", "March", "May", 
    "November", "October", "September"), class = "factor"), BIRTH_TOTAL = structure(c(29L, 
    9L, 49L, 27L, 57L, 52L, 58L, 59L, 36L, 41L), .Label = c("5,414", 
    "5,459", "5,515", "5,627", "5,667", "5,730", "5,754", "5,801", 
    "5,833", "5,853", "5,925", "5,976", "5,979", "5,981", "6,040", 
    "6,051", "6,070", "6,085", "6,135", "6,158", "6,162", "6,194", 
    "6,198", "6,208", "6,212", "6,221", "6,227", "6,242", "6,250", 
    "6,260", "6,261", "6,318", "6,341", "6,342", "6,380", "6,385", 
    "6,396", "6,438", "6,444", "6,459", "6,466", "6,469", "6,489", 
    "6,506", "6,509", "6,510", "6,520", "6,531", "6,570", "6,583", 
    "6,616", "6,735", "6,781", "6,803", "6,820", "6,834", "6,858", 
    "6,933", "7,291"), class = "factor")), .Names = c("YEAR", "MONTH", 
    "BIRTH_TOTAL"), row.names = c(NA, 10L), class = "data.frame")

EDIT 2: I have completed the suggestions below. Each data frame has been further refined (eliminated spaces, converting to numeric, adding DATE variables, etc.). I'm now attempting to use the ccf() function to find a correlation between extreme winter weather events, and (possible) spikes in births nine months later.

I have included below my stripped down basic code. Right now, I've written two attempts at ccf(): the first one uses the newly created birth.data$DATE, and compares to combined.weather$DATE; the second is the same, except that it adds a second variable, combined.weather$EVENT_TYPE.

So I've made some basic progress. I just figured out to create a different weather data frame, using rbind again, with just the years I want:

 combined.weather.birth <- rbind(winter2007,winter2008,winter2009,winter2010,winter2011)

My other concern - and this is basic programming but I'm not sure how to do this - is how to filter out the less extreme winter weather events, so that they don't unduly influence my correlation model i.e. a simple frost/freeze warning is probably insufficient to keep people indoors in the winter, and thus lead to a spike in births. Right now frost/freeze and hail are weighted the same as winter storm, and I need to get address that. Any tips?

NOTE: please note, I've gotten rid of combined.weather1, and am now working exclusively with combined.weather and combined.weather.birth, to keep things simple.

Here is some of the new code:

birth.data$DATE = ymd(paste0(birth.data$YEAR, '-', birth.data$MONTH, "-", 01)) #adds DATE variable
combined.weather$DATE = ymd(paste0(combined.weather$YEAR, '-', combined.weather$MONTH, "-", 01)) #adds DATE variable
#Create special weather comparison for birth records:
combined.weather.birth <- rbind(winter2007,winter2008,winter2009,winter2010,winter2011)
combined.weather.birth$DATE = ymd(paste0(combined.weather.birth$YEAR, '-', combined.weather.birth$MONTH, "-", 01)) #adds DATE variable
correlation1 <- ccf(birth.data$DATE, combined.weather.birth$DATE, lag.max = NULL, type="correlation", plot=TRUE)
correlation2 <- ccf(birth.data$DATE, combined.weather.birth$DATE+combined.weather$EVENT_TYPE, lag.max = NULL, type="correlation", plot=TRUE)
$\endgroup$

migrated from stackoverflow.com Apr 24 '14 at 16:19

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Can you provide us your sample data? and how you want to compare both of these data sets? any statistics?? $\endgroup$ – ramesh Apr 24 '14 at 3:10
  • $\begingroup$ Hi Ramesh. Is there a way to upload the individual data files here? They're .csv file types. I want to filter out certain weather types (like "hail"), and total up the major weather events (like "winter storm"), and find a way to see if there exists a correlation between spikes in extreme weather with baby births nine months later. I'm new to stats and R-Studio, and not sure how to best proceed. My original plan was to use associative rules mining, but I don't think that will work here. $\endgroup$ – user3491754 Apr 24 '14 at 3:26
  • 1
    $\begingroup$ It would probably be better if you post maybe 10 or 20 rows from each of the weather and birth data frames. Use the dput function to put it in a form that's easy to copy and paste into an R script. For example, to provide the first 10 rows of your weather data, run dput(combined.weather1[1:10,]) and paste the output into your question. $\endgroup$ – eipi10 Apr 24 '14 at 3:53
  • 2
    $\begingroup$ Maybe the cross-correlation function (correlation between two time series at various lags) would give you what you want. The function is ccf(). Run ?ccf for details on how it works. $\endgroup$ – eipi10 Apr 24 '14 at 4:03
  • 1
    $\begingroup$ For filtering, maybe use can use gerp function to find those words and subset those rows. $\endgroup$ – ramesh Apr 24 '14 at 6:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.