Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am trying to understand if newspaper ads have an effect on the number of visitors to a museum. There are two main newspaper the museum advertises on. The ads are in different size. What I've done is I have put in a column the visitors count for that month and then created a series of dummy variable, one for each ad size in each of the newspapers, with 1 if the ad was present during that month and 0 if it was not. Then I ran the regression.
Can I trust the outcome?
Thanks a lot!
As Peter has said, there are a great many things beyond just what you've written that determine whether or not you can "trust" the outcome - that the type of regression you're using is appropriate, that the relationship between the ad size and visitors is correctly specified, and that you haven't missed any variables (for example, what if its not the size of the ad but that small ads are placed in the back, and large ads near the front?).
But assuming everything else is done right, your dummy indicators appear to have been set up correctly. For more complicated setups, you may indeed want to look at automated versions of this - there is usually a way to tell your statistical program that a particular variable is categorical, and it will take the appropriate steps, but how precisely to do this depends on the package you're using.
I agree that there is really not enough information in the question to give a definitive answer, but there is enough information to point to a few areas of concern.
The first thing you should do is try to understand (by talking to the relevant people, ideally) how decisions are made about when, where, and how big to run the ads. This will help you hunt down and kill some sources of spurious correlation. For example, maybe the museum runs lots of big ads in the winter because people like to go to the museum when it is cold and raining outside, like during the winter. This would make it look like big ads are very effective since lots of visitors show up when the big ads are running. But maybe they are showing up only because it is cold and raining and the big ads do nothing. So, if this is the case, you would want to control for month-of-year or maybe even weather outside in your regression. Advertising campaigns are often part of a larger marketing effort or keyed somehow to other business decisions. So, maybe the museum runs big ads when it is having a sale on the admission price or maybe it runs big ads when it has a special, new display. Each of these would also cause you to overstate the effect of ads if you don't find a way to control for them. It is easy to think of lots of stories like this.
This brings up a second point. Since it is easy to make up such stories, you need to figure out which of these stories is likely to be important and which not. You can't control for everything, but you try to control for (or at least think about the effect of) every important thing. You need to understand how the museum business operates. What are the busy times? What are the slow times? Why are some times busy and some times slow? What is the museum trying to accomplish with its advertising? Etc. Sadly, statistics is not a substitute for regular, old scholarship and expertise. You have to understand the application area.
Another thing I might worry about here are lags. Does advertising this month only affect visitors this month? Or is there some lasting effect? Investigate this by putting in lags of advertising in your regression. Or, perhaps, by talking to the marketing people at the museum. Or by reading the relevant scholarly literature in marketing.
Another thing you might worry about a bit is multicollinearity. Does the museum advertise in "bursts," so that there are lots of big ads in all the papers one month and very little in other months? If so, you will have trouble telling the difference between the effects of advertising in one paper and advertising in another paper. You will get big standard errors for your estimates. And you should definitely not "trust" accepting a null hypothesis as evidence that the null hypothesis is true in a case like this.
Strictly speaking, no.
Museum visitor group in a city is usually quite small; nowadays "fluid populations" like tourists mostly rely on the Internet for deals and events, their chance of reading a local newspaper is close to none. This means your "targets" are not equally comparable at the beginning of every month when you try a new ad combo. Each month, you're actually starting with a more hard-to-reach group, or less-interested-in-museum group. Putting them into the same regression is not going to cause any mayhem, just that you'd need to interpret them very carefully, particularly with the sequence of your ad combos.
For this reason, I'd suggest why not stratify by month? Implement some tricks such as i) bring this ad to the admission for a gift/discount, ii) text this code or mention this code during online ticket purchase (which can be different on each type of ad, and on each wave so that you can account for lag effect) to get a discount, etc. After each month, count the ads or ask for the online purchase record from your IT person for analysis. That way at least you can say something about the effect between different ads, month by month. And you can be more certain what brought them to your museum.
I'd also recommend you to communicate with the publishers that their circulation has not changed in the test period, as it's not controlled in the model. A small ad may suddenly outdo a prominent ad just because it has increased some distribution points. You should also keep a sample of their publications to make sure it's not confounded by other major promotion campaigns which may have boosted the readership temporarily.
Same goes for your museum. Make sure there isn't any rotation in the theme exhibits. And beware of the combos happening across major holidays and summer vacations, etc.
Finally, any advertisement department of any reasonably big newspaper publisher would have conducted customer profile survey and compiled penetration statistics. If you buy ad from them, ask for that information. That can help you narrow down your selection to some more demographically aligned publications. And document all the costs as well.
I think my concluding statement is that it's a very interesting question, but regression may not be the right tool for this purpose, at least at this stage.
