3
$\begingroup$

I am using data from sub-reddits like [this][1] or [this][2], where users discuss their thoughts on the Federal government unemployment insurance and its fairness. Specifically, I wonder if it makes sense to use the policy announcement as the treatment variable with a measure such as online sentiment as my outcome. For example, if I first 1-divide all users to either a control or treatment group based on whether they state they have qualified and received the Federal UI benefits during Covid-19, then 2-estimate the following sentiment scores for all posts (from 0 to 10, with 10 being the highest or positive sentiment) in the sub-reddit.

         before      after  
group   sentiment   sentiment   difference
treatment      4    8              4
control        4    3              -1

Does the research design above allow for making causal statements, i.e., is it possible to state that individuals who qualified for the UI benefits had higher sentiment on average, relative to those who didn't on Reddit, and that this is due to them receiving the treatment (i.e. qualifying and receiving UI benefits)?

Or can we only interpret the results descriptively, and state that higher UI benefits are correlated with higher positive sentiment for those receiving them?

Here is my response to the good points mentioned by dipektov:

I think the challenges you mention apply to most data sources, including surveys by the Bureau of Labor Statistics (BLS), i.e. individuals might state that they received the UI benefits or that they are actively looking for a job when they aren't, and we can't really verify their statements, here in the U.S. at least, because no one is comparing survey-based data to admin records of individuals receiving Federal UI benefits.

Regarding bias, I would argue that it also occurs with conventional surveys, for instance, jobless individuals that pick up the phone and respond to BLS surveys on their job status, income, etc are a self-selected sample of the unemployed population in the U.S. In fact, there is extensive literature arguing that many men in the U.S. who are either unemployed or out of the labor force are missing from most public statistics and we have limited information as to why they are jobless or not looking for a job.

In conclusion, I would argue that many of the issues that exist with data sources such as Reddit, are also prevalent in traditional data sources like surveys, yet we as social scientists still heavily rely on the latter to make causal statements about social or economic trends like unemployment.

Update: Valid points were raised below about the "nonrandomness" of our treatment variable, receiving Federal UI benefits. However, I believe that the qualification for the benefits had a luck and randomness component. For instance, not to over rely on anecdotes, but I know several people like my brother with a HS degree pre-pandemic and who worked in places like movie theaters and were thus seen as "nonessnetial" employees and qualified for the + Federal benefits. Meanwhile, our neighbor's son also had a HS degree and earned similar income pre-pandemic to my brother, but did not qualify because he worked in Trader Joe's and was classified as an "essential" worker. As a result, despite having similar educational attainment and income before Covid, some individuals were lucky enough to be in jobs that were seen as nonessential and received generous benefits in the pandemic, while others were seen as essential and thus couldn't qualify for the benefits. [1]: https://www.reddit.com/r/unpopularopinion/comments/g043rt/unemployed_people_should_not_be_making_more_than/ [2]: https://www.reddit.com/r/dataisbeautiful/comments/glwtxv/many_americans_are_getting_more_money_from/

$\endgroup$
10
  • 1
    $\begingroup$ This data is observational and there are many questions about bias. For example, what population do reddit users who self-select to write in these two sub-reddits represent? You don't even know that they have received the benefits but that they have said on reddit to have received the benefits. (And on reddit there may be a difference.) You can describe the results but meaningful interpretation will be challenging. $\endgroup$
    – dipetkov
    Jul 29 at 8:00
  • 1
    $\begingroup$ I'm unclear on what made you think of causal inference in this purely observational uncontrolled context. There are strong nonrandom qualifications to receive the resources. $\endgroup$ Jul 29 at 11:13
  • 1
    $\begingroup$ Let me make an analogy to make sure I understand your argument. Consider election polling. Your argument is equivalent to saying: There are issues with polling (and there are); therefore polls are as accurate as checking what reddit users share about their voting intentions. I find this argument unconvincing. There are methods to correct polls for bias (even these methods are not perfect). I suspect that the BLS does so as well with their data. Do you know how to correct for bias on reddit? $\endgroup$
    – dipetkov
    Jul 29 at 12:26
  • 2
    $\begingroup$ If you are interested in the population of people who participate on reddit, that's fine. But you won't be able to argue convincingly that your analysis applies to the US population as a whole. Why would we expect that the reddit data is more or even as representative as surveys? One needs access to the internet to participate. So you've substituted the assumption that people will pick up their phone and participate in a survey with the assumption that they'll go online and participate in reddit. (I won't even bring up what we are learning about who participates in social media platforms.) $\endgroup$
    – dipetkov
    Jul 29 at 12:27
  • 1
    $\begingroup$ By now you've realized that I have serious concerns about the trustworthiness of what's said on social media, so I won't belabor the point. Instead I'll elaborate on Prof. Harrell's comment. By "strong nonrandom qualifications" he means that the population of those who received the benefit and those who didn't are different, by design, before the policy announcement: the criteria for who qualified are not random. $\endgroup$
    – dipetkov
    Jul 30 at 21:42

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.