I am using data from sub-reddits like [this] or [this], where users discuss their thoughts on the Federal government unemployment insurance and its fairness. Specifically, I wonder if it makes sense to use the policy announcement as the treatment variable with a measure such as online sentiment as my outcome. For example, if I first 1-divide all users to either a control or treatment group based on whether they state they have qualified and received the Federal UI benefits during Covid-19, then 2-estimate the following sentiment scores for all posts (from 0 to 10, with 10 being the highest or positive sentiment) in the sub-reddit.
before after group sentiment sentiment difference treatment 4 8 4 control 4 3 -1
Does the research design above allow for making causal statements, i.e., is it possible to state that individuals who qualified for the UI benefits had higher sentiment on average, relative to those who didn't on Reddit, and that this is due to them receiving the treatment (i.e. qualifying and receiving UI benefits)?
Or can we only interpret the results descriptively, and state that higher UI benefits are correlated with higher positive sentiment for those receiving them?
Here is my response to the good points mentioned by dipektov:
I think the challenges you mention apply to most data sources, including surveys by the Bureau of Labor Statistics (BLS), i.e. individuals might state that they received the UI benefits or that they are actively looking for a job when they aren't, and we can't really verify their statements, here in the U.S. at least, because no one is comparing survey-based data to admin records of individuals receiving Federal UI benefits.
Regarding bias, I would argue that it also occurs with conventional surveys, for instance, jobless individuals that pick up the phone and respond to BLS surveys on their job status, income, etc are a self-selected sample of the unemployed population in the U.S. In fact, there is extensive literature arguing that many men in the U.S. who are either unemployed or out of the labor force are missing from most public statistics and we have limited information as to why they are jobless or not looking for a job.
In conclusion, I would argue that many of the issues that exist with data sources such as Reddit, are also prevalent in traditional data sources like surveys, yet we as social scientists still heavily rely on the latter to make causal statements about social or economic trends like unemployment.
Update: Valid points were raised below about the "nonrandomness" of our treatment variable, receiving Federal UI benefits. However, I believe that the qualification for the benefits had a luck and randomness component. For instance, not to over rely on anecdotes, but I know several people like my brother with a HS degree pre-pandemic and who worked in places like movie theaters and were thus seen as "nonessnetial" employees and qualified for the + Federal benefits. Meanwhile, our neighbor's son also had a HS degree and earned similar income pre-pandemic to my brother, but did not qualify because he worked in Trader Joe's and was classified as an "essential" worker. As a result, despite having similar educational attainment and income before Covid, some individuals were lucky enough to be in jobs that were seen as nonessential and received generous benefits in the pandemic, while others were seen as essential and thus couldn't qualify for the benefits. : https://www.reddit.com/r/unpopularopinion/comments/g043rt/unemployed_people_should_not_be_making_more_than/ : https://www.reddit.com/r/dataisbeautiful/comments/glwtxv/many_americans_are_getting_more_money_from/