3
$\begingroup$

I have a dataset of online Reddit posts for unemployed individuals in two neighboring U.S. states, where unemployed workers in a given state received higher unemployment insurance UI benefits than the other (i.e. my treatment variable), and I am trying to test if receiving higher UI benefits causally correlated with more positive sentiment.

Specifically, I track online sentiment for both groups for around 7 months before the treatment and around 9 months post-treatment. However, one challenge I am facing is that the 'treated' unemployed workers become less active on reddit once their state passes the higher UI benefits (i.e. similar to attrition bias in surveys where individuals from a treatment/control group drop out of the survey). I have read about techniques explained in the blog article above on how to address and reduce attrition bias with survey data, but I wonder if applying the same techniques makes sense also with administrative or observational online text data?

Here is a sample of the data's structure: Each row represents an online post written by an individual. The first variable refers to username, followed by the date of a given post, the predicted ML sentiment/mood for that post. The column treatment_time refers is a binary variable telling us whether the post was written before or after treatment, while the variable treatment_status tells us whether the observation received treatment or is in the control group (i.e. the neighboring state where UI benefits were not extended during Covid).

For instance, I have observations such as Kenny and Cartman from the treatment group who only remain active for a few days post-treatment with the last posts they write trending more positive, relative to pre-treatment. However, I am not sure if I can interpret their attrition (i.e. leaving the data) as a proxy evidence that the policy was effective in that individuals no longer felt the need to describe their job situation on Reddit?

On the other hand, individuals like Mr. Garrison from the control group remain active on Reddit post-treatment and continue posting statements that are classified as negative or neutral.

# Treatment (i.e. extended UI benefits) occurs on Sept 6, 2020.
username        date         mood         treatment_time    treatment_status 
Kenny       2020-09-02.       negative       pre               treatment 
Kenny      2020-09-03.        negative       pre               treatment
Kenny      2020-09-07.        positive       pre               treatment
Cartman   2020-09-03.       negative       pre               treatment 
Cartman  2020-09-06.        negative       pre               treatment
Cartman  2020-09-08.        positive       post              treatment
Mackey   2020-09-03.       negative        pre               control 
Mackey   2020-09-04.       negative        pre               control
Mackey  2020-09-08.        negative        post              control
Mackey  2020-09-13.         neutral         post             control
Mackey  2020-09-14.         neutral         post             control
Mackey  2020-09-23.         neutral         post             control
Garrison   2020-09-03.      negative        pre              treatment   
Garrison  2020-09-04.       negative        pre              treatment
Garrison  2020-09-04.       neutral         pre              treatment
Garrison  2020-09-05.       negative         pre             treatment
Garrison  2020-09-14.       neutral         post             treatment
Garrison  2020-09-19.       negative        post             treatment
Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Attrition is loss-to-follow-up. There is a careful distinction between that and sparse data collection. A standard mixed model would account for clusters that "thin out" in later times - but the important bit is that employment seems to be a time-varying covariate in your analysis. It might help to clarify the design and data structure a bit more as well as define precisely what you take attrition to mean here. $\endgroup$
    – AdamO
    Nov 14, 2022 at 19:37
  • $\begingroup$ Thanks Adam! See my updated post. $\endgroup$
    – nesta1992
    Nov 15, 2022 at 10:56
  • $\begingroup$ You care about a causal estimand and you are using observational data. It'd be a much better start for you and for people answering your problem if you'd show what is your assumed causal model (as in a DAG) of your problem. You will not be able to do proper inference without stating these. $\endgroup$
    – Kuku
    Nov 17, 2022 at 10:09

2 Answers 2

Reset to default
2
$\begingroup$

A key issue in the analysis of longitudinal data with attrition is whether you can make the missing at random (MAR) assumption. [Aside: this is related to the independent censoring assumption in time-to-event analysis.] If you can't make that assumption there are not many available tools for analyzing the data unless you are in the once-in-a-lifetime situation of knowing how to model the dropout process. To be able to assume MAR requires a certain frequency of data collection so that there are few long gaps. That is because MAR means that conditional on baseline variables and on previously measured longitudinal responses, missingness of the response at the current follow-up time is completely random. In other words, we need the only things that predict missingness at time $t$ to be baseline variables and responses between time zero and time $t$. Thus previous responses are assumed to provided a "preview" of missingness/dropout. Under these assumptions longitudinal models may be used as long as they are full likelihood models (i.e., generalized least squares, Markov models, random effects models, not GEE).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, this is helpful! However, I believe the post-treatment missingness in our data is not random, and it's more similar to qualitative challenges in Dev economics RCTs where treated individuals in a developing country become less responsive to surveys once they receive a cash benefit or when individuals in the control group stop responding to the survey because they are frustrated for not receiving a given cash/in-kind benefits. $\endgroup$
    – nesta1992
    Nov 30, 2022 at 14:48
1
$\begingroup$

I can't point to relevant literature. But I have a suggestion. How about using outcomes positive/drops out/negative? Then, secondarily, if you can predict dropping out as an outcome, also report the average sentiment in last post.

It seems to me that this approach could result in a useful and convincing story from the data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.