1
$\begingroup$

I have historical data of a problem that can be described as:

A person represented by features X has to wait T minutes on a queue so she can receive a treatment (which is equal for everyone), and by the end of the treatment (takes a few minutes), we observe the binary outcome of Y.

I know for a fact that features contained in X, specially seasonality features (like when she entered the queue), directly influence T, at the same time that X and T also have a direct impact on Y. I also know that X might not contain all the features that influence T, and there's no way to be sure.

I also know the "shape" of the effect of T on Y: the probability of Y=1 should decrease as T increases, and saturates at a certain point, like a logistic curve.

Now the issues: I need a robust, optimized and validated model that estimates the CATE / ITE, but how can I run hyperparameter search, feature selection or any kind of model selection if I don't know the true effect of T on Y, since I only observe one T per data point, and I also don't know if the selection bias is mitigated? How can I know if my model actually works, comparable to running cross-validation for regular classification/regression problems? And what are the most optimal techniques to be used in this scenario?

Stuff that I tried already:

I tried making T binary by simply splitting the dataset based on a quantile threshold, and then use meta-learners from the causalml package, and then validate it using metrics like the Area Under the Uplift Curve and Qini Curve, but it doesn't seem ideal and I don't see how these metrics account for selection bias.

I also tried fitting some learners from econml package, but most of them are very slow and I couldn't get useful output. And I also used scikit-learn's PolynomialFeatures to add non-linearity and account for heterogeneity (force the interaction of T with X), and then train a simple logistic regression, which yields a seemingly reasonable result (the shape of the curve generated when varying T is according to my expectations), but it outputs more than 25k features, which makes the training process very slow, especially if using Lasso.

PS: This dataset has > 100 features and > 3M rows.

Improve this question
$\endgroup$
2
  • $\begingroup$ Do you also have an attrition problem? That is, does everyone make it to the end of the waiting period? Can people arrive or return when things are less congested? At what point in the process do people learn their T? $\endgroup$
    – dimitriy
    Commented May 29, 2020 at 21:28
  • $\begingroup$ Framing the problem in a chat context, so being treated means being answered: Not everyone makes it to the end of the period, but they are all answered, so they are all logged as "treated". For those cases, we will always observe Y=0 and usually a high waiting time. People can return, but they will be considered another data point, though there's a feature that indicates how many chats he opened recently, and we only observe Y=1 once in a large timeframe (> 7 days) for the same person. People learn their T once they get answered, regardless if they answer back. $\endgroup$
    – loglossbb
    Commented May 29, 2020 at 22:20

1 Answer 1

Reset to default
1
$\begingroup$

This is a mediation problem. You have a causal diagram as follows:

enter image description here

Here $U$ represents unmeasured influences on $T.$

There is no backdoor path from $X$ to $Y,$ so you do not need to use anything like instrumental variables, the backdoor adjustment formula, or the front-door adjustment formula. This is a pure mediation problem with no confounders.

So, if you want the conditional average treatment effect (CATE) of $X$ on $Y,$ there's nothing stopping you from performing the usual calculation.

Your question, though, seems to be more concerned about not knowing what constitutes $U,$ the unmeasured variables. If you examine the causal effect of $T$ on $Y,$ you see that $T\leftarrow X\to Y$ is a backdoor path from $T$ to $Y.$ So you could use the backdoor adjustment formula, $$P(Y=y|\operatorname{do}(T=t))=\sum_x P(Y=y|T=t,X=x)\,P(X=x),$$ to get the true causal effect of $T$ on $Y.$

Improve this answer
$\endgroup$
1
  • $\begingroup$ Using the chat example from the OP's comment on the question, is it possible the times with long T are also high demand periods, so that there is a path from U to Y, which would attenuate the effect of T? I suppose some of that is taken care of by X. But another channel for bias is that people who are serious about buying the product might ask questions that longer to answer, so some of the variation in T is not exogenous in way that may not be adequatelly controlled for by conditioniton on the date and time the question is asked. $\endgroup$
    – dimitriy
    Commented Jun 4, 2020 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.