- Is RL the right framework under such constraints?
It looks possible, but maybe some small detail that you have not given would make other approaches more feasible. For instance, if the notification events can be treated as more or less independent, then a supervised learning approach might be better, or at least more pragmatic.
More practically it is not 100% clear what your state, timesteps and action choices will be. These need to be well-defined for RL approaches to work. In addition, you want to be able construct states that have (or nearly have) the Markov property - essentially that anything known and non-random about expected reward and next state is covered by the state.
- How can we learn the optimal policy offline in such situations
You want both an offline (data is historical, not "live") and off-policy (data is generated by a different policy to the one you want to evaluate) learner. In addition, I am guessing that you don't know the behaviour policies that generated your data, so you cannot use importance sampling.
Probably you can use a Q-learning approach, and work through your existing data either by replaying each trajectory using Q($\lambda$) in batches, or some variant of DQN using sampled mini-batches.
This is not guaranteed to work, as off-policy learning tends to be less stable than on-policy, and may require several attempts to get hyper-parameters that will work. You will need a good number of samples that cover optimal or near optimal choices on each step (not necessarily in the same episodes), because Q-learning relies on bootstrapping - essentially copying value estimates from action choices backwards to earlier timesteps so as to influence which earlier states the agent prefers to take actions to head towards.
If your state/action space is small enough (when you fully enumerate the states and actions), you may prefer to use the tabular form of Q-learning as that has some guarantees of convergence. However, for most practical problems this is not really possible, so you will want to look at options for using approximation functions.
... and how do we evaluate the same?
If you can get realistic-looking converged action-values from your Q-learning (by inspection), then there are only 2 reasonable ways to assess performance:
By running the agent in a simulation (and maybe further refining it there) - I don't expect this is feasible for your scenario, because your environment includes decisions made by your customers. However, this is a good stepping-stone for some scenarios, for instance if the environment is dominated by basic real-world physics.
By running the agent for real, maybe on some subset of the workload, and comparing actual rewards to predicted ones over enough time to establish statistical confidence.
You could also dry-run the agent alongside an existing operator, and get feedback on whether its suggestions for actions (and predictions of reward) seem realistic. That will be subjective feedback, and hard to assess performance numerically when the actions may or may not be used. However, it would give you a little bit of QA.