How to compare the results of surveys for two different type of users

I am working on an online store as analyst. We are constantly running a satisfaction survey (rated -3 to 3). The response rate is very low (10%). We want to compare the results of surveys between experienced users (being with us more than a year) and the new users (being with us less than a month).

I see a couple of issues and I am not sure the best way to resolve them:

1- For the non-response bias, I am thinking of doing an inverse probability of treatment weighting (treatment being choosing to respond). Is that practical giving the very low response rate?

2- How do I control for the intrinsic difference between new and experienced users? Do I need to control behavioral/demographic variables and run a regression. Is that enough (assuming regression criteria are met)?

Any word of advice is highly appreciated.

Non-Response

For non-response, you would use something more like inverse probability of sampling weights. THIS PAPER describes the inverse probability weights you might consider and the overall process. While it is described in the context of a randomized trial and a general population, there is a clear parallel to the scenario you describe.

As for whether this is practical, it is more important to consider whether (1) you know what factors describe non-response and are associated with satisfaction rating, and (2) whether that data is available for all your users. If you believe you reasonably have those met, then you can start to consider the sample size. As your question is worded, it is hard to tell whether 10% is enough. As an extreme example 10% of 10 would not be but 10% of 100 000 000 would be.

New / Current Users

Whether you need to control for differences between the types of users depends on your question of interest. If you only want to describe overall user-satisfaction at a snapshot in time, then there is no need to control. If you are interested in the trajectories of satisfaction over time that the user has been a member of the site, this is more difficult.

Controlling for these differences is complicated. In the context of survival analysis, this problem is called left-truncation or late-entry. It is generally recommended to do a new-user design, where only new-users are analyzed (akin to throwing out the data for the current users). The other extreme is to allow for "late-entry" however, this requires a strong assumption. Particularly, you are assuming no relationship between late-entry (or being a current user after $$t$$ time) and user-satisfaction.

I would recommend clarifying the question you are interested in (to determine if you need to control for new versus current users). If you do need to control for new versus current users, I would recommend running the analyses stratified by new/current users.