It sounds like your design dramatically limits the possible missing data methods you may apply.
If the only data available to you are the averaged responses and the number of responses for each question and group. What you do not know is: exactly how many responses were omitted for each item, what the variability of the mean response was for each item, or any individual level information whatsoever.
In light of this, the best solution is the use a complete case analysis with observations at the subject by item-response level. This is the analysis you would have done had you had complete data. I assume some form of frequency weighted linear regression is what you intend to do. The imbalance in counts will only affect the precision of analyses. You can make a strong case that non-response is more a feature of the questionnaire design than it is of the data analysis. If the tool garners missing data and does not advise how to analyze those data, then any approach developed in this sample would be dependent on the features of that sample and therefore have little generalizability.
Recount the sensitivities to non-response on survey items: recall, invasive, fatigue, lack of clarity in item wording, and so on. In many cases, non-response is very much a subtle response: "Jeez, I can't remember", "How dare you ask me that question", "I already told you that", "Heck if I know". Data analysis can tell you very little of how good a survey is. So it bothers me very little that missing data are a feature of your data analysis, we are already at the point that we must accept the questionnaire, including it's tendency to render non-response among some respondents.