# Steps in the analysis of a survey data which measures the knowledge about a subject of a population

I have a survey to measure the level of knowledge of a population X on a certain topic.

The survey consists of N categorical questions of several answer options with a single valid option (the success of each question has the same weight).

What are the steps to follow to perform an analysis of the data obtained from the survey and be able to generalize the good or bad knowledge that the X population has on the subject?

The ability to generalise validly from a sample to a population depends on the sampling method that was used, and the sample frame that the sample was selected from. In surveying work we have a population of interest, for whom we want to make inferences. We choose a corresponding sampling frame that is as close as feasible to the population of interest, while containing information that is sufficient to identify and contact each of the people in the frame. We then select a sample from this sampling frame using an appropriate randomisation procedure (e.g., simple-random-sampling, block randomisation, etc.).

In order to assess the reliability of using a sample of values to estimate the characteristics of a population, it is prudent to assess the possible sources of error in statistical inference problems of this kind. By "error" we refer here to any disparity between the reported estimates in the data and the inferential object of interest.$$^\dagger$$ The sources of error in a statistical problem of this kind are illustrated in the diagram below. This shows the process of information flowing from an initial population down to the outputs of the survey, which in this case are estimates of quantities pertaining to the population of interest. The goal of the analysis is to estimate the information in the blue box by giving some outputs in the yellow box. Each part of transition between these two nodes involves the possibility of error, in the sense of imposing some effect that causes a disparity between the estimates and the actual state of the population of interest. If we use an appropriate randomised sampling method, we can use statistical methods to make inferences about the size of the "sampling error" in this process. This allows us to make an inference about quantities in the sampling frame, and then if the sampling frame is close to the population of interest, we generally take this as an inference about the population of interest. Observe from the diagram that even when we use statistical methods to estimate the likely sampling error, there are still many sources of potential error that we cannot measure, including sampling-frame error, non-response error, measurement error, processing error, and model error. You also have to consider whether you wish to generalise the particular questions in your survey to make inferences about some wider conceptual conclusion that is illustrated by those particular questions.

Application to your problem: You have not specified your population, sampling frame, or sampling method, in your question. These are things that you will need to consider in order to determine whether you have validly generalise from the data in your sample to the (unknown) characteristics of the population of interest. In your survey you have asked particular questions on a topic, and you wish to use these questions to generalise to a measure of knowledge on that topic, so you will also need to consider whether scoring these particular questions provides a reasonable measure of subject knowledge.

You question is framed in a way that suggests that you have already conducted your survey, and you are now at the point where you want to analyse the data. You have specified that you will measure knowledge of the subject by a set of questions with the same weight, which implies that knowledge will be measured by the total score of correct answers to the $$N$$ questions. If your sample was a randomised sample from a sampling frame then you can use statistical methods to make an inference about the mean number of correct answers across the sampling frame. If this sampling frame is equivalent to, or close to, your population of interest, then this will give you a reasonable generalised inference about your population. The accuracy of your inference will depend on the size of your sample (higher sample size gives lower expected sampling error), but you should also consider other sources of error, such as model error, processing error, etc.

$$^\dagger$$ It is important to note that some sources of error in sample surveys are unavoidable, so when we refer to "error" we are not referring to any shortcoming of the survey analysis, nor any mistake by the surveyors. We are referring only to a disparity between the desired information of interest and the reported outputs in the data. Since these are usually expressed as estimates subject to uncertainty there is usually already an acknowledgement that these outputs are imperfect.

• When you say "If your sample was a randomised sample from a sampling frame then you can use statistical methods to make an inference about the mean number of correct answers across the sampling frame" what kind of statistical methods do you mean? And when you comment "If this sampling frame is equivalent to, or close to, your population of interest..." how do you know if its accurate? – Juan Apr 15 at 17:19
• Statistical methods for inference from surveys is a broad field, so I cannot summarise it here. If you are estimating a population mean then a standard method would be to use a confidence interval using the T-statistic, with finite-population correction. – Ben Apr 16 at 5:26
• As to the closeness of the population to the sampling frame, if these are divergent, that is usually because you cannot identify the exact composition of the population, and so you do not know exactly how accurately the sampling frame approximates the population (e.g., approximating all adults living in a region by people on electoral roll). Again, this is an aspect of sampling theory that is a broad field - if you are not familiar with the use of sampling frames then you will need to read some material on this to understand it fully. – Ben Apr 16 at 5:29