4
$\begingroup$

I conducted a survey research where I have used two delivery modes for the questionnaire - online and offline (hardcopy mode).

The survey questions measure perceptions of employees on certain organisational behavior variables such as - organisational commitment, organisational justice, perceived organisational support, job satisfaction etc. All items are measured with a likert scale (1-7)

The research objective is to test for moderation hypotheses (a total of 15). I observe that different set of variables are emerging as moderators when I run moderation test separately for online and offline(hardcopy) data.

How should I proceed? Should I just ignore these differences and focus on overall data or report data according to mode of delivery?

$\endgroup$
3
  • $\begingroup$ sampling procedure - HR department of companies were contacted for approval of the study. Once approved based on how HR wanted to proceed, online or hardcopy method was selected. For online, HR would send the questionnaire (hosted on questionpro) to their mail group by email and employees responded to the survey. The responses reached directly to me. For hardcopy - It was for the group that had no access to computers and were also not able to read English. Questionnaire was translated and distributed by me among employees (at the company location). Convenient sampling was used. $\endgroup$
    – Rahul
    Aug 14, 2012 at 11:22
  • $\begingroup$ So, each company was either online only or paper only? And they tended to select one or the other based on computer access and english ability? $\endgroup$
    – Jonathan
    Aug 14, 2012 at 22:32
  • $\begingroup$ No it was a mix (for most of the companies). They selected online for employees having computer access and paper for others. $\endgroup$
    – Rahul
    Aug 16, 2012 at 3:04

2 Answers 2

5
$\begingroup$

There are a couple things that come to mind about what might be happening here in general:

  1. The sampling procedure behind the different modes (online, paper) may have been different. As a simple example of how this might have come about, if all employees with a computer were given the online survey and all employees without a computer were given the paper survey, it's easy to see why there would be large differences.
  2. Attitudinal questions are more sensitive to differences in survey administration than factual questions, because they don't represent a fixed or standard quantity - our moods are changeable minute to minute. Seemingly small things can have an impact on how participants respond, for example, the timing or environment. Surveys taken immediately after lunch, when the brain just got a nice rush of glucose, will probably reflect higher overall satisfaction. Surveys taken in group settings vs. individual settings may have different responses as social conformity may come into play.

On how to proceed....

Based on your description, it sounds like both effects are at play. Overall, my suggestion would be to take one action to correct what might be a major sampling effect, and then roll together the paper and online data for analysis.

Sampling effects for your situation: The online and paper surveys reached very different parts of the working population - those who use computers are typically going to be office workers and will be paid differently, be evaluated by their managers differently, work on different tasks, etc. Their responses to an employee survey would be expected to be very different. In addition to the survey targeting very different groups, the method by which you sampled the employees was different. 100% of computer-using employees received an invitation, vs. an unknown convenience sample of non-computer users. Their response rates may have been different.

Addressing sampling effects: The fact that online and paper populations are different types of employees and therefore respond differently is not a concern. They are different in reality and the survey data reflects that. If understanding these differences is valuable, it would be better to use a variable like "job category" or something if you collected that, but if not, you could keep the online/paper variable for analysis as a rough way to divide office and non-office workers.

The fact that different proportions of the relevant populations may have been reached is a concern - if all office workers responded, but only half non-office workers responded, then your office workers are over-represented. If possible I would try to weight the paper and online , by company, to the proportion of the relevant populations in each company. The fact that convenience sampling was used for paper surveys is unfortunate, as it may not represent the population, but I can't think of anything to do now.

Survey administration effects in your situation: There is likely some effect, but these will be hard to define and correct for. Online surveys were probably taken at the convenience of respondents at their desk, while it sounds like paper surveys may have been taken in a group setting at a set time. This probably has some effect on responses but would be difficult if not impossible to adequately define and correct for.

$\endgroup$
1
  • $\begingroup$ Thank you Jonathan for your response. I have added additional details. $\endgroup$
    – Rahul
    Aug 14, 2012 at 11:23
4
$\begingroup$

Given that the research objectives are about moderation, I would either report the two sets of results separately or add another term to the moderations for interaction.

Who was given the survey? The employees' supervisors? Co-workers? or who?

How was the hard copy survey given out? In person? By mail?

$\endgroup$
5
  • $\begingroup$ Thanks Peter. Survey was given to both managerial and non-managerial employees. Hardcopy was distributed and collected personally be me. $\endgroup$
    – Rahul
    Aug 14, 2012 at 11:42
  • $\begingroup$ That points to a couple things that might drive differences - degree of perceived anonymity, pressure to respond, difficulty of responding. $\endgroup$
    – Peter Flom
    Aug 14, 2012 at 11:46
  • $\begingroup$ It was adequately stated both in online/hard copy that their employing organization will have no access to their responses. Also, they were not required to write their names etc./ Pressure to respond could be one possible explanation but I can't seem to find any study/article in this area. Difficulty in responding is related to cognitive ability and it is difficult to raise questions on cognitive ability just because they are answering a hard copy of the questionnaire. $\endgroup$
    – Rahul
    Aug 14, 2012 at 11:57
  • $\begingroup$ By 'difficulty in responding' I meant the effort it took to respond. When it takes more effort to respond, you are more likely to get extreme values. There's a lot of literature on mode of survey administration, but it's been a while since I looked at it and when I did it wasn't about this kind of survey $\endgroup$
    – Peter Flom
    Aug 14, 2012 at 12:11
  • $\begingroup$ Thank you very much for your help. I will search for the literature. $\endgroup$
    – Rahul
    Aug 14, 2012 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.