Determining optimum sample size and statistical tests

Question

I'ma stats newbie and do not know how to go about this. So I will lay it out as best as i can in the hope that some one here can help me out !

I work for a IT Infrastructure (Datacenter services) company XYZ. This company has sets up wintel, Unix boxes, Databases and Networks for it's clients as part of its services offering.

I am trying to determine whether we can do some statistical test using sampling to test the quality of the the work performed by our engineers(Hardware setup plus software installation+any software programs written as part of all this work).

So let's say we provide 5 different services (wintel, Unix, DB, Networks, Middleware). All these have two kinds of projects. Standardized projects(like a shrink-wrapped product) and Non-standard/custom projects. Please let me know if you need any more details.

If I can use statistical tools for testing, what kind of tests would they be ? what kind of sampling mechanism can I use to determine an optimum sample size ?

The idea is to use these tests, get results, look at root causes and then maybe run some six sigma type improvement programs for the deficiencies.

Thanks in advance to your replies

Edit (in response to comments)

Broadly we would like to measure "Quality of Performed Work"

We are thinking of two types of quality indicators.

1) One would be done at the end of the project - customer survey. 10 questions; responses(5-1 points for Excellent, very good et al)

2) One quality indicator to check during progress of project. for a 3 month project we could have a safety checklist of 4 items, one safety check per every 3 weeks.

The idea is to compare is "in-process" project safety check scores with the customer survey scores to get an integrated view of the course of the project.

This would help identify if and where a problem has occurred, customer perception, PM communication, engineer execution, process stability, insufficient safety checks etc.

Since this entire process is in conceptual stage, there is only limited information about it with me.

I cannot characterize it's shape right now. I am assuming that Let me see if I can get some information on that.

Edit (migrated from a reply that functioned as a comment)

Michelle,

The Org. i work for is a Matrix organization with the a director level person serving as the Project Sponsor. We are also heavily ITIL based organization where everything is a Service.

The Engineers have their Direct reporting Managers and also have dotted line reporting to the Project managers. In this situation where all org.s have their priorities, it will be very hard to tie at Engineer level , their performance goals to some Quality measures.

But the VP has defined some Goals for the org, part of which is that everything should be measurable and Imp. measures should be on a Dashboard. So compliance to this will be measured for each group under the org.

I would like some more advice on the kind of qualitative measures that we can use in this approach (as stated in my last two posts). This is to measure Project Delivery by the org.

Thanks to all who contribute.

Rolando2- Thank you for your contribution !

Answer 1

This should be a comment under rolando2's but I don't have enough rep to comment so please excuse this as an answer. The quality dimensions you want to assess don't seem to be complete, for example you mention that sometimes the engineers write code, but then you don't seem to include that as a quality indicator. I suggest the following steps:

1. If you don't have a senior management champion/sponsor, get one now.
2. Look at the job expectations of the engineers, in particular the ones that they are assessed on for performance. Make a list of them, and look at what measures of activities you might introduce to assess them. Make the link between each measure and successful performance explicit, as these are critical assumptions.
3. Take the list, measures, and logic links to the engineer supervisor(s) and see what they think. Are the assumptions accurate? Are the measures fair? Are there a number of alternative measures for some activities? Are some activities performed so infrequently they aren't of practical use for assessment? Are some measures really subjective and may be biased if used? Once you have a list of good measures, ask them to weight the activities so that you can see the relative importance of them in measuring engineer performance. I don't think you want to assume that all engineer activities are equally weighted.
4. Look at how often each activity occurs. The sampling scheme will be based on that information.

Eventually - get management sign-off, using your sponsor/champion.

Answer 2

The short answer is to read a good book or take a class on Statistical Process Control.

The long answer is the class and/or book.

A good class/book on this subject will help you define meaningful measures of quality and show how to monitor these measures over time to distinguish between natural variation in the processes (that can only be changed by changing the process itself) and special cause variation that should be addressed and fixed.