How to analyse a risk assessment questionnaire where each risk is rated for both probability and impact? I would like your opinion on how to analyze a questionnaire on risk assessment in construction projects. For each question, concerning a specific risk for the project, there are two answers: An estimate for the probability of occurrence for the risk (on the scale from 1 to 5) and one for the estimated impact of the risk to the project (also in the range 1-5).
A common way to combine these two values is by multiplying them. Then, according to a chosen (by the organization) "Probability/Impact" matrix, I could determine which combinations of probability and impact result in a risk’s being classified as "high risk", "moderate risk", or "low risk". I could then use these qualitative responses to analyze my data. 
However, I do not want to lose the information about a risk's probability because a risk classified as "high" should be treated differently, if this arises from high probability instead of high impact. The impact cannot be changed but the probability might be reduced by taking appropriate measures. What would the form of the data matrix be, if I want to keep a qualitative and a quantitative response for each question?. What methods do you suggest for the analysis?
 A: Economists (and some project managers) usually use cost functions to analyse this kind of problem.
So, imagine that there is a cost associated with the occurrence of the bad output (lets call that $\theta = 1$) equal to $c$ (e.g., how much will you lose in lawsuit from a employee injure, how much will cost to rebuild a broken structure).
Imagine also that the probability of a bad output occur is a function of the effort to avoid that, and we can convert this effort in monetary terms, let's call it $x$ (e.g. double check the safety specifications will demand extra workers and therefore extra salaries). So this function could be like this:
$$P(\theta=1|x)= f(x)$$
So your problem is to choose x that minimize $f(x)*c+x$.
Numeric example:
Suppose that:


*

*double check cost 25 dollars;

*from historical data, with double check the probability was 0.005 and without double check the probability was 0.01;

*the cost associated with a bad output is 6,000 dollars;


Them we have:
$$\text{total cost} = f(25)*6,000 + 25 = 30 + 25 = 55$$
and
$$\text{total cost} = f(0)*6,000 + 0 = 60 + 0 = 60$$
The preferable $x$ is 25.
So, the answer for your question is:
Do not make a questionnaire to ask for the risk, use previous data. Take this previous data and check if some preventive actions influences the chance of output (you can do that with a probit model). Get the total cost associated with the preventive action and put side-by-side with the probability estimated with the probit model (you will get a table with $f(x_i)$ for each $x_i$). Use previous data to determine the average cost of a bad output (the $c$). Multiply every $f(x_i)$ by $c$. Choose $x_i$ that result in the smaller $f(x_i)*c+x_i$.
If, and only if, you do not have previous data, make a questionnaire asking for the probability of the risk for each set of safety measures being used and when no safety measures are being used. Then ask for the cost of the safety measures. If you also do not have historical data about the cost of the bad outputs, ask this too. Then apply the formula of the total cost expected to find the optimum amounts of money that should be putting toward prevention.
A: If I understand correctly your issue, it works like sensitivity and specificity. Look at sensitivity, specificity analysis, there are many things to read on this. The concept is that with these two measures you construct ROC curves and then you deermine an acceptable threshold. 
