3
$\begingroup$

The greatest gain of the statistics classes in both school and university seems to be that I now have an inkling of which QA site to use for this question. :)

I'm a programmer and I'm making a questionnaire with a financial background. There will be many (probably over 100) questions, which will be presented to the user one-by-one, like in a wizard. Since it answering them all will be quite time consuming (likely over an hour), it would be best if the user interface could show the estimated time remaining - both for the current question and all remaining together.

As there is no way of determining beforehand how long a question will take, my idea was to gather statistical data about it (especially during alpha/beta stages). The naive approach would be to just average the times taken for each question, but (thanks again to the above mentioned statistics classes) I'm aware that this isn't really the best way to do it. There are all kinds of normal distribution curves, error calculations, approximation methods, and other arcane mathematical artifacts that I could not even begin to dream about.

Some more information:

  • Questions are of very varying lengths. There are simple yes/no questions, and there are questions that will make you scratch your head. There are questions that ask to enter your contact details, and there are questions that require to meticulously fill a 30-row table. Some questions will ask you to calculate stuff. Some questions will require you to fetch some paperwork from your employer or something. Which brings up another point:
  • The whole process can be suspended/saved and resumed at a later time. This shouldn't happen very often, but given the length of the process, and that some people don't pay attention to the pre-requirements, it can happen.
  • Some questions may not be applicable based on answers to previous questions. For example, there might be a row of questions which are only applicable to businesses and not private persons.
  • Some questions are essentialy duplicated based on previous questions. For example, if you entered that in the past year you worked for 4 employers, you will need to enter the same data for each of them.
  • I can detect:
    • When the user starts/finishes a question. Hence, I can calculate the duration.
    • When the user goes idle. This is rather imprecise. I can only detect when there is no keyboard/mouse activity. This can be because the user has gone away, but it can also mean that he is digging through the papers looking for numbers. Or (since it will be a webpage), maybe he simply switched to another program - or another tab in the browser.
    • Which questions have been answered, which definately will need to be answered, which definately won't need to be answered, and which might need to be answered. For the last type of questions, I'm not sure if it might be possible to calculate a meaningful probability based on previous answers. Perhaps.

So... any ideas on how to tackle this estimate? Remember - I'm going for maximum accuracy (or is it precision? better make it both), and would rather have better estimates than lower resource usage.

Improve this question
$\endgroup$
5
  • $\begingroup$ How much freedom do you have to change question ordering? For example, a first page that has three or four questions that divide users into several classes (business, private person one job over last X years, private person Y jobs over last X years, private person who does consulting on the side, etc) that might have more homogenous question paths and times. Also, can questions be skipped or must they be answered in order? $\endgroup$
    – Wayne
    Commented Jan 2, 2011 at 15:56
  • $\begingroup$ Also, I'd suggest that you display a range, like "Remaining time: 20-45 minutes" that narrows as time goes by, not just a point estimate. $\endgroup$
    – Wayne
    Commented Jan 2, 2011 at 15:59
  • $\begingroup$ @Wayne - yes, I too had a range in mind, though didn't voice it. I think that's a rather minor implementation detail. But anyway, nice to know that others think the same way. :) About the ordering - for the most part I can order them any way I like. It makes more sense to group similar questions together though. And, of course, if a question depends on a previous question, then that order is fixed. But big class-defining questions will definitely go up front. $\endgroup$
    – Vilx-
    Commented Jan 2, 2011 at 19:59
  • $\begingroup$ Questions cannot be skipped as per se, but some can be left empty. Like, if you didn't have any income abroad, then you just leave that table empty. However in most cases I think that will be preceded by a yes/no question in the form "did you have any income abroad"? Or something like that. If you answer "no", then the question with the table will be skipped. It's the "not applicable" point in the original question. $\endgroup$
    – Vilx-
    Commented Jan 2, 2011 at 20:00
  • $\begingroup$ I would keep completion statistics per question and assume that median time for each question is the best estimate for the time remaining (that is, total time remaining is sum of median completion time of all unanswered questions). If you want better than that, you'll probably need some kind of deep learning AI network where the AI can fuzzy match the new person filling in the questions to match some existing case (e.g. it might turn out that some pre-existing domain specific knowledge makes it faster to complete the questionaire and questions 2, 5 and 11 happen to correlate with that). $\endgroup$
    – Mikko Rantalainen
    Commented Nov 12, 2020 at 7:09

2 Answers 2

Reset to default
3
$\begingroup$

I do not see why averages should not work. You might want to use winsorized or trimmed averages to guard against outliers, which are likely due to inability to tell exactly whether user is idle or not. Then for the estimated time remaining show the sum of the averages for each question. You could build more complicated and precise methods if you collected data on the completed time together with user characteristics. For that though you will probably need another questionaire, so unless you are expecting a lot of feedback in alpha and beta phases, I suggest using the simple averages. Sometimes simplest methods work the best.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Hmm... you think it will be enough? Well, I guess that would be easier for me. And, on second thought, after I gather my data, I will always be able to change the algorithm if I see that another one gives better results. $\endgroup$
    – Vilx-
    Commented Dec 31, 2010 at 20:14
  • $\begingroup$ (Oh, btw, you have a typo - it's "winsorized", not "windsorized". Took me a moment to figure it out, since I hadn't heard about it before.) $\endgroup$
    – Vilx-
    Commented Dec 31, 2010 at 20:18
  • $\begingroup$ @Vilx, thanks, I fixed a typo and added wikipedia links. I posted answer via mobile, so that is why I missed typo and did not add links. Also in your place I would wait for other answers, since you cannot expect a lot of them in new year's eve :) $\endgroup$
    – mpiktas
    Commented Dec 31, 2010 at 20:40
  • $\begingroup$ I know, but recently I learned that the "accepted" mark can be removed and re-assigned at any time, so I better accept the answer now, because later I'll just forget to do it. If there is a better answer, I'll accept that. :) $\endgroup$
    – Vilx-
    Commented Dec 31, 2010 at 20:47
  • $\begingroup$ @Vilx, cheers:) $\endgroup$
    – mpiktas
    Commented Dec 31, 2010 at 21:26
3
$\begingroup$

As @mpiktas points out, simple is good. But I can't help thinking there will be substantial variation among users. Roughly, we can conceive of the time to complete each question as depending on the question and on the user, so the trick is to tease out the question-specific timing factors from the experience of previous users and to estimate the user-specific timing factor from the user's experience with the questions they have already tackled.

This opens up the possibilities greatly. A good answer would rely on exploratory analysis of preliminary data. In general, though, a combination of (1) determining a suitable way to express question completion times (logarithms? square roots?) and (2) robust analysis of variance techniques (such as median polish) or IRLS regression is likely to do far better than ignoring the user-specific factors ("simple averaging").

Once you have estimated question-specific timing factors, which could be as simple as computing median or mean times per question, you could easily estimate the timing factor for a new user by comparing their times to typical times for the questions they have already attempted. For instance (this is truly simplistic, but gives the flavor of the idea) if they take twice as long to complete the first five questions as the sum of the typical completion times for those questions, you could estimate that this user will take twice as long as the average to complete the next 95 questions. You can then add the average completion times for those 95 questions and double the sum.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Sounds complicated and unstable. On one hand I understand that statistics and predictions are inherently unstable and imprecise, on the other hand the programmer in me wants an absolute and 100% correct answer. XD I think now that it might indeed better to start out simple (simple winsorized mean sounds good), and then, when I have gathered a significant amount of raw data, I can start thinking about upgrading the algorithm. $\endgroup$
    – Vilx-
    Commented Dec 31, 2010 at 22:18
  • $\begingroup$ P.S. Happy new year! :) (It's 2011 here already) $\endgroup$
    – Vilx-
    Commented Dec 31, 2010 at 22:19
  • $\begingroup$ @Vilx Definitely more complicated than simple averages! -- but you stated you are "going for maximum accuracy," which I took as a strong hint that complications wouldn't be an obstacle for you. As far as "unstable" goes, it's hard to tell what you mean here. What you might do is monitor the total completion time for the first few users just to see how much variation there is. This will indicate how much (or little) would be gained by accounting for their individual differences. $\endgroup$
    – whuber ♦
    Commented Dec 31, 2010 at 22:22
  • $\begingroup$ Well, no, complications aren't an obstacle, as long as someone helps me write out the relevant formulas. XD And sorry about the "unstable". :( I know that the estimates are going to be just that - estimates, and that they will only approximately represent the users remaining time. But since you're even guessing at the method to calculate them (logarithms? square roots?) then that gives the impression of very approximate results. But then - you know your craft much better than I do and it's not really for me to say. Sorry! :( $\endgroup$
    – Vilx-
    Commented Jan 1, 2011 at 8:36
  • $\begingroup$ @Vilx The business about logarithms or roots has to be a guess for me because I don't have the data. For you it won't be a guess because there are standard and rigorous ways to ascertain appropriate re-expressions of data; they are part of exploratory data analysis. $\endgroup$
    – whuber ♦
    Commented Jan 1, 2011 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.