Conducting analyses with participants who contribute varying amounts of data?

Question

I have data from approximately 200,000 people all of whom do a task at least once, and I record the time it takes them to complete the task. They can complete the tasks in one of two ways (method A or method B). Most people choose to use method B, but some use method A, and some use both (considering most people repeat the task a number of times). If I wanted to determine whether one method was more efficient (or faster) than the other method, is it okay to stick to parametric statistical measures given the following qualities of the data?

The number of data that each person contributes varies due to there being no limit to the amount of times each person repeats the tasks. Thus, for example, one person might contribute 50 trials (trials referring to one repeat of the task) using method B and 20 trials using method A. Another person might contribute 1 trial of method A and 1 of method B. Another person might contribute 10 trials of only method B.

If I averaged all the trials per method for every person, and then conducted a within-subject t-test according to method used, then method A will be much more variable than method B. One workaround I have considered is to remove any people who did not contribute data to both methods, and to randomly remove data from each person until their number of trials for each method are equal (thus equal variance between methods). The unfortunate consequence of this is that the vast majority of people end up having only a single trial for each method remaining (because most people, if they contribute to both methods, only use method A one time). Is this still an okay workaround to use if I am restraining myself to a parametric approach? Or am I thinking too much and do not need to resort to removing data at all? This also still does not get around the possible problem that people who contribute more trials to both methods are left with an average that is less variable than those who contribute one trial to both.

Answer 1

It depends on what you'd like to find, the properties of methods A and B, and what mechanisms you suspect might be in play for each method. If your intent is to compare methods A and B directly, I suggest comparing times from the first execution of method A with times from the first execution of method B.

Other analyses using other subsets of your data will be appropriate for other purposes. With no limits given to participants on how many trials to undertake, individual-level variations will govern that number. Averaging times for method B may introduce some error formerly in your error term into the measure itself, regardless of effects on variance.

From my reading of your question it sounds like a major issue might be practice-- people may become better (presumably represented in data by shorter task times) at either method as they continue to use it. If so, the varying numbers of trials could be a big issue. The comparison is no longer between A and B, all else being equal, but between less-practiced A and variably practiced B (where B is nearly always practiced more than A, perhaps substantially so).

Other questions certainly exist, and different stratification and data schemes will probably suggest different approaches.