0
$\begingroup$

I have two sets of data from two groups of participants that are identical in all but one way: one set of data were collected using software A and the other set was collected with software B. It's possible that the outcome variables are roughly equivalent across groups, and it's possible that they're not. (The groups are quite similar otherwise; I've run chi-square tests on gender and race and there are no differences by the latter and only minor differences by the former. Regardless, such differences should not be systematic.)

What I want to do is run some preliminary tests to see whether there are significant effects of software engine on my three primary outcome variables. If there are, I'd then treat these as two separate experiments. If there are not, I'd instead group them together.

What I had in mind was to use a straightforward linear regression, as so (I'm using R, but obviously this is not specific to the stats program):

combined <- rbind(data1, data2)
reg1 <- lm(outcome1 ~ engine, data=combined)
summary(reg1)
# and repeat for outcome2 and outcome3

If this regression shows a significant effect of engine for any of these outcome variables, then that would mean that I should keep the groups separate. It also occurred to me to do this with other covariates in the model:

combined <- rbind(data1, data2)
reg1 <- lm(outcome1 ~ engine + age + gender + race, data=combined)
summary(reg1)
# and repeat for outcome2 and outcome3

In this case, I'd be seeing if engine had a significant effect in the presence of other relevant variables.

Any thoughts on which method makes more sense, or if there's a better way?

| cite | improve this question | | | | |
$\endgroup$
0
$\begingroup$

If you are unsure then including higher order polynomials and interactions in the above regression would serve as an additional safety net. I would also suggest a pair-wise scatter matrix. In the end, one could spend an eternity attempting locate potential differences ie run some latent allocation models to see if they discriminate between the two samples. However, I think simple tests should suffice.

The safest way to play this, in my opinion, would be to run the tests/simulations(whatever you are doing) on the combined dataset and then use the individual tests as a robustness check. If the results of your combined test yield something that differs from the individual tests then red flags should go up, otherwise, you should be fine.

| cite | improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.