1
$\begingroup$

Say I have 2 groups of people, selected randomly. For each person, I know some characteristics about them, in the form of continuous and categorical variables. For example, that could be the person's weight (continuous integers), whether they smoked or not (binary variable), number of 10 min+ walks per week (1, 2, 3, 4+) etc...

At this point, I am interested in deciding wether the 2 groups look identical, according to my control variables stated above.

I know I can compare the 2 groups for each variable. I could perform a t-test on the group's weight distributions. I could run a chi-squared test when the variables are categorical.

I would like an overall test that will, at the end, give me one p-value describing the difference between the 2 groups and decide if I can consider them significantly different or not.

Does such a test exist? If not, how would you tackle this problem?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

If the purpose of this exercise is to assess balance for the purposes of comparing groups in an observational study, you shouldn't be using hypothesis tests to do so. This is commonly referred to as the "balance test fallacy". In a balance test, you only care about assessing whether the groups in your sample differ from each other. See Imai, King, & Stuart (2008) and Ho, Imai, King, & Stuart (2007) for a discussion.

If you are truly interested in making an inference about the population from which your sample was drawn, what you probably want is a multivariate t-test (MANOVA), or the logistic regression approach could work as well. One problem with the logistic regression approach is that it tests for conditional associations between each variable and the grouping variable, when you actually want marginal associations. The MANOVA makes some assumptions about normality.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hello Noah, thanks for the reply. So far, I have used the Hotelling's T squared method and it seems that according to your comment, it was appropriate. According to this PDF: ibgwww.colorado.edu/~carey/p7291dir/handouts/manova1.pdf "MANOVA is the multivariate analogue to Hotelling's T2". in my case, I have only 2 groups, so Hotelling T square test would be appropriate. Cheers! $\endgroup$
    – Damien
    Jul 5, 2019 at 20:32
0
$\begingroup$

At first, when there is random selection, then it is not needed to perform such tests.

In cases, where there is a reason to think, that selection was not random, but possibly the populations do not differ in features, we might build a selection model. First choice would be logit with group as dependent variable. Testing for significance of whole model would give the right answer, as long you have chosen good variables.

It is actually necessary to have one test for all the comparisons, because we avoid multiple comparison problem this way.

To perform joint test for significance of all variables in logit model, first-choice test would be Likelihood-Ratio test. Some statistic software does it automatically, when you estimate the model. One can thing of this test as equivalent of the F test for OLS regression.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hello @cure, thanks for the reply. The model would be something of the form y = f(categorical or continuous variables) where y takes the value Group_1 or Group_2. I would train on the data that I already have. Once the model is trained, I can test for the significance of the parameters. However, you are mentioning the significance of the whole model. Is there a particular procedure you would follow? Or Do you have a link/example you could share? Thanks! $\endgroup$
    – Damien
    Jul 3, 2019 at 15:13
  • $\begingroup$ Hello @Damien, I added paragraph suggesting a joint test for logistic regression model. If you looking for procedure to estimate such model, at first you need to choose software (you probably have already using one). Then I think you should just go through some tutorials for logistic regression using such soft. I hope this would be enough to perform such thing! $\endgroup$
    – cure
    Jul 4, 2019 at 14:22
  • $\begingroup$ Sounds good, thanks for the suggestions! I will have a look. So far, I have tried the Hotelling T square distribution, which seems to do the trick, but I still have to look more into it. $\endgroup$
    – Damien
    Jul 5, 2019 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.