Test for statistically significant between two techniques

Question

Let's say there are two groups of data A and B, each uses a different technique. In group A, there are independent variables X (numerical), for each x, the associated technique produces corresponding value y (numerical). Similar for group B.

How do I test for statistically significant difference between two techniques? If there is only one value in each group then I can use ANOVA, but now each group has (X, y) so I have no idea what to do.

I am thinking of fitting a linear regression using OLS for each group then test for significant difference of respective coefficients between two regressions. But I am not sure this approach is correct.

I am a beginner in Statistics, I don't know what keyword to search for so I ask here. I am happy and willing to learn new things.

I am also wondering for the more generalized version, if X and Y are categorical variables, or X is numerical and Y is categorical, ... then how the above problem can be solved?

Answer 1

Assuming your experiment is designed correctly and there are no confounding factors, your suggestion to analyze the data via OLS is fine.

A model

$$ y = \beta_0 + \beta_1 x + \beta_2I(\mbox{group}=B)$$

should suffice assuming you posit no interaction between group and $x$. The test for $\beta_2$ should tell you if, conditioned on $x$, the difference in groups is consistent with a null effect or not.

If $y$ is categorical, you can perform a multinomial regression or ordinal regression (depending on if $y$ is an ordinal variable or not) where things are slightly different but similar. In this case, I would refer you to Agresti's Categorical Data Analysis for more.