I have a project for my regression course. I'm curious how one chooses the dependent/independent variables. From my understanding, we want a model that has as little variance as possible, so would I choose the model that has the fewest outliers?
I am thinking you might be confusing a couple of things. Variables can be independent or dependent. The dependent variable is the one you are interested in knowing the outcomes of; for example, if you were interested in how much people earn (we will call that variable PAY), you might think there are certain predictors for the differences, and let's say you've got a hunch that level of education is a predictor. In that case, EDUCATION would be independent, and PAY would be dependent.
A separate issue from that is that in linear regression models, you need to ensure the residuals are independent. This is a 'model assumption' for linear regression. Outliers are part of your dataset, so no, you don't want to avoid them. What you have to do is work out whether you can fit your regression model to the data (ie. the data fits the model assumptions) or if for some reason (such as perhaps undue influence of an outlier) it does not fit, then you have to fit another model. Sometimes its OK to handle an outlier problem by analysing the data excluding that one bit of data, but you have to tell people you've done it and why, and what the outcome would have been if you'd left the outlier in the data.
The variance issue is that the residuals need to have constant variance. This is another linear regression model assumption. You check it by creating a scatterplot of the residuals.