Which kind of regression should I use for my variables

Question

I checked for related questions before posting this. I found some similar questions (like this), but I couldn't find any question that answers mine.

I have two independent variables, one continuous and one categorical (1 or 0). My dependent variable is continuous in [0,1].

I want to analyze the effect of each independent variable. I understood I should apply regression analysis. However, according to resources I read like here and here, there is more than one way to do that. I tried to do Linear Regression (Ordinary Least Squares). The $R^2$ I obtained is 0.509. Here is the residual plot I obtained:

I think that $R^2$ is not good enough. So, I want to improve my model. I tried to analyze the residual plot by using information from here. Unfortunately, I think my plot doesn't look like any other plot provided on the last link I shared. It is said that there shouldn't be any pattern in the residual plot but I cannot decide whether my plot contains some kind of pattern.

To sum up, I don't know what should I do to improve my model: Should I try to use a non-linear model? Should I transform my data points ?

EDIT

As requested in the comments I put some other figures:

1) X0 - Y chart

2) X1 - Y chart

3) Some of the data points I have

Answer 1

A regression model $Y = a + b_0 X_0 + b_1 X_1$ (the slightly awkward and non-standard notation matches your predictor names) plots as $a + b_0 X_0$ for $X_1 = 0$ and $a + b_0 X_0 + b_1$ for $X_1 = 1$, i.e. two parallel lines in $X_0, Y$ space. Here you can see a tilt in the main cluster of points which does not match the regression especially well, but you can also see that data points in the NE and E corners of each plot exert leverage pulling up the regression surface.

Although not shown here, a quadratic in $X_0$ (with $X_1$ as extra predictor in the same way) does not seem to offer worthwhile improvement. I don't have other models to suggest, nor do I suggest any transformations. It's absolute that $X_1$ cannot be transformed usefully (as any pair of values other than 0 and 1 would yield equivalent models), while it's hard to envisage a useful transformation for $Y$ given its narrow range. $X_0$ is the most obvious focus of attention.

The narrow range of the outcome from about 0.48 to 0.55 seems unusual, at least until the OP offers some kind of explanation. We have no context here on what kinds of relationship are substantively possible or plausible (or conversely impossible or implausible).

For $R^2$ Stata gives me 0.3725.

Answer 2

I think a lot of people look at R-squared as a goal post (the higher the better). In my opinion, this is a little misguided. The R-squared is intended to tell you what proportion of variance your covariates explain. It can be shoe horned into a model fit metric, but it isn't a good idea. R squared can be low simply because the data are noisy, not because your model is a bad fit. I think that is what is happening here.

Take a look at your residual plot. There are a lot of points around 0.52, but they take on wildly different values. No amount of non-linear modelling will help reduce that variance without overfitting the hell out of the data.

If you are insistent about using non-linear methods, some black box ML method might to best. I can't say more without having the data myself, however.