Will the coefficient of determination $R^2$ increase if we remove the outliers in linear regression? Or will the correlation coefficient $p$ would get closer to $0$?
It depends on what kind of outlier / the location of the outlier with respect to the regression line. Imagine a bivariate data cloud without the outlier. X and Y covary, so the cloud is oblong. Draw the regression line. If the outlier lies along that regression line, but far from the cloud so that the observation has high leverage, then the outlier strengthens the X-Y relationship--indeed, one outlier along the regression line could produce a "significant" regression relationship even if X and Y are otherwise orthogonal (spherical). But an outlier lying far off the regression line will pull the regression line away from the other observations, worsening fit and biasing parameter estimates. Your Studentized-deleted residuals will tell you if the observation is likely having such an effect.
If you have multiple outliers, it is more difficult to tell the effect of removing all of them at the same time.