When you regress portfolio excess returns against relative benchmark excess return you get a model in which the beta (slope) could be interpreted as the one you get from the CAPM, that is systemic risk. According to Jensen the alpha (intercept) represent the stock picking ability of the fund manager. Evaluating the alpha let you make decision about the fund manager grade of activity: if alpha is 0 you can say the investment policy is passive, if alpha is negative you can say the fund manager ability to select asset was not good. The more positive alpha is the better is the fund manager ability in security selection. How do you interpret the significance level of the alpha? What can you say about the fund manager ability in stock picking looking at significance level? I would say that if the beta is significative, the R squared is high and the F statistic is significative the model fitting is good for the portfolio returns. But if the alpha is not significative?
This comes down to hypothesis testing the intercept of a simple linear regression: $y=\alpha+\beta x$.
If you test $H_0: \alpha=0$ versus $H_a: \alpha \ne 0$, you can say if the $\alpha$ is significantly different from 0. However, it matters if that is a positive or negative difference, so you would pay attention to whether or not the calculated $\alpha$ value is positive or negative.
If the p-value is significant, and the calculated $\alpha>0$, the manager is outperforming passive investment. If the p-value is significant, and the calculated $\alpha<0$, the manager is underperforming passive investment.
If all you want to know is if the manager is outperforming passive investment, you can do a one-sided test of $H_0: \alpha=0$ versus $H_a: \alpha >0$.
You don’t want to do the usual F-test in this case, as that will test the slope, not the intercept. That is, the “F-test” output of a typical software, say R, will not give you information about the intercept.