How can I statistically test that the skew towards pass (1) is greater than if pass and fail results were generated randomly?

Testing this hypothesis would lend itself to a binominal test comparing the number of students passing, out of total students tested, with the default proportion 0.5.

But is this really what you want to know? It is probably not a relevant question whether the proportion of students getting the correct answer is different than 50%. And as @whuber pointed out, there is no reason to suspect that for this kind of task that incapable students would get the correct answer 50% of the time. (This would be the case if it were a multiple choice question with two options.)

My suggestion would be to calculate the proportion of students passing, and calculate a confidence interval about this proportion. This result would probably be meaningful for what you are trying to determine.

When calculating the confidence interval, be sure to use a method appropriate for proportions.

The following is the R code to find the confidence interval for the Example Data in the question. There are other methods for confidence intervals for proportions.

Pass.Fail = c(1,1,1,1,1,0,0,1,0,0,1,1,1,1,1,1,1,1,1,0,1,1,0,1,1) 
Pass = sum(Pass.Fail)
Total = length(Pass.Fail)
binom.test(Pass, Total, 0.5)

This gives the results: 19 pass out of 25, for a proportion of 0.76, and a confidence interval of about 0.55 to 0.91.

This still allows you to compare to proportions of interest, based on whether the confidence interval overlaps a given proportion. So in this case you can say statistically that the proportion of students capable of getting the correct answer is greater than 0.50, but you cannot say it is greater than 0.75. Likewise, the proportion of students is likely less than 1.00.

How can I statistically test that the skew towards pass (1) is greater than if pass and fail results were generated randomly?

Testing this hypothesis would lend itself to a binominal test comparing the number of students passing, out of total students tested, with the default proportion 0.5.

But is this really what you want to know? It is probably not a relevant question whether the proportion of students getting the correct answer is different than 50%. And as @whuber pointed out, there is no reason to suspect that for this kind of task that incapable students would get the correct answer 50% of the time. (This would be the case if it were a multiple choice question with two options.)

My suggestion would be to calculate the proportion of students passing, and calculate a confidence interval about this proportion. This result would probably be meaningful for what you are trying to determine.

When calculating the confidence interval, be sure to use a method appropriate for proportions.

The following is the R code to find the confidence interval for the Example Data in the question. There are other methods for confidence intervals for proportions.

Pass.Fail = c(1,1,1,1,1,0,0,1,0,0,1,1,1,1,1,1,1,1,1,0,1,1,0,1,1) 
Pass = sum(Pass.Fail)
Total = length(Pass.Fail)
binom.test(Pass, Total, 0.5)

This gives the results: 19 pass out of 25, for a proportion of 0.76, and a confidence interval of about 0.55 to 0.91.

This still allows you to compare to proportions of interest, based on whether the confidence interval overlaps a given proportion. So in this case you can say statistically that the proportion of students capable of getting the correct answer is likely greater than 0.50, but you cannot say it is likely greater than 0.75. Likewise, the proportion of students is likely less than 1.00.