Hiding features from your client I have to automate a yes/no type business decision problem for a customer (think: Is the use of chemical compound X beneficial in combination with chemicals A,B,C?). He dumped on me a very large dataset that contains all the data that I need (and much more) and basically said "I don't care what you do, as long as in the end a yes or no answer comes out in which I can be fairly confident".
From this dataset I tried various sets of features, so that I can achieve a very good prediction score via a binary classification algorithm.
This model I have incorporated in a software that I will deliver to him, where the input is his whole dataset. Internally the software then computes the features that I have identified in my own analysis as being good ones and makes the prediction.
The problem is: The customer now wants me to prove to him that the good prediction score that I claim is actually true. But it seems to me I cannot prove that to him, unless I give him a good part of my features, so that he can check himself, which I don´t want to do, since that is my IP....
 A: 
The customer now wants me to prove to him that the good prediction score that I claim is actually true

As analytical chemist, I think this a totally valid request. It is part of the necessary method validation. 
As business owner, I'd say that who is to do the work (you, the customer themselves or a third, independent party) is a matter of contract and possibly regulations. Likewise, the question of whether the features should be handed over to the client is a matter of the contract/license. 

But it seems to me I cannot prove that to him, unless I give him a good part of my features

I'm sorry to be so harsh, but at least in the context of analytical chemistry/chemometrics (which your application description hints at as the relevant field) this is so blatantly wrong that it would make me quite wary in accepting any claim of your's about the predictive quality of your model. 


*

*Verification of your model's predictive performance can (and should) be done with a (well designed) test of test cases which are subject to blinded or double-blinded prediction.
Depending on the application, this may be subject to regulations that may even prescribe these blinded tests to be done by an independent third party, in regular ring trials or the like.
While we work a lot with resampling validation up to some point, we are also aware of the limitations of resampling: it is often very hard (or plainly impossible within a given dataset) to achieve statistical independence in the splitting procedure for resampling. To the point where it is usually less expensive to check performance against a set of new samples, where independence is easier to achieve (acquired later = comprises drift, at a different facility/reactor, etc.).
The client may be thinking of such a verification which would not imply that you reveal the features.

*Validation has a wider scope, and as soon as your predictions do have a certain importance in term of harm that may be done by wrong predictions, I'd at least think it very desirable that the actually evaluated features are subject to some scrutiny during validation. I.e. customer (or the third party doing the validation) should compare your findings from the data with their knowledge about the application.
Even many kinds of models that are hard to interprete in terms of their features allow at least to check whether the evaluated features do not contradict with known behaviour of the application system.
So in the context of such a wider validation, I'd think it a sensible request from the customer to ask for your features. 
So I'd recommend to 


*

*find out what exactly the customer wants, and then 

*if they are asking for the features and your contract really does not include handing over the features, you can surely prepare an offer to them at which price you are willing to sell the features.  

A: You could provide the predictions themselves and have your client compare them to the observed values himself. But then your client has no guarantee that you trained the model properly, with cross-validation or something, instead of just giving him back the observed values with a little noise added to make them look like predictions. In general, the client has no way to check that the model isn't cheating in some sense without being able to see the model.
An exception would be if the client sends you only the covariates for some observations, keeping the true corresponding values of the dependent variable to himself. He could then get predictions from you which he knows you couldn't have cheated to get. But this is only an option if he hasn't already given you all the data he has, or he can get more data.
Even more generally, it's never wise to make decisions with proprietary software, especially software you can't even see the source code of. Instead of putting a client in this position, make your work transparent and publish your source code.
