What kind of classifier should I be using? I have a script that scans text files and searches for keywords related to cryptography. And I'm scanning source code files. A lot of the times, I get back some evidence of cryptography like "AES_encrypt", but sometimes it can be a false positive, e.g. the keyword "crypt" in the string "cryptic words". 
To get rid of false positives, I designed an app that allows a human expert to go in and flag each hit as positive or false-positive. I want to gather this data and train a supervised classifier that learns from the human's expertise and gives a probability that a certain future hit is a positive. My training data looks something like this:
keyword | keyword_group | is_source_file | P/FP
"aes_encrypt" | "aes" | True | P
"aes_encrypt" | "aes" | True | P
"aes_encrypt" | "aes" | False | P
"aes_encrypt" | "aes" | True | P
"aes_encrypt" | "aes" | True | FP
"aes_encrypt" | "aes" | False | FP
"decrypt_aes" | "aes" | True | P
"decrypt_aes" | "aes" | True | P
"decrypt_aes" | "aes" | True | P
"decrypt_aes" | "aes" | True | FP
"decrypt_aes" | "aes" | False | P
"decrypt_aes" | "aes" | False | FP
"decrypt_aes" | "aes" | False | FP
"rsa_key" | "rsa" | True | P
"rsa_key" | "rsa" | True | P
"rsa_key" | "rsa" | True | P
...

Keywords are organized in groups. For example, "aes_encrypt" and "aes_decrypt" are both under "aes" group. is_source_file indicates whether this hit was found in a source code file like a ".cpp" or a ".py" file as opposed to other file extensions like a ".txt". 
I want to give the classifier a keyword and its group and whether it was found in a source file, and I want back a probability that it is a positive. What kind of classifier does this job?
 A: Basically any classifier can do this. For example a Artificial Neural Network that outputs a softmax layer, which can be seen as probability. You would design a ANN that gets as input a numerical representation of what you described as features and outputs a value $p$. That value basically says "the probability of the input belonging to class 0 (e.g. P) is $p$ and belonging to class 1 is therefore $1 - p$.
There are a lot of implementations already ready to go.
A: It might be overwhelming to start off, so I will try to give you some advice and some references that hopefully can get you started.
A simple approach is to treat each row as a text document with three words.
The "keyword_group" and the "keyword" columns need to be represented as vectors.
The simplest way to do that is to convert your document into a (sparse) vector, where each column is the count of a certain "keyword" or "keyword_group" (so the number of columns will be equal to size of the vocabulary). There are other ways of doing word embeddings, but this is the easiest.
If you are familiar with python, here is a good reference of how to do that.
The "is_source_file" is a binary variable, so you can easily represent it as 0 or 1 and add it as a new feature.
At that point, you can use any classifier. Here is an example where a lot of them are used. It's a famous dataset for which the task is to to classify documents into 20 categories. You only have two, but the problem is similar.
Finally, as others have mentioned, you can add more words to you documents taking combinations of those.
