# Is machine learning a viable approach to extract license references from source code files?

I am a complete newcomer to the field of machine learning. I do have a lot of experience in computer programming, but nothing related to ML.

My question is whether or not ML would be a good approach to isolate portions of a source code file that contain references to a software license. For example, often times a source file will include information at the top of the file describing the code's license, as well as the file's author and copyright info, such as this:

// Copyright (C) 2007-2015 Free Software Foundation, Inc.
//
// This file is part of the GNU ISO C++ Library. This library is free
// software; you can redistribute it and/or modify it under the terms
// Foundation; either version 3, or (at your option) any later
// version.

// This library is distributed in the hope that it will be useful, but
// WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
// General Public License for more details.

// Under Section 7 of GPL version 3, you are granted additional
// permissions described in the GCC Runtime Library Exception, version

// You should have received a copy of the GNU General Public License and
// a copy of the GCC Runtime Library Exception along with this program;
// see the files COPYING3 and COPYING.RUNTIME respectively. If not, see


This would be an example of a "GPL-3.0+" license. What I would like to do is use ML to isolate and extract this portion of text from the entire source file.

I have access to an existing corpus of such text snippets along with their license classification, so my initial thought is that this would be a supervised learning problem using a classifier of some kind. From what I've read, I also thought that the text can be turned into a vector representation using the bag of words approach, possibly using bigrams instead of single words to preserve some positional information.

1. There are probably 50 or so different kinds of licenses. Is this too many categories for a classifier?
2. For each type of license, I'm guessing there may be about 5 to 50 examples of the license that could be used for training. Will this be enough data, or are many more examples needed? (Or will this approach never work well no matter how much data is available)?

Like I said, I don't have any experience with LM, so I'd appreciate any thoughts or advice on whether or not this approach makes sense at all for this type of problem.

One more bit of background info: I'm currently using Fossology to flag files that may contain licenses. But as far as I'm aware, Fossology doesn't use ML, it just uses keywords and regular expressions to find candidate files that may contain a certain license.

• But keywords are totally enough to identify the licenses, aren't they? Licenses have standard forms and standard names, there is no "noise". Moreover, if someone uses modified form of a license, then from legal point of view it's a different license. Maybe I'm wrong, but using ML in here sounds like an overkill. – Tim Jun 2 '17 at 17:00
• It's true that the licenses have a standard form, but the license references can be quite a bit different from file to file. That's where keywords come in, but the problem is that they can find false positives. There's also the question of how to extract the relevant text. You may be right that ML is overkill, maybe the best approach is to enhance keyword/regex searching. I'm just trying to explore additional options at this point. – Travis Jun 2 '17 at 17:17
• ML is an overkill for this problem. Simple pattern matching will do the job. – Aksakal Jun 2 '17 at 17:53
• I'm already using Fossology, which does use simple pattern matching. It "does the job", but it doesn't always do it that well. I guess I'm asking if ML could do a better job. – Travis Jun 2 '17 at 18:11