I am a complete newcomer to the field of machine learning. I do have a lot of experience in computer programming, but nothing related to ML.
My question is whether or not ML would be a good approach to isolate portions of a source code file that contain references to a software license. For example, often times a source file will include information at the top of the file describing the code's license, as well as the file's author and copyright info, such as this:
// Copyright (C) 2007-2015 Free Software Foundation, Inc. // // This file is part of the GNU ISO C++ Library. This library is free // software; you can redistribute it and/or modify it under the terms // of the GNU General Public License as published by the Free Software // Foundation; either version 3, or (at your option) any later // version. // This library is distributed in the hope that it will be useful, but // WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU // General Public License for more details. // Under Section 7 of GPL version 3, you are granted additional // permissions described in the GCC Runtime Library Exception, version // 3.1, as published by the Free Software Foundation. // You should have received a copy of the GNU General Public License and // a copy of the GCC Runtime Library Exception along with this program; // see the files COPYING3 and COPYING.RUNTIME respectively. If not, see // <http://www.gnu.org/licenses/>.
This would be an example of a "GPL-3.0+" license. What I would like to do is use ML to isolate and extract this portion of text from the entire source file.
I have access to an existing corpus of such text snippets along with their license classification, so my initial thought is that this would be a supervised learning problem using a classifier of some kind. From what I've read, I also thought that the text can be turned into a vector representation using the bag of words approach, possibly using bigrams instead of single words to preserve some positional information.
A couple of specific questions I have about this approach are:
- There are probably 50 or so different kinds of licenses. Is this too many categories for a classifier?
- For each type of license, I'm guessing there may be about 5 to 50 examples of the license that could be used for training. Will this be enough data, or are many more examples needed? (Or will this approach never work well no matter how much data is available)?
Like I said, I don't have any experience with LM, so I'd appreciate any thoughts or advice on whether or not this approach makes sense at all for this type of problem.
One more bit of background info: I'm currently using Fossology to flag files that may contain licenses. But as far as I'm aware, Fossology doesn't use ML, it just uses keywords and regular expressions to find candidate files that may contain a certain license.