When is it okay to label data yourself? (And semi-supervised learning)

i'm pretty new to machine learning so i think this might be a realy basic question.

Let's imagine the following scenario: I want to classify projects as either active or inactive. Projects can be defined by number of contributors etc. The goal is to identify projects with a high risk of abandonment by their owner / contributors. A project can be labeled as "inactive" by its owner. I made the following assumption: if someone labeled his project as inactive, its very unlikely that its active. On the other hand, not having the label does not imply its active, because not everyone goes back to label a project inactive after loosing interest. For the sake of this question: lets say that 10% of the data is labaled as inactive, but in reality, 30% of projects are inactive.

At first i thought: ok, i'll just go through all the data and check for the active ones if they are inactive in reality. But, if i do that, why would i need a model anyways? I would base this labelling on something like "no activity in one year, 0 contributors etc.). And now i am confused: is it right that in this scenario, its not okay to label the data myself? Is it only okay to let humans label data on something like images? Is there any scientific literature that i could quote on this topic?

Additional question: would semi-supervised learning be a use case for this? My idea would be to use a supervised method to train a classifier on the 10% labelled data and then use the unlabeled data aswell (like here: http://matpalm.com/semi_supervised_naive_bayes/does_it_do_any_better.html).

But for testing, can i only use projects that are actually labeled (from the 10%)? So, lets say i use 50% of the projects that are labeled as inactive for training the classifier and 50% for testing. I could validate my model with the test data. Now i would use semi-supervised learning in my modell aswell and compare it again. Is it possible that this would get me better results?

If you are using some simple rules to determine if a project is active or inactive ("no activity in one year, 0 contributors etc"), you don't really need to use machine learning to predict the outcome of those rules. You can just apply the rules directly to the new data points.

What machine learning could be used for is to predict beforehand which projects will be labeled as inactive with your rule system. In business churn models are used to predict which customers might become inactive in the future. A simple churn model could output the probability that a project will have any activity within the next one month.

This approach is not able to use the data on the projects that have been labeled as inactive by the users. It could still be used as one of the features, but I would avoid using it as the only training data as it does not sound very reliable source of information.

I think most of your questions are answered when you more rigorously specify your problem. So let's back up and define the basics...

First, we need to define our target. It sounds like it's inactive, where inactive might be defined as "marked 'inactive' by the project owner", or "no activity in a year", or "no followers", or "0 contributors", or some combination of the above or something else entirely. Let's assume for now that inactive means that activity has been 0 for 12 months. (activity may itself be an aggregation of underlying variables, of course, but let's treat it as a single time series variable.)

Once we've firmed this up, you're correct that you don't need a model if you have the information you need about a project to see if it meets this definition or not. So if you decide that inactive means "no activity in the last year" and you have a projects activities for the last 12 months sitting in front of you, you don't need a model. (You might still want software to calculate this across many projects for you, but you don't need a model.)

So now you need to define a decision and a decision point: the purpose for a model.

At some point in time, you will have some information about a project, but not your target information (activity and hence inactive). Your model cannot use any information beyond what you expect to have at the decision point to make its prediction. In supervised training, you will have this information and the target, of course, but not at predict/score (i.e. decision) time.

So what is the decision point and what is the decision? Do you want to say, "For a project that is at least 12 months old, given the number of commits in the first 3 months, the current number of contributors, the current number of followers, and the number of forks over the last year, what is the probability that the project will be inactive a year from now?" You don't know what the activity will be for the next year so you need a model to predict. (Let's assume that activity is more than forks.)

Or you could say, "For a project that is at least 12 months old, given the number of commits in the first 3 months, the current number of contributors, and the current number of followers -- but not knowing anything about the activity for the last 12 months -- what is the probability that the project is actually currently inactive?" In this case, you're not talking about the future, so you could conceivably have activity information for the last 12 months but a model is only needed if you in general will not have that information at decision time. (Maybe activity stats are only updated at the end of the year, or maybe projects can hide activity but not the other variables.)

Say you adopt the "no activity for the previous 12 months" definition of inactive and the first statement of your problem -- the one in the future -- as your goal. If you have, say, five years of commits, contributors, followers, forks, and overall activity (your target) you can easily write a program that chunks this information up per week and creates your target based on the previous 12 months.

It will be trivial to calculate your target for the first four years, though you won't be able to calculate it for the last year. (Even hand-labeling can't do this, based on your definition of inactive. A human could guess that a project that's had no _activity` in 11 months will almost surely not have any for the next month, but that is itself a prediction.)

CAN you hand-label data? Yes, if you have the information and expertise to do so. Can you use semi-supervised learning? Yes, but it's tricky and may make things worse. Do you have to address that in your current problem? It doesn't look like it.