1
$\begingroup$

Suppose that I am doing a classification problem where I classify people into two categories as bullied or not. In such type of research there are several datasets available such as; cyberbullying in students, cyberbulling in social media, cyberbulling in women etc. i.e. the problem is same (i.e. cyberbullying), but the test population is different.

What I currently do is using my features in each dataset and train models seperately for each dataset using 10-fold cross validation at get results.

However, since the underlying concept of these datasets are same (i.e. it tries to detect the bullied people), I am wondering if I can train machine learning model in one dataset using all its data points (say cyberbulling in students dataset), then test it on remaining datasets fully using its data points (say cyberbulling in social media, cyberbulling in women etc.) and get only the results of the tested datasets. However, I am not sure, what is the conclusion that I am trying to make through this experiment.

Is it like I am saying, I trained the model fully on student data, and applied it on social media and women. Does it show any generalisability of features? What are the conclusions that I can make through this?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Do these data sets share any relevant common features? Generally you can do some research into domain-adaptation and transfer-learning, as those cover the described scenario well. $\endgroup$
    – deemel
    Dec 5, 2019 at 8:12
  • $\begingroup$ @deemel thank you for the comment. The objective of my dataset is same (i.e. they are all labelled using same conditions). However, the population is different (i.e. students, social media, women etc.). Since the objective is same, just wondering whether transfer learning is suited to me? $\endgroup$
    – EmJ
    Dec 5, 2019 at 11:06
  • 1
    $\begingroup$ The populations might be different, but are they at least described with some common features? $\endgroup$
    – deemel
    Dec 5, 2019 at 11:49
  • $\begingroup$ @deemel yes, I am using the same feature set in every dataset. My features are only linguistic features, so, they can be extracted easily using the text in each dataset. So, the same features are used in the datasets. Please let me know your thoughts. Thank you :) $\endgroup$
    – EmJ
    Dec 5, 2019 at 22:47

1 Answer 1

Reset to default
1
$\begingroup$

Since you wrote that you actually use the same features for all data sets it is of course possible treating them as one data set.
My suggestion would be joining them in one data set while introducing an identifier for the original source data set they came from, e.g. a categorical feature 'source' that indicates whether its from the women, social media, students data sets etc. .

The answer of possible generalization hasn't even been touched though, based on what you describe. For that you need to investigate the relationship between the new 'source' feature and the class, which can be approached in multiple ways.
A start to this could be training a classification model on the merged data set while leaving out the 'source' feature, then examine the resulting predictions with regard to the source of the instances.
Have a look how the classification performance is for the different data sources - if the source of the data does not matter and the concept of 'bullied or not' is similar in all source contexts, we'd assume that we observe no difference regarding the classification.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for the suggestion. So, if I get you correctly, you are proposing to merge the three datasets and perform one classification. So, for example, think that I get 69%, 75%, and 78% correct in each classification in the three datasets. In the one classification (that you proposed) I get 68%. Does this means that my features are universal? Please correct me if I am wrong. Looking forward to hearing from you :) $\endgroup$
    – EmJ
    Dec 6, 2019 at 9:17
  • 1
    $\begingroup$ "68% correct" reduces the problem evaluation in an uninformative manner - what you are interested in is "what did the model get right, and what not?", that's what I meant with evaluating the model with respect to the source. It might as well be that most of the incorrect classifications come from one source - would you really say, that the features are universal then? So, look at your results carefully and ask yourself questions along this line, otherwise you can't really answer your question. $\endgroup$
    – deemel
    Dec 6, 2019 at 9:33
  • $\begingroup$ I see, I think I get what you described in the answer now. However, I am not clear the conclusion that we make from this line if the source of the data does not matter and the concept of 'bullied or not' is similar in all source contexts, we'd assume that we observe no difference regarding the classification. If the source does not matter and the results is mostly similar when doing it individually or merged, what does that mean? Looking forward to hearing from you. Thank you :) $\endgroup$
    – EmJ
    Dec 6, 2019 at 10:07
  • 1
    $\begingroup$ Comments aren't for extended discussion - let's move this to a dedicated chat room $\endgroup$
    – deemel
    Dec 6, 2019 at 10:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.