Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

0
votes
0answers
3 views

Tools in Data Science

I am a new one in Data Science and Machine Learning. I have some experience in Java, Python, SQL, Jupyter and most polular libraries like scikit-learn, numpy, pandas, tensorflow, keras. What tools ...
0
votes
0answers
5 views

Finding most related variables independent of variable type

Problem I am analysing a dataset containing variables of different types: continuous, ordinal and categorical. To prioritise in which order to analyse the variables, I would like to evaluate the ...
0
votes
0answers
4 views

Pointwise Mutual Information using spacy or just detailed explaination

So, I have been trying to play around with NLP recently and decided to work on a project involving Emotional Analysis. I have been following this particular research, http://www.cse.yorku.ca/~aan/...
1
vote
0answers
30 views

How to derive a ranking function by analysing feature correlations

I am analysing some employee details to find the efficiency of the employees. Ideally I want some rankings to rank them based on these features. My features include; current salary projects ...
0
votes
0answers
8 views

Import ARFF dataset using RWeka in RStudio [closed]

I am currently using R for Windows verison 3.5.3 and RStudio version 1.2.1335. My goal is to import an ARFF dataset using the RWeka package in order to do some Association analysis, more specifically,...
0
votes
1answer
20 views

Comparing Strings Sequences of Different Length

What's the 'best' way to compare strings of characters of different lengths, which are sequentially ordered, against an ideal sequential string of characters? The sequences come from an experiment ...
0
votes
0answers
14 views

determine suitable values for the parameters of the distance function for this graph

Hi I've been learning data mining and came across this question. I couldn't seem to figure it out myself. So we have a large single undirected graph(without attributes) G = (V,E) and want to detect ...
1
vote
0answers
16 views

How to use Machine Learning to discover important biomarkers in an unbalanced small data set

I have a project which I am just starting out, I am only just learning machine learning and statistics so I am somewhat unsure as to what approaches will be good to start off with, and I am sorry if ...
0
votes
0answers
11 views

Models for identifying anomalies in process data

We have a manufacturing production line with sensors collecting material weights at different locations along the line. The data structure looks like something below: We have noticed instances that ...
0
votes
1answer
23 views

What kind of analysis to use for continues value data set [closed]

I'm new in the data mining field and I'm using RStudio to do the analysis. I have a dataset and I don't know what could be the best method to use. I've tried using correlation and I don't know if ...
0
votes
0answers
10 views

Optimation Association Rule Using Genetic Algorithm

i have association rules that i got from FP-Growth algorithm and i want to optimize that association rules using Genetic Algorithm so i can get the strong rules. Does anyone know how to bring the ...
0
votes
1answer
19 views

Partitioning Around Medoids: Choosing a cluster number larger than the “optimal” one?

I asked a number of 71 'experts' to sort 92 different psychological constructs based on their similarity. Based on their answers, I constructed a dissimilarity matrix. Initially, I wanted to analyse ...
0
votes
1answer
22 views

High Precision and low recall score for TF-IDF when using KNN algorithm

I have twitter data which is labelled with the sentiment(Postive, Negative, Neutral) and I have evaluated the performance of Tf-Idf and Doc2Vec feature extractor using the KNN algorithm and logistic ...
0
votes
1answer
26 views

Can one reverse-engineer and deduce the underlying data for a constrained (max/min) optimization problem?

If one has the result, the constraints are known, but one does not have the input data.
0
votes
1answer
93 views

The reason why Gini index is in favor of multivalued attributes

I am a beginner in data science and I try to understand the similarity and dissimilarity between information gain and Gini index. In my lecture notes, the professor states one of disadvantage of Gini ...
4
votes
2answers
262 views

Differences between prior distribution and prior predictive distribution?

While studying Bayesian statistics, somehow I am facing a problem to understand the differences between prior distribution and prior predictive distribution. Prior distribution is sort of fine to ...
0
votes
0answers
21 views

Can anyone help me with step by step procedure in MATLAB for missing data imputation using Singular Value Decomposition (SVD [duplicate]

I need step by step procedure for imputing data using SVD in Matlab. Please provide resources if any !
0
votes
0answers
13 views

Which Algorithm(s) to apply for in-game behavioural data (choice-based-game)

In the near future, I'm going to analyse in-game and log data. The main purpose is to make predictions about which choices people are going to make in the future when playing this game. As log-data ...
2
votes
0answers
37 views

Is there a well established algorithm to match two documents on a semantic level?

I have a set of documents from a wide variety of topics and I would like to retrieve the ones that are more similar to a new document provided. A search based on common words is not good enough, so ...
0
votes
0answers
16 views

Assessing performance of an agent based on commission rate, market share and revenue

I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following: Number ...
0
votes
1answer
25 views

Principal Component Analysis output interpretation

I am not exactly getting what that selected attributes means....I tried with iris and german dataset and got the first 2 and 10 attributes as result.....whether it gives the first attributes or ...
1
vote
1answer
50 views

Indications about the population when PCA essentially “fails” to reduce dimentionality

This is more of a conceptual question rather than a methodological one I guess. Let's assume that we have a dataset coming from a questionnaire and after some feature scaling we run a PCA to reduce ...
0
votes
0answers
29 views

Measure Naive Bayesian Classifier certainty and error of prediction

So I am fairly new to this so please be patient :) I am using Naive Bayesian Classifier to find the probability of a class (Yes), then I use this probability in another process. Now I am being asked ...
0
votes
1answer
30 views

How to normalize Medical data to further use in Electronic Health Records? [closed]

How can we normalize medical data extracted from patients to be used later in Electronic Health Records? Example for data: Age, temperature, time, blood test. I have come across several papers that ...
1
vote
1answer
61 views

Local variable importance vs Global Variable Importance

Is there any technique to find local variable important? For example in credit card concept, is there any way to specify why a person is not eligible for a credit card? or which features cause this ...
1
vote
1answer
106 views

Meta-learning techniques

what are the meta-learning approaches (methods)? are bagging, boosting, ... meta-learning techniques? is there a good reference for meta-learning techniques? Please give a description in your answer.
0
votes
1answer
13 views

In-sample evaluation with different classifiers

I've tested in-sample evaluation with different classifiers (Decision trees, Random Forests, Gaussian Naive Bayes) within sklearn and Iris datasets. ...
0
votes
0answers
9 views

Comparing two Models gained from process discovery

I have a rather basic question on process mining that I however hardly get answered: currently, I do some research on regulated learning in educational/psychological research: I already drew up two ...
0
votes
1answer
49 views

Random Forest and preprocessing in Data Mining

When applying the Random Forest classification technique, do we have to do preprocessing or is it true that it is not needed for Random Forest ?
0
votes
0answers
14 views

Algorithm MOEA-FHUI

I have a problem in some of the concepts of algorithm MOEA-FHUI. One of my problems is to get the initial population P with the proposed problem-specific initialize strategy. Is it possible to provide ...
0
votes
1answer
94 views

Algorithm to find the attributes that comprise the greatest concentration

I have a porfolio of mortgage loans where each loan has a number of attributes attr1, attr2, .., attrN. I would like to analyze the portfolio credit risk concentration (see below) using these ...
0
votes
1answer
237 views

Calculating lift

I was reading about lift for frequent itemsets and came across this exercise here which was taken from a data mining textbook. I tried plugging in the numbers, but could not get the 0.89 value for ...
3
votes
1answer
460 views

Too many dummy variable in regression model

we have about 50000 models of mobile phone (like Galaxy S7, iPhone 9) in database and the size of data is about 3 million. We want to find the mobile phones that have the least call success rate ( ...
0
votes
1answer
25 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
5
votes
1answer
199 views

Functional Data Analysis and Splines Regression

I'm new to Functional Data Analysis so my question will'be very simple for an expert in this topic. Operatively, when I fit a model like this, in R : model<- lm( y ~ bs( x, df = k, degree = l ) )...
0
votes
1answer
57 views

Statistical time series analysis using Percentile and Empirical Rule

I analysing time-series to see the range of an asset. I have data for GBPUSD so i could only analyze this so far. I have used both empirical rule and percentile. Of course they give different ...
0
votes
0answers
7 views

Supervised ML opinion mining of medical entities: tools?

I would like to extract relations between the entities of 'quality of life' and 'health interventions' out of a corpus of medical texts via supervised machine learning. I wonder whether there is a ...
1
vote
0answers
20 views

Log mining machine learning approach

I am looking for guidance on a log mining problem I am encountering. My dataset comprises of lines like this with low veracity: I am trying to predict whether or not there is going to be a delay on ...
0
votes
0answers
89 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
0answers
6 views

Combining 2 different spatial datasets

I've tried to keep the formulation as general as possible. Problem: For a given location (e.g. a county, country), I have 2 different datasets (D1, D2) where the rows are of the form D1: ...
-2
votes
1answer
99 views

Why isn't Matlab's Classification Learner the End All? [closed]

After building some machine learning models in Python, R and Matlab, I found the latter's Classification Learner App to be immensely powerful. In the time it took me to build a single model in Python ...
1
vote
2answers
37 views

What are possible approaches for learning causal DAG of events?

I have historical data of event logs. Each event has an associated contextual Id, which can be used to tell that event A happened first in some context, then event B happened in same context and then ...
0
votes
1answer
87 views

Violating “compactness” in single linkage hierarchical agglomerative clustering

While I was studying Hierarchical Agglomerative Clustering in the book Elements of Statistical Learning in the chapter Unsupervised Learning, I came through the following : The statement The ...
1
vote
1answer
61 views

Clustering similar user reviews

I am using Amazons Video Games dataset ( 1.3 mil user reviews ) in order to make a recommendation system using LSTM networks. What I would like to do is to cluster ...
0
votes
1answer
69 views
0
votes
0answers
28 views

Formulating an Anomaly Detection method for a dataset based on time

I have the following data (temporal feature) for n videos of given length: start times of all the users who watched the video duration of the video viewed Now, given a video (given the start times ...
3
votes
1answer
655 views

When is weighted average of $F_1$ scores $\simeq$ accuracy in classification?

Example where accuracy $\simeq$ weighted average of $F_1$ scores I have a classifier which classifies between 2 classes, $\mathbf{A}$ and $\mathbf{B}$. Let's say that we have the confusion matrix ...
0
votes
0answers
222 views

classification With ordinal data in R ?

I have a survey data which is Likert scale ordinal data . I want ordinal classification with R . I try several data mining algorithm with this data set but my accuracy is too low . How can i improve ...
1
vote
0answers
537 views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?