Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

2
votes
0answers
32 views

Is there a well established algorithm to match two documents on a semantic level?

I have a set of documents from a wide variety of topics and I would like to retrieve the ones that are more similar to a new document provided. A search based on common words is not good enough, so ...
0
votes
0answers
12 views

Assessing performance of an agent based on commission rate, market share and revenue

I have a set of data for agents selling properties (apartments) for a company in different states. The company would like to assess the performance of the different agents given the following: Number ...
0
votes
1answer
25 views

Principal Component Analysis output interpretation

I am not exactly getting what that selected attributes means....I tried with iris and german dataset and got the first 2 and 10 attributes as result.....whether it gives the first attributes or ...
1
vote
1answer
43 views

Indications about the population when PCA essentially “fails” to reduce dimentionality

This is more of a conceptual question rather than a methodological one I guess. Let's assume that we have a dataset coming from a questionnaire and after some feature scaling we run a PCA to reduce ...
0
votes
0answers
22 views

Measure Naive Bayesian Classifier certainty and error of prediction

So I am fairly new to this so please be patient :) I am using Naive Bayesian Classifier to find the probability of a class (Yes), then I use this probability in another process. Now I am being asked ...
0
votes
1answer
26 views

How to normalize Medical data to further use in Electronic Health Records? [closed]

How can we normalize medical data extracted from patients to be used later in Electronic Health Records? Example for data: Age, temperature, time, blood test. I have come across several papers that ...
1
vote
0answers
32 views

Local variable importance vs Global Variable Importance

Is there any technique to find local variable important? For example in credit card concept, is there any way to specify why a person is not eligible for a credit card? or which features cause this ...
1
vote
1answer
100 views

Meta-learning techniques

what are the meta-learning approaches (methods)? are bagging, boosting, ... meta-learning techniques? is there a good reference for meta-learning techniques? Please give a description in your answer.
0
votes
1answer
13 views

In-sample evaluation with different classifiers

I've tested in-sample evaluation with different classifiers (Decision trees, Random Forests, Gaussian Naive Bayes) within sklearn and Iris datasets. ...
0
votes
0answers
8 views

Comparing two Models gained from process discovery

I have a rather basic question on process mining that I however hardly get answered: currently, I do some research on regulated learning in educational/psychological research: I already drew up two ...
0
votes
1answer
30 views

Random Forest and preprocessing in Data Mining

When applying the Random Forest classification technique, do we have to do preprocessing or is it true that it is not needed for Random Forest ?
0
votes
0answers
13 views

Algorithm MOEA-FHUI

I have a problem in some of the concepts of algorithm MOEA-FHUI. One of my problems is to get the initial population P with the proposed problem-specific initialize strategy. Is it possible to provide ...
0
votes
1answer
89 views

Algorithm to find the attributes that comprise the greatest concentration

I have a porfolio of mortgage loans where each loan has a number of attributes attr1, attr2, .., attrN. I would like to analyze the portfolio credit risk concentration (see below) using these ...
0
votes
1answer
150 views

Calculating lift

I was reading about lift for frequent itemsets and came across this exercise here which was taken from a data mining textbook. I tried plugging in the numbers, but could not get the 0.89 value for ...
3
votes
1answer
233 views

Too many dummy variable in regression model

we have about 50000 models of mobile phone (like Galaxy S7, iPhone 9) in database and the size of data is about 3 million. We want to find the mobile phones that have the least call success rate ( ...
0
votes
1answer
25 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
5
votes
1answer
148 views

Functional Data Analysis and Splines Regression

I'm new to Functional Data Analysis so my question will'be very simple for an expert in this topic. Operatively, when I fit a model like this, in R : model<- lm( y ~ bs( x, df = k, degree = l ) )...
0
votes
1answer
39 views

Statistical time series analysis using Percentile and Empirical Rule

I analysing time-series to see the range of an asset. I have data for GBPUSD so i could only analyze this so far. I have used both empirical rule and percentile. Of course they give different ...
0
votes
0answers
6 views

Supervised ML opinion mining of medical entities: tools?

I would like to extract relations between the entities of 'quality of life' and 'health interventions' out of a corpus of medical texts via supervised machine learning. I wonder whether there is a ...
1
vote
0answers
20 views

Log mining machine learning approach

I am looking for guidance on a log mining problem I am encountering. My dataset comprises of lines like this with low veracity: I am trying to predict whether or not there is going to be a delay on ...
0
votes
0answers
59 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
0answers
5 views

Combining 2 different spatial datasets

I've tried to keep the formulation as general as possible. Problem: For a given location (e.g. a county, country), I have 2 different datasets (D1, D2) where the rows are of the form D1: ...
-2
votes
1answer
78 views

Why isn't Matlab's Classification Learner the End All? [closed]

After building some machine learning models in Python, R and Matlab, I found the latter's Classification Learner App to be immensely powerful. In the time it took me to build a single model in Python ...
1
vote
2answers
30 views

What are possible approaches for learning causal DAG of events?

I have historical data of event logs. Each event has an associated contextual Id, which can be used to tell that event A happened first in some context, then event B happened in same context and then ...
0
votes
1answer
84 views

Violating “compactness” in single linkage hierarchical agglomerative clustering

While I was studying Hierarchical Agglomerative Clustering in the book Elements of Statistical Learning in the chapter Unsupervised Learning, I came through the following : The statement The ...
1
vote
1answer
45 views

Clustering similar user reviews

I am using Amazons Video Games dataset ( 1.3 mil user reviews ) in order to make a recommendation system using LSTM networks. What I would like to do is to cluster ...
0
votes
1answer
64 views
0
votes
0answers
28 views

Formulating an Anomaly Detection method for a dataset based on time

I have the following data (temporal feature) for n videos of given length: start times of all the users who watched the video duration of the video viewed Now, given a video (given the start times ...
3
votes
1answer
483 views

When is weighted average of $F_1$ scores $\simeq$ accuracy in classification?

Example where accuracy $\simeq$ weighted average of $F_1$ scores I have a classifier which classifies between 2 classes, $\mathbf{A}$ and $\mathbf{B}$. Let's say that we have the confusion matrix ...
0
votes
1answer
167 views

classification With ordinal data in R ?

I have a survey data which is Likert scale ordinal data . I want ordinal classification with R . I try several data mining algorithm with this data set but my accuracy is too low . How can i improve ...
1
vote
0answers
422 views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
2
votes
0answers
38 views

Decision tree ,information gain and overfitting

If i use the information gain in order to evaluate the best split in a decision tree, why using a binomial split reduces the risk of overfitting ? Is the information gain test misleading if we have a ...
0
votes
0answers
12 views

What are the characteristics of a good cluster?

I am analysing a data set using Clustering Algorithm in SAS Enterprise Miner. I have tried to change various settings to get the best results. But,I dont really know how to decide whether the model ...
1
vote
0answers
94 views

Simplistic Write-Up of XGBoost Algorithm

I was wondering whether anybody could check through my very simplistic (e.g., no detail of greedy split finding algorithm) write-up of the XGBoost algorithm for binary classification. Algorithm: ...
0
votes
0answers
270 views

Where could I find pseudocode for XGBoost?

I was wondering whether anybody knew where I could find pseudocode for the XGBoost algorithm?
2
votes
1answer
3k views

What is the “binary:logistic” objective function in XGBoost?

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised. $$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + ...
2
votes
2answers
369 views

How to scale predictions from a neural network in R when the output is not a part of the dataset

I've been using a neural network to make predictions. So my training data is in one .csv file which I read-in and then scale. My test data is in another file that I read-in and is also scaled. However,...
0
votes
0answers
244 views

Why the contour of lasso and ridge regression are drawn only at that position and how do they draw like that?

Why all of the contour were drawn only at this kind of position and how do this contour drawn. Thanks
0
votes
0answers
17 views

Discretize numeric attribute values

I have a numeric attribute (e.g. revenue attribute), I must discretize its values before I use WEKA tool, but based on what I must divide them into categories? Since this may affect the classification ...
2
votes
0answers
51 views

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? [closed]

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? Because upon trying to do data mining on such data set, even basic PCA in sklearn runs into ...
3
votes
1answer
90 views

How to make a regression plot more insightful and engaging to an end user

I have an interview soon for a graduate position. I will to do 10-15 presentation. So, I was sent out a regression plot by the recruiter and I have to present how I can make the regression plot more ...
0
votes
1answer
90 views

Does it make sense to apply feature selection on a data set that has undergone principle component analysis?

I have a data set that consists of credit card transactions. This data set contains only numerical values which are the result of principle component analysis. I'm currently trying improve my models ...
1
vote
0answers
100 views

What is a good percentage value for SMOTE in Weka

I am using the oversampling technique named as SMOTE to balance my dataset in Weka. To balance my dataset I needed to use 1300 as my -p (percentage) value. The default -p (percentage) value is 100. ...
0
votes
0answers
36 views

Using LASSO to do the feature selection [duplicate]

When we do a LASSO feature selection, do we have to standardize each feature?
4
votes
1answer
1k views

Mix of text and numeric data

I have to train a classification model with 15 classes based on data which contains both textual and numeric data. For instance: product description(textual), product length'(numeric). I have ...
1
vote
0answers
278 views

Anomaly detection across repeated time-series with time delay

I am considering repeated time series (for example on day 1, day 2, ..., day(n)), and I want to find the anomalous points in the day(n+1) given that some delay between samples across different days is ...
0
votes
0answers
39 views

Best method for clustering product arrival dates

I work for a retailer, and I am trying to analyze the performance of each "collection". It is not an attribute we have formally defined in our systems, so I am trying to create it via a Clustering ...
0
votes
0answers
20 views

Determine whether feature discriminates between two classes

I have a large dataset of documents for which I have calculated the normalized occurrence rate of each word (i.e., normalized term-frequency) and have a label. Each document is labeled as either "HIGH"...
0
votes
0answers
30 views

am I doing the sampling techniques correctly and how would I analyse different algorithms?

I'm doing an undergrad project in which I compare the performance of various classification algorithms which are logistic regression, decision trees, and support vector machines. The data set I chose ...
2
votes
0answers
46 views

Is Multiple Correspondence Analysis applicable to Multi-valued Categorical Variables?

I have a data-set containing only Categorical Variables. I needed to do Principal Component Analysis on the data set. Eventually, I found Multiple Correspondence Analysis and learnt it. But, in MCA, ...