Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

0
votes
0answers
8 views

Comparing two Models gained from process discovery

I have a rather basic question on process mining that I however hardly get answered: currently, I do some research on regulated learning in educational/psychological research: I already drew up two ...
0
votes
1answer
15 views

Random Forest and preprocessing in Data Mining

When applying the Random Forest classification technique, do we have to do preprocessing or is it true that it is not needed for Random Forest ?
0
votes
0answers
12 views

Algorithm MOEA-FHUI

I have a problem in some of the concepts of algorithm MOEA-FHUI. One of my problems is to get the initial population P with the proposed problem-specific initialize strategy. Is it possible to provide ...
0
votes
0answers
27 views

Best method for analysing survey data

I have a data set of 10 categorigal variables and one quantitative variable describing the population plus 1 score or measure from 10 seperate experts. Variables 1 to 5 describe a customer profile e....
0
votes
1answer
82 views

Algorithm to find the attributes that comprise the greatest concentration

I have a porfolio of mortgage loans where each loan has a number of attributes attr1, attr2, .., attrN. I would like to analyze the portfolio credit risk concentration (see below) using these ...
0
votes
0answers
18 views

SVM Regression n R

Hi The following graph represents original data, linear regression and Support vector regression. I would like to know if this is a decent plot of SVR and not understanding how to predict for a new ...
0
votes
1answer
83 views

Calculating lift

I was reading about lift for frequent itemsets and came across this exercise here which was taken from a data mining textbook. I tried plugging in the numbers, but could not get the 0.89 value for ...
0
votes
0answers
15 views

Identifying patterns

We have logs coming from a system that indicates the number of items bought in a particular transaction and the classes that are involved in handling this took how much time in each system. From the ...
3
votes
1answer
136 views

Too many dummy variable in regression model

we have about 50000 models of mobile phone (like Galaxy S7, iPhone 9) in database and the size of data is about 3 million. We want to find the mobile phones that have the least call success rate ( ...
0
votes
1answer
22 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
5
votes
1answer
110 views

Functional Data Analysis and Splines Regression

I'm new to Functional Data Analysis so my question will'be very simple for an expert in this topic. Operatively, when I fit a model like this, in R : model<- lm( y ~ bs( x, df = k, degree = l ) )...
0
votes
1answer
27 views

Statistical time series analysis using Percentile and Empirical Rule

I analysing time-series to see the range of an asset. I have data for GBPUSD so i could only analyze this so far. I have used both empirical rule and percentile. Of course they give different ...
0
votes
0answers
4 views

Supervised ML opinion mining of medical entities: tools?

I would like to extract relations between the entities of 'quality of life' and 'health interventions' out of a corpus of medical texts via supervised machine learning. I wonder whether there is a ...
1
vote
0answers
19 views

Log mining machine learning approach

I am looking for guidance on a log mining problem I am encountering. My dataset comprises of lines like this with low veracity: I am trying to predict whether or not there is going to be a delay on ...
0
votes
0answers
37 views

Data Mining and Multiple Testing

I have a dataset A which consist of 100000 data points. I am trying to find hidden relations in the data. I am following the steps below: Step 1. I partition data in 5 different random sets (20000 ...
0
votes
0answers
22 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
0answers
5 views

Combining 2 different spatial datasets

I've tried to keep the formulation as general as possible. Problem: For a given location (e.g. a county, country), I have 2 different datasets (D1, D2) where the rows are of the form D1: ...
-2
votes
1answer
65 views

Why isn't Matlab's Classification Learner the End All? [closed]

After building some machine learning models in Python, R and Matlab, I found the latter's Classification Learner App to be immensely powerful. In the time it took me to build a single model in Python ...
0
votes
2answers
25 views

What are possible approaches for learning causal DAG of events?

I have historical data of event logs. Each event has an associated contextual Id, which can be used to tell that event A happened first in some context, then event B happened in same context and then ...
0
votes
1answer
83 views

Violating “compactness” in single linkage hierarchical agglomerative clustering

While I was studying Hierarchical Agglomerative Clustering in the book Elements of Statistical Learning in the chapter Unsupervised Learning, I came through the following : The statement The ...
1
vote
1answer
39 views

Clustering similar user reviews

I am using Amazons Video Games dataset ( 1.3 mil user reviews ) in order to make a recommendation system using LSTM networks. What I would like to do is to cluster ...
0
votes
1answer
52 views
0
votes
0answers
24 views

Formulating an Anomaly Detection method for a dataset based on time

I have the following data (temporal feature) for n videos of given length: start times of all the users who watched the video duration of the video viewed Now, given a video (given the start times ...
3
votes
1answer
323 views

When is weighted average of $F_1$ scores $\simeq$ accuracy in classification?

Example where accuracy $\simeq$ weighted average of $F_1$ scores I have a classifier which classifies between 2 classes, $\mathbf{A}$ and $\mathbf{B}$. Let's say that we have the confusion matrix ...
0
votes
1answer
110 views

classification With ordinal data in R ?

I have a survey data which is Likert scale ordinal data . I want ordinal classification with R . I try several data mining algorithm with this data set but my accuracy is too low . How can i improve ...
1
vote
0answers
316 views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
2
votes
0answers
31 views

Decision tree ,information gain and overfitting

If i use the information gain in order to evaluate the best split in a decision tree, why using a binomial split reduces the risk of overfitting ? Is the information gain test misleading if we have a ...
0
votes
0answers
12 views

What are the characteristics of a good cluster?

I am analysing a data set using Clustering Algorithm in SAS Enterprise Miner. I have tried to change various settings to get the best results. But,I dont really know how to decide whether the model ...
1
vote
0answers
74 views

Simplistic Write-Up of XGBoost Algorithm

I was wondering whether anybody could check through my very simplistic (e.g., no detail of greedy split finding algorithm) write-up of the XGBoost algorithm for binary classification. Algorithm: ...
0
votes
0answers
208 views

Where could I find pseudocode for XGBoost?

I was wondering whether anybody knew where I could find pseudocode for the XGBoost algorithm?
1
vote
1answer
2k views

What is the “binary:logistic” objective function in XGBoost?

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised. $$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + ...
2
votes
2answers
225 views

How to scale predictions from a neural network in R when the output is not a part of the dataset

I've been using a neural network to make predictions. So my training data is in one .csv file which I read-in and then scale. My test data is in another file that I read-in and is also scaled. However,...
0
votes
0answers
182 views

Why the contour of lasso and ridge regression are drawn only at that position and how do they draw like that?

Why all of the contour were drawn only at this kind of position and how do this contour drawn. Thanks
0
votes
0answers
16 views

Discretize numeric attribute values

I have a numeric attribute (e.g. revenue attribute), I must discretize its values before I use WEKA tool, but based on what I must divide them into categories? Since this may affect the classification ...
2
votes
0answers
51 views

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? [closed]

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? Because upon trying to do data mining on such data set, even basic PCA in sklearn runs into ...
3
votes
1answer
87 views

How to make a regression plot more insightful and engaging to an end user

I have an interview soon for a graduate position. I will to do 10-15 presentation. So, I was sent out a regression plot by the recruiter and I have to present how I can make the regression plot more ...
0
votes
1answer
79 views

Does it make sense to apply feature selection on a data set that has undergone principle component analysis?

I have a data set that consists of credit card transactions. This data set contains only numerical values which are the result of principle component analysis. I'm currently trying improve my models ...
1
vote
0answers
92 views

What is a good percentage value for SMOTE in Weka

I am using the oversampling technique named as SMOTE to balance my dataset in Weka. To balance my dataset I needed to use 1300 as my -p (percentage) value. The default -p (percentage) value is 100. ...
0
votes
0answers
35 views

Using LASSO to do the feature selection [duplicate]

When we do a LASSO feature selection, do we have to standardize each feature?
3
votes
1answer
812 views

Mix of text and numeric data

I have to train a classification model with 15 classes based on data which contains both textual and numeric data. For instance: product description(textual), product length'(numeric). I have ...
1
vote
1answer
238 views

Anomaly detection across repeated time-series with time delay

I am considering repeated time series (for example on day 1, day 2, ..., day(n)), and I want to find the anomalous points in the day(n+1) given that some delay between samples across different days is ...
0
votes
0answers
38 views

Best method for clustering product arrival dates

I work for a retailer, and I am trying to analyze the performance of each "collection". It is not an attribute we have formally defined in our systems, so I am trying to create it via a Clustering ...
0
votes
0answers
18 views

Determine whether feature discriminates between two classes

I have a large dataset of documents for which I have calculated the normalized occurrence rate of each word (i.e., normalized term-frequency) and have a label. Each document is labeled as either "HIGH"...
0
votes
0answers
30 views

am I doing the sampling techniques correctly and how would I analyse different algorithms?

I'm doing an undergrad project in which I compare the performance of various classification algorithms which are logistic regression, decision trees, and support vector machines. The data set I chose ...
2
votes
0answers
32 views

Is Multiple Correspondence Analysis applicable to Multi-valued Categorical Variables?

I have a data-set containing only Categorical Variables. I needed to do Principal Component Analysis on the data set. Eventually, I found Multiple Correspondence Analysis and learnt it. But, in MCA, ...
1
vote
1answer
68 views

Regression with more features than samples

I am new to the subject and I encountered this question in my text book: Suppose that the number of features is greater than the number of samples. e.g: $𝑦_1 = b_1𝑋_{11} + b_2𝑋_{12} + b_3𝑋_{13}$ ...
0
votes
0answers
5 views

grouping and categorizing differently spaced out events

I have data of client's visits to a food shelter. For each day, over 10 years, there is a 1 if the person went to the pantry and 0 if they didn't. I'm trying to group people whose pattern of use is ...
1
vote
0answers
40 views

Comparing clustering stability over period of time [duplicate]

Suppose I am applying K-means clustering on two datasets generated by same process. How to compare clustering stability over a period of time? Let's say I have applied clustering in 2016 on a data set,...
0
votes
0answers
152 views

scores and loadings in princomp (PCA)

I applied PCA on the dataset, but I am confused what should I use to build the model, the components in scores or in the loadings? @ usεr11852 ...
1
vote
0answers
67 views

use test data set after 10 Cross-validation

I applied 10 Cross-validation but I am a bit confused, I am not sure what is a correct way. 1- Should I apply 10 Cross-validation on all dataset, divide it into 10 folds and sum all the 10 matrices ...