Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

0
votes
1answer
17 views

Why does creating training sets with replacement lead to better performance?

Pasting generally suffers from lower performance than bagging, because it training sets without replacement. Why does creating training sets with replacement lead to better performance?
2
votes
0answers
36 views

Functional Data Analysis and Splines Regression

I'm new to Functional Data Analysis so my question will'be very simple for an expert in this topic. Operatively, when I fit a model like this, in R : model<- lm( y ~ bs( x, df = k, degree = l ) )...
0
votes
1answer
24 views

Statistical time series analysis using Percentile and Empirical Rule

I analysing time-series to see the range of an asset. I have data for GBPUSD so i could only analyze this so far. I have used both empirical rule and percentile. Of course they give different ...
0
votes
0answers
4 views

Supervised ML opinion mining of medical entities: tools?

I would like to extract relations between the entities of 'quality of life' and 'health interventions' out of a corpus of medical texts via supervised machine learning. I wonder whether there is a ...
1
vote
0answers
13 views

Log mining machine learning approach

I am looking for guidance on a log mining problem I am encountering. My dataset comprises of lines like this with low veracity: I am trying to predict whether or not there is going to be a delay on ...
0
votes
0answers
31 views

Data Mining and Multiple Testing

I have a dataset A which consist of 100000 data points. I am trying to find hidden relations in the data. I am following the steps below: Step 1. I partition data in 5 different random sets (20000 ...
0
votes
0answers
10 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
0answers
5 views

Combining 2 different spatial datasets

I've tried to keep the formulation as general as possible. Problem: For a given location (e.g. a county, country), I have 2 different datasets (D1, D2) where the rows are of the form D1: ...
-2
votes
1answer
52 views

Why isn't Matlab's Classification Learner the End All? [closed]

After building some machine learning models in Python, R and Matlab, I found the latter's Classification Learner App to be immensely powerful. In the time it took me to build a single model in Python ...
0
votes
1answer
16 views

What are possible approaches for learning causal DAG of events?

I have historical data of event logs. Each event has an associated contextual Id, which can be used to tell that event A happened first in some context, then event B happened in same context and then ...
0
votes
1answer
28 views

Violating “compactness” in single linkage hierarchical agglomerative clustering

While I was studying Hierarchical Agglomerative Clustering in the book Elements of Statistical Learning in the chapter Unsupervised Learning, I came through the following : The statement The ...
1
vote
1answer
36 views

Clustering similar user reviews

I am using Amazons Video Games dataset ( 1.3 mil user reviews ) in order to make a recommendation system using LSTM networks. What I would like to do is to cluster ...
0
votes
1answer
28 views
0
votes
0answers
21 views

Formulating an Anomaly Detection method for a dataset based on time

I have the following data (temporal feature) for n videos of given length: start times of all the users who watched the video duration of the video viewed Now, given a video (given the start times ...
3
votes
1answer
69 views

When is weighted average of $F_1$ scores $\simeq$ accuracy in classification?

Example where accuracy $\simeq$ weighted average of $F_1$ scores I have a classifier which classifies between 2 classes, $\mathbf{A}$ and $\mathbf{B}$. Let's say that we have the confusion matrix ...
0
votes
1answer
51 views

classification With ordinal data in R ?

I have a survey data which is Likert scale ordinal data . I want ordinal classification with R . I try several data mining algorithm with this data set but my accuracy is too low . How can i improve ...
1
vote
0answers
79 views

Information Gain vs Gain Ratio

In the building of a decision tree, when it's better to prefer the information gain criterion to the gain ratio criterion ? And why ?
2
votes
0answers
24 views

Decision tree ,information gain and overfitting

If i use the information gain in order to evaluate the best split in a decision tree, why using a binomial split reduces the risk of overfitting ? Is the information gain test misleading if we have a ...
0
votes
0answers
11 views

What are the characteristics of a good cluster?

I am analysing a data set using Clustering Algorithm in SAS Enterprise Miner. I have tried to change various settings to get the best results. But,I dont really know how to decide whether the model ...
1
vote
0answers
45 views

Simplistic Write-Up of XGBoost Algorithm

I was wondering whether anybody could check through my very simplistic (e.g., no detail of greedy split finding algorithm) write-up of the XGBoost algorithm for binary classification. Algorithm: ...
0
votes
0answers
116 views

Where could I find pseudocode for XGBoost?

I was wondering whether anybody knew where I could find pseudocode for the XGBoost algorithm?
1
vote
1answer
480 views

What is the “binary:logistic” objective function in XGBoost?

I am reading through Chen's XGBoost paper. He writes that during the $\text{t}^{\text{th}}$ iteration, the objective function below is minimised. $$ L^{(t)} = \sum_{i}^n l(y_i, \hat{y}_i^{(t-1)} + ...
2
votes
2answers
73 views

How to scale predictions from a neural network in R when the output is not a part of the dataset

I've been using a neural network to make predictions. So my training data is in one .csv file which I read-in and then scale. My test data is in another file that I read-in and is also scaled. However,...
1
vote
0answers
51 views

Why the contour of lasso and ridge regression are drawn only at that position and how do they draw like that?

Why all of the contour were drawn only at this kind of position and how do this contour drawn. Thanks
0
votes
0answers
15 views

Discretize numeric attribute values

I have a numeric attribute (e.g. revenue attribute), I must discretize its values before I use WEKA tool, but based on what I must divide them into categories? Since this may affect the classification ...
2
votes
0answers
51 views

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? [closed]

What data mining methods can I use for a large (800x100000) matrix that don't require a super computer? Because upon trying to do data mining on such data set, even basic PCA in sklearn runs into ...
3
votes
1answer
83 views

How to make a regression plot more insightful and engaging to an end user

I have an interview soon for a graduate position. I will to do 10-15 presentation. So, I was sent out a regression plot by the recruiter and I have to present how I can make the regression plot more ...
0
votes
1answer
50 views

Does it make sense to apply feature selection on a data set that has undergone principle component analysis?

I have a data set that consists of credit card transactions. This data set contains only numerical values which are the result of principle component analysis. I'm currently trying improve my models ...
1
vote
0answers
48 views

What is a good percentage value for SMOTE in Weka

I am using the oversampling technique named as SMOTE to balance my dataset in Weka. To balance my dataset I needed to use 1300 as my -p (percentage) value. The default -p (percentage) value is 100. ...
0
votes
0answers
32 views

Using LASSO to do the feature selection [duplicate]

When we do a LASSO feature selection, do we have to standardize each feature?
2
votes
1answer
247 views

Mix of text and numeric data

I have to train a classification model with 15 classes based on data which contains both textual and numeric data. For instance: product description(textual), product length'(numeric). I have ...
1
vote
1answer
199 views

Anomaly detection across repeated time-series with time delay

I am considering repeated time series (for example on day 1, day 2, ..., day(n)), and I want to find the anomalous points in the day(n+1) given that some delay between samples across different days is ...
0
votes
0answers
30 views

Best method for clustering product arrival dates

I work for a retailer, and I am trying to analyze the performance of each "collection". It is not an attribute we have formally defined in our systems, so I am trying to create it via a Clustering ...
0
votes
0answers
16 views

Determine whether feature discriminates between two classes

I have a large dataset of documents for which I have calculated the normalized occurrence rate of each word (i.e., normalized term-frequency) and have a label. Each document is labeled as either "HIGH"...
0
votes
0answers
24 views

am I doing the sampling techniques correctly and how would I analyse different algorithms?

I'm doing an undergrad project in which I compare the performance of various classification algorithms which are logistic regression, decision trees, and support vector machines. The data set I chose ...
2
votes
0answers
19 views

Is Multiple Correspondence Analysis applicable to Multi-valued Categorical Variables?

I have a data-set containing only Categorical Variables. I needed to do Principal Component Analysis on the data set. Eventually, I found Multiple Correspondence Analysis and learnt it. But, in MCA, ...
1
vote
1answer
49 views

Regression with more features than samples

I am new to the subject and I encountered this question in my text book: Suppose that the number of features is greater than the number of samples. e.g: $𝑦_1 = b_1𝑋_{11} + b_2𝑋_{12} + b_3𝑋_{13}$ ...
0
votes
0answers
5 views

grouping and categorizing differently spaced out events

I have data of client's visits to a food shelter. For each day, over 10 years, there is a 1 if the person went to the pantry and 0 if they didn't. I'm trying to group people whose pattern of use is ...
1
vote
0answers
32 views

Comparing clustering stability over period of time [duplicate]

Suppose I am applying K-means clustering on two datasets generated by same process. How to compare clustering stability over a period of time? Let's say I have applied clustering in 2016 on a data set,...
0
votes
0answers
87 views

scores and loadings in princomp (PCA)

I applied PCA on the dataset, but I am confused what should I use to build the model, the components in scores or in the loadings? @ usεr11852 ...
1
vote
0answers
39 views

use test data set after 10 Cross-validation

I applied 10 Cross-validation but I am a bit confused, I am not sure what is a correct way. 1- Should I apply 10 Cross-validation on all dataset, divide it into 10 folds and sum all the 10 matrices ...
0
votes
0answers
118 views

How to calculate probability of Bayes network?

Hi I'm new into Bayes network I'm trying to calculate P(F&G) but don't have the slightest clue how because everything i find online doesn't have a network that is this complex. I will really ...
0
votes
0answers
15 views

Difference in definition of support for frequent itemsets and association rules

My professor at university made the statement, that there's a difference between the definition of "support" for FISs and ARs, because "when looking for FIS, support relates to the set of products ...
0
votes
1answer
103 views

AdaBoost algorithm question

In the boosting algorithm,AdaBoost ,those observations which were misclassified by the classifier in the (m-1)th step have their weights increased in the mth step, and those which were correctly ...
0
votes
1answer
38 views

Is this an unsupervised pattern recognition problem?

Consider a feature set containing values of operating condition of an automation unit such as pressure of the valve, temperature, fuel consumption. I want to discover unknown relationship between ...
0
votes
0answers
18 views

Standardization of cyclical features for Clustering

I am about to start a project in data mining sales data using the well-known algorithm, t-SNE. Specifically, I have 4 millions of observations and 75 attributes for every sale. A graphical example ...
0
votes
1answer
177 views

K nearest neighbours model complexity

My question is about the 1-nearest neighbor classifier and is about a statement made in the excellent book The Elements of Statistical Learning, by Hastie, Tibshirani and Friedman. The statement is "...
0
votes
1answer
87 views

Should text pre-processing come before or after POS-tagging?

I am currently working on an assignment to produce a sentiment classifier for Twitter. One of the initial, and most important, stages in NLP classification is text pre-processing. For Twitter, this ...
0
votes
1answer
45 views

Technique to find spurious correlations among huge number of time series datasets?

I came across this tongue-in-cheek website that lists lots of spurious correlations. It's not lost on me that the author's main point is to discourage the brute-force-search of correlations, but his ...
0
votes
0answers
45 views

What is information gain ratio?

With respect to data mining, what is information gain ratio? I'm a complete beginner to data analytics and mining, so please explain at a low level of understanding.