Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
1 vote
0 answers
8 views

For the given data find the clusters. Assume the relevant parameters needed [closed]

Below is the given data, how can I make clusters using symmetric matrix?
user avatar
0 votes
0 answers
29 views

Explainable clustering for broker churn

I have a dataset like as below You can see that my dataset is longitudinal. There will be multiple records for the same broker with multiple or same customer, selling same or different products. Now, ...
user avatar
  • 1,726
-1 votes
1 answer
25 views

why random model response for X decile is fixed at X%?

I was reading online tutorials on lift and gain charts here, here, here In all of these tutorials, I read or see that random model curve is drawn with the expectation at each Xth decile, we get X% ...
user avatar
  • 1,726
3 votes
2 answers
245 views

A universal measure of the accuracy of linear regression models

I have a dataset that contains both outliers and multicollinearity. I applied three different regression models to that dataset: ordinary least square, absolute linear regression, and Huber regression....
user avatar
  • 1,487
0 votes
0 answers
26 views

Creating a binary attribute for each of the M nominal states

I am preparing for an exam in data mining and need help with the following question: Assume we have categorical data. One method to define a distance between two data objects is (p-m)/p where p is ...
user avatar
  • 81
1 vote
0 answers
31 views

Dealing with class imbalance and data complexity issues

I am doing a classification task for a 5-class imbalanced dataset. Class distribution shows 2-majority & 2-minority clasess, as far: ...
user avatar
  • 231
2 votes
1 answer
49 views

Classification methods with only nominal attributes

I would like to know what classification methods can naturally deal with only nominal attributes. An example is decision trees created by C4.5. I imagine that other classifiers based on decision trees ...
user avatar
1 vote
0 answers
44 views

How do non binary decision trees deal with categorical values that weren't in training?

I've been implementing Random Forest from scratch as a learning exercise. While most algorithms for decision trees seem to deal exclusively in binary yes/no questions, leading to binary trees, ...
user avatar
  • 11
1 vote
1 answer
40 views

Class separability and Overlap

I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average. This is a classification task and ...
user avatar
  • 215
0 votes
0 answers
26 views

Height limits relation to anomaly score in Isolation Forest

I am trying to implement the Isolation Forest algorithm in Python and faced an issue when dealing with the max_depth and the height limit (l) from the white paper. (See the 2. set height limit l) ...
user avatar
0 votes
0 answers
10 views

time series residual, seasonal and trend

I'm trying to understand time series Series = trend + seasonal + residual I read that centred moving average of period 'n' eliminated seasonal of period 'n' then CMA = residual + trend and I also read ...
user avatar
0 votes
0 answers
58 views

Local Outlier Factor (LOF) in Financial Time Series

I'm using Local Outlier Factor (LOF) in financial data with 40 features. When I use the algorithm I can achieve scores outliers, but I can't understand how I can get my algo to tell me the connection ...
user avatar
0 votes
0 answers
21 views

RMSE increases in test data while correlation also increases

I use deep networks to predict stock future return, and find that in the training data, the RMSE decreases and the correlation between label and the predection increases, which is normal. However, ...
user avatar
  • 111
0 votes
0 answers
10 views

Is it the correct practice to remove highly correlated features before association rules mining?

I want to use association rules mining on a dataset (Apriori algorithm or FP growth algorithm). Although my data is not market basket analysis, I'm treating each row similar to a market basket, with ...
user avatar
1 vote
1 answer
82 views

Statistical modeling vs data mining [duplicate]

What is the difference between "statistical modeling" and "data mining"? I have searched the internet, but I can't see it clearly. Is there any overlap? Can they be considered ...
user avatar
1 vote
0 answers
11 views

Data mining for gene dataset with pairwise comparison [closed]

I have a table of sample a b c's gene data pairwise comparison, but I really don't know how to start data mining for this specific dataset. I wonder if anyone can come up with some idea, maybe in some ...
user avatar
  • 11
0 votes
0 answers
15 views

How to get two variable relationship using statistical model?

How to get two variable non-linear relationships using statistical model with other variables fixed? for example, how to analysis the relationship between customer churn and number of message the ...
user avatar
0 votes
0 answers
21 views

Is linear regression appropriate for this analysis ? what additional operations to do?

I haven't done statistics / econometrics for a long time. I haven't done a lot by the way. But I would like advice on some methods and tests to be performed on two excel files that I have. Each excel ...
user avatar
1 vote
1 answer
297 views

How To Determine Number Of Clusters In T-SNE And Best Clustering Algorithm?

I used TSNE method to cluster my DataSet. ...
user avatar
4 votes
1 answer
100 views

What is the purpose of conducting Simple Random Sampling WITH Replacement?

Part of data preparation is simple random sampling. Random sampling can be of two forms with replacement or without replacement. With replacement, subset sampling simply might contain duplicates of ...
user avatar
1 vote
0 answers
24 views

How autocorrelation work based on the data plot?

Let's suppose we have a time series is a=[1,1,1,3,3,3,1,1,1,3,3,3] as then the autocorrelation figure for this time series is The lag here is 4, for lag = 1, the autocorrelation is between [1,1,1,3,...
user avatar
  • 113
1 vote
1 answer
90 views

What does the meaning of the autocorrelation in this picture?

From this figure, how should I understand what is the lag on the top figure? and when in the bottom figure for example the autocorrelation is 0.45 what does tell us about the above figure? Another ...
user avatar
  • 113
0 votes
0 answers
20 views

Can one feature included in more than one PCA in PCA analysis

These are the eighen values and the cumulative I have got with PCA analysis, but I'm struggling with determining the features related to each PCA. This is the correlation matrix plot I have got but ...
user avatar
  • 1
1 vote
0 answers
31 views

Applying ML to Diagnostic Analytics on Financial Report

Background: We (a team in the IT department) are currently helping the financial team solve their problem, by automating their tasks. The task is this: they review the financial report monthly and ...
user avatar
0 votes
0 answers
17 views

Can someone give me an example of frequent large subgraphs being outlier?

I read it from a lecture note: ...
user avatar
  • 101
3 votes
2 answers
159 views

What is the difference between p-hacking and data mining bias?

From my understanding, data mining bias occurs when someone repeatedly searches through a data set to find statistically significant results. How is this any different from p-hacking? What is the ...
user avatar
  • 145
1 vote
0 answers
24 views

How to compare the peformance of different clusterings without true labels

Firstly, I know some scores like silhouette score and Davies–Bouldin score to compare the performance in one clustering method. However, I am not sure how to compare results in different clustering ...
user avatar
  • 111
1 vote
1 answer
26 views

How to Find out the Best Way to Encode Data in ML?

I have been thinking in this problem for quite a while, I cannot figure out a way of knowing, (based on the task) what will be the best data encoding I can do for training the model. Imagine I have a ...
user avatar
  • 111
0 votes
0 answers
18 views

How to best visualize time interval difference and compute a single measure

I have a table like as shown below ...
user avatar
  • 1,726
2 votes
1 answer
27 views

Visualizing shared instances of p-values<alpha across large numbers of treatments

Assume a data table that presents the p-values of a large number of independent runs of a statistical hypothesis test. Each run represents a single test with two possible hypotheses (i.e., null and ...
user avatar
2 votes
1 answer
35 views

How to make a decision - when there is a tie and no human expert

We have two algorithms (simple rule-based) working on labeling the dataset as "Yes" and "No" for a disease. There is no ML involved in this task. For ex: If Algo 1 says subject 1 ...
user avatar
  • 1,726
0 votes
1 answer
51 views

Find all possible clusterizations

I need help to find all possible clusterizations via the k-means method in Python. Let's assume for simplicity that I have the following table: height | weight | country of origin (X/Y/Z) | flag (1/0) ...
user avatar
  • 121
0 votes
1 answer
236 views

the ratio of validation set and test set should be equal?

I always heard that the common ratio of the train:validation:test is 70:15:15 or 80:10:10 or 60:20:20, sounds like the validation set and test set should be equal size. Assuming that I wanna use 5 ...
user avatar
0 votes
0 answers
24 views

Machine Learning for choosing the right cover box to contain smaller item boxes

I am studying on choosing the right box to contain item boxes. When customers order items, items have their own cases and ordered items are packed with the right cover box to contain them to deliver. ...
user avatar
  • 2,585
0 votes
0 answers
40 views

How can I do one class learning for outlier detection?

I understand I can use various sampling techniques when dealing with imbalanced datasets. However, I wonder how I can build a classification model from the training dataset only including data that ...
user avatar
  • 2,585
2 votes
1 answer
73 views

Why does my model produce unrealistic output?

I am trying to run a binary classification problem on people with diabetes and non-diabetes. For labeling my datasets, I followed a simple rule. If a person has T2DM...
user avatar
  • 1,726
1 vote
0 answers
29 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
user avatar
1 vote
0 answers
22 views

A space of functions and their Fourier Transforms?

Conjugate variables and the Fourier transform are often used to analyze different states of a single object. For example in Quantum Mechanics it can be used to describe changing information about ...
user avatar
2 votes
2 answers
75 views

Understanding the determination of principal components

The idea of PCA is to find the directions (in high dimensional space) in which the essential structures (with regard to large variance, scatter) of the data lie. The assumption is that original ...
user avatar
  • 165
0 votes
1 answer
63 views

How can i cite Sequential Forward Feature Selection (SFFS)?

I've seen many papers/books about this technique but none cite its author. Is it ok to cite any machine learning theory book that explains it? Thanks.
user avatar
  • 133
0 votes
1 answer
48 views

What is the threshold for "frequent" in the Apriori algorithm?

I'm reading about the Apriori algorithm using the textbook Introduction to Machine Learning (Ethem Alpaydin) and had a question. I've noticed that the textbook and many other resources I find online ...
user avatar
  • 2,331
0 votes
1 answer
84 views

Using my models with NGBoost?

I've come across this new tool of NGBoost from the Machine Learning group of Stanford, I was curious if peopel have started using it yet. They say that one can have a Base learner such as a regression ...
user avatar
1 vote
1 answer
94 views

Is it appropriate to use regularised regression for low-dimensional N>>p variable selection problems?

I am currently examining which of sixteen variables are the most important in predicting a binary outcome. There are 907 observations, so obviously $N$ is much larger than $p$ In the last six months ...
user avatar
  • 1,451
1 vote
0 answers
15 views

Neural network for generating interesting sets

I'm facing the following problem: There are $n$ items $U=\{i_1,...,1_n\}$. I have many examples $X$ where $X\subseteq P[U]$ (the power set of $U$). The goal is to create a system that given these ...
user avatar
  • 111
0 votes
1 answer
42 views

Need help finding the right vocabulary so I can go research more

Looking for some vocabulary to help me refine my research so I can tackle this problem. Here's an overview of the problem statement I'm working on. At my company we manufacture various products, ...
user avatar
1 vote
0 answers
8 views

How the pattern ranking performed in this SDM paper?

I am following this SDM paper "Diversified Trajectory Pattern Ranking in Geo-Tagged Social Media" that I found very intersting and inspiring. However due to my limited knowledge in mathematical ...
user avatar
  • 592
1 vote
0 answers
63 views

How to compare the behaviour of different pretrained RL models?

I have a number of pretrained RL models (PPO2, ACER, ACKTR, ...) and I want to compare their behaviour in the environment. This includes their performance in respect of the episode-reward as well as ...
user avatar
  • 121
0 votes
1 answer
912 views

Yeo-Johnson does not increase normality

I have used Box-Cox Yeo-Johnson transformation to make my skewed data columns less skewed and more normal so that I can remove outliers. e.g. originally most of my columns have a 'skewness' of 400! ...
user avatar
  • 299
0 votes
0 answers
400 views

removing outliers in skewed data for xgboost

i have a couple of columns in my data which are postively skewed. they are non-normal from the hist plots. plotting a qq plot further cinfirms this. i should remove outliers from my data for xgboost. ...
user avatar
  • 299
1 vote
0 answers
59 views

Is K-medoids / partitioning around medoids (PAM) appropriate for clustering data with many zero values?

I need to cluster a matrix which contains zero values. I am clustering three separate sets of 24 values. The first two are non-zero and represent hourly ambient temperature (in K) and electrical ...
user avatar
  • 11

1
2 3 4 5
24