Questions tagged [data-mining]
Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.
1,180
questions
0
votes
0
answers
12
views
Data mining, Hierarchical Clustering [closed]
Suppose we have five points: (0,0.1), (0.1,0), (1, 1.1), (1, 0.9), (1,1). We have
two clustering results for the five points. Result 1 has two clusters: CL1 =
{(0,0.1), (0.1,0)}, CL2 = {(1, 1.1), (1, ...
0
votes
0
answers
10
views
Clustering data points by difference in distributions of a response variable?
My question is along the lines of this: Clustering based on interaction between the variables. However, my situation is kind of the reverse--the independent variable is binary and the response ...
- 111
4
votes
1
answer
178
views
Why do we not have a true upper limit for dissimilarity measure?
The similarity measure between two attributes always falls in the range of $[0,1]$, why is this not true in the case of dissimilarity where the value falls in the range of $[0, \infty)$?
Can't the ...
0
votes
0
answers
7
views
Association rules - Single item transactions
I am currently handling a dataset which involves customers' purchase history data.
our data consists of columns like purchase_date, item_id, trasaction_id, volume, price etc
However, I see that most ...
- 2,310
1
vote
0
answers
25
views
Similiarity between two corpus of text
I have two separated corpus of text, and i would like to understand wheter these are similiar or not using cosine similarity.
I'm not sure on how to approach this problem, but i was thinking as a ...
- 23
0
votes
0
answers
17
views
Relation between AIC of each predictor variable and final AIC
I ran logistic regression by R. I want to know what relation there is between AIC of each predictor variable and final AIC?
0
votes
1
answer
27
views
Prudent to reduce data size for the sake of model performance?
I am currently working on predicting the customer revenue in next 3,6 or 9 months using the below two methods
a) Buy Till you die probabilistic models
b) Tweedie regression and other regression ...
- 2,310
0
votes
0
answers
49
views
Non data hungry models for prediction
I have a dataset of customer transactions containing revenue, customer id, region, product category, product id, support team, date of trqnsaction etc.
I intend to use buy till you die models for ...
- 2,310
1
vote
0
answers
10
views
Determining Patterns Across Lists
Given several lists:
List 1
List 2
List 3
Shirt
Delivery
Outcome
Ladder
Outcome
Football
Football
Football
Teacher
Teacher
Teacher
Operation
Airport
Apple
Engineering
I am looking to:
Detect ...
0
votes
0
answers
74
views
Multiclass ROC curve about one vs one
I am learning ROC about multiclass problem.
I read this article https://towardsdatascience.com/multiclass-classification-evaluation-with-roc-curves-and-roc-auc-294fd4617e3a .
I am confused about ovo ...
- 1
1
vote
1
answer
49
views
Distance function that captures both circular and "appear as line" clusters [closed]
based on what I know in k-mean clustering, if i use single linkage distance it can capture clusters of thread shapes but it is not suitable for capturing circular clusters.
Also If we use complete ...
- 11
0
votes
0
answers
170
views
Jenks Natural breaks - Interpreting Goodness of Variance Fit
I am trying to find breaks in a multiple continuous type variables.
So, I tried the jenks natural breaks algorithm.
Based on the code from here, I managed to find ...
- 2,310
1
vote
0
answers
91
views
Cardinality of Hypothesis Space H
Assume I have the below dataset, features M and N are numerical, label is binary.
M
N
Label
2
3
y
6
1
y
1
12
y
3
9
y
11
15
n
7
13
n
4
8
n
9
10
n
My decision tree with binary split only has 1 ...
- 11
4
votes
1
answer
299
views
Why do I get different results after shuffling data using DBSCAN
Sometimes, by simply shuffling my data, not changing the parameters, I get a different cluster result using sklearn.DBSCAN. Why this happens?
I mean, by shuffling data, the data distribution is the ...
- 43
0
votes
0
answers
41
views
What are exactly QSAR models?
I'm reading on the internet about QSAR models and I don't really understand what they are. I see some webs in which some data mining techniques like random forest, CART, etc., are mentioned. What are ...
- 55
1
vote
0
answers
96
views
How do I build a decision tree model with a dataset that only has categorical values
kI'm trying to build a decision tree model on a dataset that only has categorical values, an example fragment of the dataset is below. My training dataset consists of 40 observations
...
0
votes
1
answer
120
views
What is the purpose of cost function for Dynamic Time Warping
In Data Analysis, Dynamic Time Warping is a method to better speify the similarity between two time series: https://en.wikipedia.org/wiki/Dynamic_time_warping
What is the purpose of introducing a cost ...
- 115
0
votes
0
answers
12
views
Lift - Class ratio as actual randomness-measure
Context
The Lift should show how a machine learning model performs better than randomness. Thus, a curve representing the ratio between the predicted class of a ...
- 153
0
votes
0
answers
10
views
Pick minimum value based on MSE and point
I am attempting to automate some processes at work. Im familiar with statisical methods such as GLM, ANOVAs, Basic OLS, etc. But I am unsure of methods which I can use for this.
problem statement: ...
- 21
0
votes
0
answers
20
views
Clustering trends across categories: which one to use?
I have a dataset with daily temperatures for every country in 4 seasons: Spring, Summer, Fall, and Winter.
I wish to group(cluster) countries with similar temperature trends across the 4 seasons. I am ...
2
votes
1
answer
57
views
buy till you die - Discounted cash flow - NBD/GAMMA [closed]
I was learning about BYTD in an online tutorial here
I understand that it is used
a) to predict the number of purchases that will be made by the customer.
b) Lifetime value of the cuatomer over a ...
- 2,310
1
vote
0
answers
83
views
What statistical tests can I use to compare two different ml algorithm on a dataset that has been augmented using two different approaches?
I used SMOTE and ADASYN on a dataset separately and used Random Forest and KNN on them. Both giving quite close accuracy(RF= 97.75 %, KNN= 97.22% where SMOTE was used) & (RF=97.67% and KNN=97.42% ...
- 11
1
vote
1
answer
2k
views
silhouette score vs Distortion score
I am working on segmenting my customers with clustering. My dataset size is 7315 rows and 30 features.
So, as a beginner to clustering, I passed all my 29 features (excluding id column) to the cluster....
- 2,310
1
vote
0
answers
10
views
For the given data find the clusters. Assume the relevant parameters needed [closed]
Below is the given data, how can I make clusters using symmetric matrix?
- 11
0
votes
0
answers
33
views
Explainable clustering for broker churn
I have a dataset like as below
You can see that my dataset is longitudinal. There will be multiple records for the same broker with multiple or same customer, selling same or different products.
Now, ...
- 2,310
-1
votes
1
answer
44
views
why random model response for X decile is fixed at X%?
I was reading online tutorials on lift and gain charts here, here, here
In all of these tutorials, I read or see that random model curve is drawn with the expectation at each Xth decile, we get X% ...
- 2,310
5
votes
2
answers
807
views
A universal measure of the accuracy of linear regression models
I have a dataset that contains both outliers and multicollinearity.
I applied three different regression models to that dataset: ordinary least square, absolute linear regression, and Huber regression....
- 1,919
1
vote
0
answers
48
views
Dealing with class imbalance and data complexity issues
I am doing a classification task for a 5-class imbalanced dataset. Class distribution shows 2-majority & 2-minority clasess, as far:
...
- 253
2
votes
1
answer
122
views
Classification methods with only nominal attributes
I would like to know what classification methods can naturally deal with only nominal attributes. An example is decision trees created by C4.5. I imagine that other classifiers based on decision trees ...
- 55
2
votes
0
answers
366
views
How do non binary decision trees deal with categorical values that weren't in training?
I've been implementing Random Forest from scratch as a learning exercise.
While most algorithms for decision trees seem to deal exclusively in binary yes/no questions, leading to binary trees, ...
- 21
1
vote
1
answer
412
views
Class separability and Overlap
I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average.
This is a classification task and ...
- 215
1
vote
1
answer
126
views
Statistical modeling vs data mining [duplicate]
What is the difference between "statistical modeling" and "data mining"? I have searched the internet, but I can't see it clearly. Is there any overlap? Can they be considered ...
- 55
1
vote
0
answers
12
views
Data mining for gene dataset with pairwise comparison [closed]
I have a table of sample a b c's gene data pairwise comparison, but I really don't know how to start data mining for this specific dataset. I wonder if anyone can come up with some idea, maybe in some ...
- 11
0
votes
0
answers
22
views
Is linear regression appropriate for this analysis ? what additional operations to do?
I haven't done statistics / econometrics for a long time. I haven't done a lot by the way. But I would like advice on some methods and tests to be performed on two excel files that I have.
Each excel ...
1
vote
1
answer
721
views
How To Determine Number Of Clusters In T-SNE And Best Clustering Algorithm?
I used TSNE method to cluster my DataSet.
...
- 13
4
votes
1
answer
542
views
What is the purpose of conducting Simple Random Sampling WITH Replacement?
Part of data preparation is simple random sampling. Random sampling can be of two forms with replacement or without replacement. With replacement, subset sampling simply might contain duplicates of ...
1
vote
0
answers
26
views
How autocorrelation work based on the data plot?
Let's suppose we have a time series is a=[1,1,1,3,3,3,1,1,1,3,3,3] as
then the autocorrelation figure for this time series is
The lag here is 4, for lag = 1, the autocorrelation is between [1,1,1,3,...
- 143
1
vote
1
answer
367
views
What does the meaning of the autocorrelation in this picture?
From this figure, how should I understand what is the lag on the top figure? and when in the bottom figure for example the autocorrelation is 0.45 what does tell us about the above figure?
Another ...
- 143
1
vote
0
answers
34
views
Applying ML to Diagnostic Analytics on Financial Report
Background:
We (a team in the IT department) are currently helping the financial team solve their problem, by automating their tasks.
The task is this: they review the financial report monthly and ...
3
votes
2
answers
321
views
What is the difference between p-hacking and data mining bias?
From my understanding, data mining bias occurs when someone repeatedly searches through a data set to find statistically significant results. How is this any different from p-hacking? What is the ...
- 145
1
vote
0
answers
39
views
How to compare the peformance of different clusterings without true labels
Firstly, I know some scores like silhouette score and Davies–Bouldin score to compare the performance in one clustering method. However, I am not sure how to compare results in different clustering ...
- 111
1
vote
1
answer
47
views
How to Find out the Best Way to Encode Data in ML?
I have been thinking in this problem for quite a while, I cannot figure out a way of knowing, (based on the task) what will be the best data encoding I can do for training the model.
Imagine I have a ...
- 111
0
votes
0
answers
20
views
How to best visualize time interval difference and compute a single measure
I have a table like as shown below
...
- 2,310
3
votes
1
answer
30
views
Visualizing shared instances of p-values<alpha across large numbers of treatments
Assume a data table that presents the p-values of a large number of independent runs of a statistical hypothesis test. Each run represents a single test with two possible hypotheses (i.e., null and ...
2
votes
1
answer
37
views
How to make a decision - when there is a tie and no human expert
We have two algorithms (simple rule-based) working on labeling the dataset as "Yes" and "No" for a disease. There is no ML involved in this task.
For ex: If Algo 1 says subject 1 ...
- 2,310
1
vote
1
answer
71
views
Find all possible clusterizations
I need help to find all possible clusterizations via the k-means method in Python. Let's assume for simplicity that I have the following table:
height | weight | country of origin (X/Y/Z) | flag (1/0)
...
- 131
0
votes
1
answer
465
views
the ratio of validation set and test set should be equal?
I always heard that the common ratio of the train:validation:test is 70:15:15 or 80:10:10 or 60:20:20, sounds like the validation set and test set should be equal size. Assuming that I wanna use 5 ...
0
votes
0
answers
58
views
Machine Learning for choosing the right cover box to contain smaller item boxes
I am studying on choosing the right box to contain item boxes.
When customers order items, items have their own cases and ordered items are packed with the right cover box to contain them to deliver. ...
- 2,695
0
votes
0
answers
67
views
How can I do one class learning for outlier detection?
I understand I can use various sampling techniques when dealing with imbalanced datasets. However, I wonder how I can build a classification model from the training dataset only including data that ...
- 2,695
2
votes
1
answer
78
views
Why does my model produce unrealistic output?
I am trying to run a binary classification problem on people with diabetes and non-diabetes.
For labeling my datasets, I followed a simple rule. If a person has T2DM...
- 2,310