Skip to main content

Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

How can I assign households to coordinates in a social space consistent with pairwise distance measures?

How can I assign households to coordinates in a social space consistent with pairwise distance measures? I have a question which is somewhat ill-defined, about creating an interesting and useful ...
andrewH's user avatar
  • 3,157
1 vote
0 answers
34 views

Best method to identify layered clusters

The Problem Hello everyone. I'm working with a dataset that has 15300 samples with 49 features each, equally distributed amongst three classes. I used TSNE to reduce the dimensions of the feature ...
Amyr14's user avatar
  • 11
0 votes
0 answers
18 views

What machine learning algorithms for process optimization

I found a lot of articles talking about manufacturing process optimization using ML but I could not find detailed methodology with state-of-the-art algorithms to use for this. Classical ML algorithms ...
vincda's user avatar
  • 1
0 votes
0 answers
16 views

Cubist Model Tree Overlapping Rules

Can someone help me understand the output of a cubist model created by the R Cubist package? The documentation package manual for objects of class "cubist" seems to be nonexistent. Looking ...
noNameTed's user avatar
  • 135
2 votes
1 answer
93 views

Data mining for categorical variables that correspond to modes of a continuous variable

Is there a good method or test to find categorical variables associated with modes of a continuous variable? Take the velocity of a car as an example. You would expect there to be at least two modes ...
noNameTed's user avatar
  • 135
3 votes
1 answer
61 views

Selecting variables using lasso algorithm

I have a question concerning a large dataset with 94 observation and 15000 variables. For data mining models (boosting, trees, neural networks...) this number of variables are too much and I have to ...
ali's user avatar
  • 71
0 votes
0 answers
9 views

Using Unordered Text Features in Machine Learning Model

I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed ...
Omnitragedy's user avatar
0 votes
0 answers
24 views

Equation for all frequent itemsets

Consider that the database has only three transactions {<a1,a2,...,a100>, <a1,a2,...,a50>, <a1,2,...,a25>}. Suppose min_sup = 2. Give the equation for all frequent itemsets. Also ...
Invisible's user avatar
0 votes
0 answers
20 views

Confidence vs order in data mining

Is there the relation between confidence and order of association rule when I am applying Apriori algorithm? Or is it dependent on the dataset? Disclaimer: This is based on a homework question. The ...
tumchaaditya's user avatar
2 votes
1 answer
1k views

Is it acceptable that my training accuracy is 0.99 and my test accuracy is also 0.99?

This is a multiclass classification task for a balanced dataset. I am using the Random Forest Classifier for classification, and I have calculated the evaluation metrics: Training set metrics: ...
Mahdi Salah's user avatar
1 vote
0 answers
70 views

Pattern mining scored sequences

I've generated some data (example below) and I'm not sure how to go about analyzing it. N #0 #1 #2 #3 #4 #5 #6 #7 #8 #9 ... ... ... ... ... ... ... ... ... ... ... 65 C A A A C B B B C C 90 C A A ...
cliesens's user avatar
2 votes
1 answer
54 views

Using ML for simple rule based to obtain likelihood

Let's say I already have a simple rule which provides me the variables that help determine the output label and relationship between variables and label is rather simple and deterministic one, can I ...
The Great's user avatar
  • 3,302
2 votes
1 answer
88 views

Can variables used for rule based labeling be treated as input features?

I am currently working on binary classification problem with imbalanced dataset (n=3419 and 69:31). However, based on the business expertise of the users, they have generated rule-based label based on ...
The Great's user avatar
  • 3,302
1 vote
1 answer
69 views

Can input variable be leaking data?

I am currently working on a binary classification problem using imbalanced data. The algorithm that I am using is random forest. The problem is about predicting whether each sales project will meet ...
The Great's user avatar
  • 3,302
4 votes
1 answer
208 views

Why do we not have a true upper limit for dissimilarity measure?

The similarity measure between two attributes always falls in the range of $[0,1]$, why is this not true in the case of dissimilarity where the value falls in the range of $[0, \infty)$? Can't the ...
Ashmin Jayson's user avatar
1 vote
0 answers
51 views

Similiarity between two corpus of text

I have two separated corpus of text, and i would like to understand wheter these are similiar or not using cosine similarity. I'm not sure on how to approach this problem, but i was thinking as a ...
user373562's user avatar
0 votes
0 answers
23 views

Relation between AIC of each predictor variable and final AIC

I ran logistic regression by R. I want to know what relation there is between AIC of each predictor variable and final AIC?
user373401's user avatar
1 vote
1 answer
36 views

Prudent to reduce data size for the sake of model performance?

I am currently working on predicting the customer revenue in next 3,6 or 9 months using the below two methods a) Buy Till you die probabilistic models b) Tweedie regression and other regression ...
The Great's user avatar
  • 3,302
0 votes
0 answers
68 views

Non data hungry models for prediction

I have a dataset of customer transactions containing revenue, customer id, region, product category, product id, support team, date of trqnsaction etc. I intend to use buy till you die models for ...
The Great's user avatar
  • 3,302
1 vote
0 answers
12 views

Determining Patterns Across Lists

Given several lists: List 1 List 2 List 3 Shirt Delivery Outcome Ladder Outcome Football Football Football Teacher Teacher Teacher Operation Airport Apple Engineering I am looking to: Detect ...
Nathaniel Eldridge's user avatar
1 vote
0 answers
142 views

Multiclass ROC curve about one vs one

I am learning ROC about multiclass problem. I read this article https://towardsdatascience.com/multiclass-classification-evaluation-with-roc-curves-and-roc-auc-294fd4617e3a . I am confused about ovo ...
李宜倫's user avatar
1 vote
1 answer
76 views

Distance function that captures both circular and "appear as line" clusters [closed]

based on what I know in k-mean clustering, if i use single linkage distance it can capture clusters of thread shapes but it is not suitable for capturing circular clusters. Also If we use complete ...
user368884's user avatar
1 vote
1 answer
761 views

Jenks Natural breaks - Interpreting Goodness of Variance Fit

I am trying to find breaks in a multiple continuous type variables. So, I tried the jenks natural breaks algorithm. Based on the code from here, I managed to find ...
The Great's user avatar
  • 3,302
1 vote
0 answers
108 views

Cardinality of Hypothesis Space H

Assume I have the below dataset, features M and N are numerical, label is binary. M N Label 2 3 y 6 1 y 1 12 y 3 9 y 11 15 n 7 13 n 4 8 n 9 10 n My decision tree with binary split only has 1 ...
Dummy person's user avatar
5 votes
1 answer
734 views

Why do I get different results after shuffling data using DBSCAN

Sometimes, by simply shuffling my data, not changing the parameters, I get a different cluster result using sklearn.DBSCAN. Why this happens? I mean, by shuffling data, the data distribution is the ...
Brown_Z's user avatar
  • 53
1 vote
1 answer
65 views

What are exactly QSAR models?

I'm reading on the internet about QSAR models and I don't really understand what they are. I see some webs in which some data mining techniques like random forest, CART, etc., are mentioned. What are ...
dreamco9's user avatar
1 vote
0 answers
268 views

How do I build a decision tree model with a dataset that only has categorical values

kI'm trying to build a decision tree model on a dataset that only has categorical values, an example fragment of the dataset is below. My training dataset consists of 40 observations ...
Milap Jhumkhawala's user avatar
0 votes
1 answer
364 views

What is the purpose of cost function for Dynamic Time Warping

In Data Analysis, Dynamic Time Warping is a method to better speify the similarity between two time series: https://en.wikipedia.org/wiki/Dynamic_time_warping What is the purpose of introducing a cost ...
PwNzDust's user avatar
  • 115
0 votes
0 answers
23 views

Clustering trends across categories: which one to use?

I have a dataset with daily temperatures for every country in 4 seasons: Spring, Summer, Fall, and Winter. I wish to group(cluster) countries with similar temperature trends across the 4 seasons. I am ...
Pikachu1998's user avatar
2 votes
1 answer
72 views

buy till you die - Discounted cash flow - NBD/GAMMA [closed]

I was learning about BYTD in an online tutorial here I understand that it is used a) to predict the number of purchases that will be made by the customer. b) Lifetime value of the cuatomer over a ...
The Great's user avatar
  • 3,302
1 vote
0 answers
137 views

What statistical tests can I use to compare two different ml algorithm on a dataset that has been augmented using two different approaches?

I used SMOTE and ADASYN on a dataset separately and used Random Forest and KNN on them. Both giving quite close accuracy(RF= 97.75 %, KNN= 97.22% where SMOTE was used) & (RF=97.67% and KNN=97.42% ...
Mahin Rahman's user avatar
2 votes
1 answer
4k views

silhouette score vs Distortion score

I am working on segmenting my customers with clustering. My dataset size is 7315 rows and 30 features. So, as a beginner to clustering, I passed all my 29 features (excluding id column) to the cluster....
The Great's user avatar
  • 3,302
1 vote
0 answers
17 views

For the given data find the clusters. Assume the relevant parameters needed [closed]

Below is the given data, how can I make clusters using symmetric matrix?
Nivedita's user avatar
-1 votes
1 answer
57 views

why random model response for X decile is fixed at X%?

I was reading online tutorials on lift and gain charts here, here, here In all of these tutorials, I read or see that random model curve is drawn with the expectation at each Xth decile, we get X% ...
The Great's user avatar
  • 3,302
8 votes
2 answers
1k views

A universal measure of the accuracy of linear regression models

I have a dataset that contains both outliers and multicollinearity. I applied three different regression models to that dataset: ordinary least square, absolute linear regression, and Huber regression....
jeza's user avatar
  • 2,109
1 vote
0 answers
57 views

Dealing with class imbalance and data complexity issues

I am doing a classification task for a 5-class imbalanced dataset. Class distribution shows 2-majority & 2-minority clasess, as far: ...
arilwan's user avatar
  • 273
2 votes
1 answer
233 views

Classification methods with only nominal attributes

I would like to know what classification methods can naturally deal with only nominal attributes. An example is decision trees created by C4.5. I imagine that other classifiers based on decision trees ...
dreamco9's user avatar
4 votes
0 answers
616 views

How do non binary decision trees deal with categorical values that weren't in training?

I've been implementing Random Forest from scratch as a learning exercise. While most algorithms for decision trees seem to deal exclusively in binary yes/no questions, leading to binary trees, ...
Acrid's user avatar
  • 41
1 vote
1 answer
646 views

Class separability and Overlap

I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average. This is a classification task and ...
super_ask's user avatar
  • 225
1 vote
1 answer
187 views

Statistical modeling vs data mining [duplicate]

What is the difference between "statistical modeling" and "data mining"? I have searched the internet, but I can't see it clearly. Is there any overlap? Can they be considered ...
dreamco9's user avatar
1 vote
0 answers
14 views

Data mining for gene dataset with pairwise comparison [closed]

I have a table of sample a b c's gene data pairwise comparison, but I really don't know how to start data mining for this specific dataset. I wonder if anyone can come up with some idea, maybe in some ...
yuan's user avatar
  • 11
0 votes
0 answers
23 views

Is linear regression appropriate for this analysis ? what additional operations to do?

I haven't done statistics / econometrics for a long time. I haven't done a lot by the way. But I would like advice on some methods and tests to be performed on two excel files that I have. Each excel ...
GIlgamesh's user avatar
1 vote
1 answer
1k views

How To Determine Number Of Clusters In T-SNE And Best Clustering Algorithm?

I used TSNE method to cluster my DataSet. ...
a.programmer's user avatar
4 votes
1 answer
997 views

What is the purpose of conducting Simple Random Sampling WITH Replacement?

Part of data preparation is simple random sampling. Random sampling can be of two forms with replacement or without replacement. With replacement, subset sampling simply might contain duplicates of ...
Abdulkarim Kanaan's user avatar
1 vote
0 answers
26 views

How autocorrelation work based on the data plot?

Let's suppose we have a time series is a=[1,1,1,3,3,3,1,1,1,3,3,3] as then the autocorrelation figure for this time series is The lag here is 4, for lag = 1, the autocorrelation is between [1,1,1,3,...
Raz's user avatar
  • 135
1 vote
1 answer
627 views

What does the meaning of the autocorrelation in this picture?

From this figure, how should I understand what is the lag on the top figure? and when in the bottom figure for example the autocorrelation is 0.45 what does tell us about the above figure? Another ...
Raz's user avatar
  • 135
1 vote
0 answers
40 views

Applying ML to Diagnostic Analytics on Financial Report

Background: We (a team in the IT department) are currently helping the financial team solve their problem, by automating their tasks. The task is this: they review the financial report monthly and ...
Jasmine Lawrence's user avatar
3 votes
2 answers
532 views

What is the difference between p-hacking and data mining bias?

From my understanding, data mining bias occurs when someone repeatedly searches through a data set to find statistically significant results. How is this any different from p-hacking? What is the ...
Flux's user avatar
  • 145
1 vote
0 answers
56 views

How to compare the peformance of different clusterings without true labels

Firstly, I know some scores like silhouette score and Davies–Bouldin score to compare the performance in one clustering method. However, I am not sure how to compare results in different clustering ...
4daJKong's user avatar
  • 111
1 vote
1 answer
80 views

How to Find out the Best Way to Encode Data in ML?

I have been thinking in this problem for quite a while, I cannot figure out a way of knowing, (based on the task) what will be the best data encoding I can do for training the model. Imagine I have a ...
Stack's user avatar
  • 111

1
2 3 4 5
24