# Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

1,173 questions
Filter by
Sorted by
Tagged with
13 views

### How to get two variable relationship using statistical model?

How to get two variable non-linear relationships using statistical model with other variables fixed? for example, how to analysis the relationship between customer churn and number of message the ...
166 views

### How to solve this bayesian statistics problem? [closed]

I tried solving this statistics problem, but got confused. What is the approach? Based on statistics of gold mining in the area, the probability of gold being present in the grounds of the University ...
21 views

### Is linear regression appropriate for this analysis ? what additional operations to do?

I haven't done statistics / econometrics for a long time. I haven't done a lot by the way. But I would like advice on some methods and tests to be performed on two excel files that I have. Each excel ...
39 views

### How To Determine Number Of Clusters In T-SNE And Best Clustering Algorithm?

I used TSNE method to cluster my DataSet. ...
56 views

### What is the purpose of conducting Simple Random Sampling WITH Replacement?

Part of data preparation is simple random sampling. Random sampling can be of two forms with replacement or without replacement. With replacement, subset sampling simply might contain duplicates of ...
20 views

### How autocorrelation work based on the data plot?

Let's suppose we have a time series is a=[1,1,1,3,3,3,1,1,1,3,3,3] as then the autocorrelation figure for this time series is The lag here is 4, for lag = 1, the autocorrelation is between [1,1,1,3,...
49 views

### What does the meaning of the autocorrelation in this picture?

From this figure, how should I understand what is the lag on the top figure? and when in the bottom figure for example the autocorrelation is 0.45 what does tell us about the above figure? Another ...
11 views

### Can one feature included in more than one PCA in PCA analysis

These are the eighen values and the cumulative I have got with PCA analysis, but I'm struggling with determining the features related to each PCA. This is the correlation matrix plot I have got but ...
29 views

### Applying ML to Diagnostic Analytics on Financial Report

Background: We (a team in the IT department) are currently helping the financial team solve their problem, by automating their tasks. The task is this: they review the financial report monthly and ...
15 views

### Can someone give me an example of frequent large subgraphs being outlier?

I read it from a lecture note: ...
76 views

### What is the difference between p-hacking and data mining bias?

From my understanding, data mining bias occurs when someone repeatedly searches through a data set to find statistically significant results. How is this any different from p-hacking? What is the ...
6 views

### Regression model between time-series clusters

I have two time-series datasets (temperature and speed of vehicle). I performed time-series clustering on them and I have two time-series clusters. I am looking for a technique to find relationship ...
8 views

### Interpreting association results in Rattle

I am new to rattle and attempting to use it for association mining. I am having trouble interpreting the rules that Rattle is generating and I am fairly certain this is because I have made a mistake ...
8 views

### Would an SVM be more successful with binned attributes or non-binned attributes?

Working on an SVM through WEKA and testing which kernel would operate better. I have some data values which are nominal, but about half of them are numerical values with all unique values. There's ...
19 views

### How to compare the peformance of different clusterings without true labels

Firstly, I know some scores like silhouette score and Davies–Bouldin score to compare the performance in one clustering method. However, I am not sure how to compare results in different clustering ...
10 views

### Link prediction Using ARIMA model, how to do it?

I am trying to replicate this paper for predicting link on dynamic network Link prediction using time series of neighborhood-based node similarity scores Here i what i did, in each network snapshot i ...
22 views

### How to Find out the Best Way to Encode Data in ML?

I have been thinking in this problem for quite a while, I cannot figure out a way of knowing, (based on the task) what will be the best data encoding I can do for training the model. Imagine I have a ...
17 views

### How to best visualize time interval difference and compute a single measure

I have a table like as shown below ...
26 views

### Data mining algorithm - Charm

I am trying to find closed frequenct itemsets by using Charm. This is my dataset. Minimum sup is 3. T1: {A B C E} T2: {B C D E G} T3: {C D G} T4: {A B E G} T5: {A C E} T6: {A B D E} Frequency A - 4 ...
25 views

### Visualizing shared instances of p-values<alpha across large numbers of treatments

Assume a data table that presents the p-values of a large number of independent runs of a statistical hypothesis test. Each run represents a single test with two possible hypotheses (i.e., null and ...
26 views

### K-means on a one-dimensional dataset [closed]

Given the data set $𝑋=\{−6,5,0,4,7\}$ and the cluster initialization $𝑉=\{ 5,6\}$ How would the cluster centers (i.e. means) look like after applying naive k-means(computing min. distance $|\cdot |$ ...
22 views

### Does upsampling before subgroup discovery make sense?

In data_analysis_2.ipynb I can find the approach to improve the subgroup discovery by upsampling some of the attributes in the data: Deal with imbalanced data "high_calorie": "...
31 views

### How to make a decision - when there is a tie and no human expert

We have two algorithms (simple rule-based) working on labeling the dataset as "Yes" and "No" for a disease. There is no ML involved in this task. For ex: If Algo 1 says subject 1 ...
13 views

### Event log mining with timing

I want to learn how to mine the data related to event logs, but could not since I do not know the exact term related to it. What I want is to mine the event logs with timing. For example, I want to ...
21 views

### Invent first, find its use later

The typical pipeline in ML is Find a data-related problem that you want to solve Build a model or algorithm that feeds on data related to the problem to try to solve the problem Check if the solution ...
13 views

### When is it admissible to combine two categories?

In a data mining project, I have a categorical variable $C$ which takes possible values $c_i$ and the outcome variable is continous. When I plot the conditional probability distribution of the outcome ...
44 views

### Find all possible clusterizations

I need help to find all possible clusterizations via the k-means method in Python. Let's assume for simplicity that I have the following table: height | weight | country of origin (X/Y/Z) | flag (1/0) ...
82 views

### the ratio of validation set and test set should be equal?

I always heard that the common ratio of the train:validation:test is 70:15:15 or 80:10:10 or 60:20:20, sounds like the validation set and test set should be equal size. Assuming that I wanna use 5 ...
21 views

### Machine Learning for choosing the right cover box to contain smaller item boxes

I am studying on choosing the right box to contain item boxes. When customers order items, items have their own cases and ordered items are packed with the right cover box to contain them to deliver. ...
31 views

### How can I do one class learning for outlier detection?

I understand I can use various sampling techniques when dealing with imbalanced datasets. However, I wonder how I can build a classification model from the training dataset only including data that ...
68 views

### Why does my model produce unrealistic output?

I am trying to run a binary classification problem on people with diabetes and non-diabetes. For labeling my datasets, I followed a simple rule. If a person has T2DM...
69 views

### Solution to a single feature logistic regression problem [duplicate]

So I'm having a hard time conceptualizing how to make mathematical representation of my solution for a simple logistic regression problem. I understand what is happening conceptually and have ...
17 views

### If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
157 views

### How to replace 2 different values in the dataset using Filler Node in IBM SPSS

Anyone can help me out? In my dataset there is two different incorrect encoding as shown in the screenshot of which I want to replace, however, I can't seem to get the filler node to work for two ...
6 views

### Weighting Comments Against Other Parameters to Predict Likes

I've got a dataset of around 5.5k social media entries. I am looking for an algortithm that'll use the text of all entries and try to predict how many likes each post got (this can be both predicting ...
22 views

### A space of functions and their Fourier Transforms?

Conjugate variables and the Fourier transform are often used to analyze different states of a single object. For example in Quantum Mechanics it can be used to describe changing information about ...
362 views

### Understanding SON algorithm - partition based Frequent itemsets mining

There are several variants of the Apriori algorithm that focus on improving efficiency. For example, the base idea of the SON ...
57 views

### Understanding the determination of principal components

The idea of PCA is to find the directions (in high dimensional space) in which the essential structures (with regard to large variance, scatter) of the data lie. The assumption is that original ...
41 views

### How can i cite Sequential Forward Feature Selection (SFFS)?

I've seen many papers/books about this technique but none cite its author. Is it ok to cite any machine learning theory book that explains it? Thanks.
30 views

### What is the threshold for "frequent" in the Apriori algorithm?

I'm reading about the Apriori algorithm using the textbook Introduction to Machine Learning (Ethem Alpaydin) and had a question. I've noticed that the textbook and many other resources I find online ...
59 views

### Using my models with NGBoost?

I've come across this new tool of NGBoost from the Machine Learning group of Stanford, I was curious if peopel have started using it yet. They say that one can have a Base learner such as a regression ...
68 views

### Is it appropriate to use regularised regression for low-dimensional N>>p variable selection problems?

I am currently examining which of sixteen variables are the most important in predicting a binary outcome. There are 907 observations, so obviously $N$ is much larger than $p$ In the last six months ...
14 views

### Neural network for generating interesting sets

I'm facing the following problem: There are $n$ items $U=\{i_1,...,1_n\}$. I have many examples $X$ where $X\subseteq P[U]$ (the power set of $U$). The goal is to create a system that given these ...
42 views

### Need help finding the right vocabulary so I can go research more

Looking for some vocabulary to help me refine my research so I can tackle this problem. Here's an overview of the problem statement I'm working on. At my company we manufacture various products, ...
8 views

### How the pattern ranking performed in this SDM paper?

I am following this SDM paper "Diversified Trajectory Pattern Ranking in Geo-Tagged Social Media" that I found very intersting and inspiring. However due to my limited knowledge in mathematical ...
50 views

### How to compare the behaviour of different pretrained RL models?

I have a number of pretrained RL models (PPO2, ACER, ACKTR, ...) and I want to compare their behaviour in the environment. This includes their performance in respect of the episode-reward as well as ...
521 views

### Yeo-Johnson does not increase normality

I have used Box-Cox Yeo-Johnson transformation to make my skewed data columns less skewed and more normal so that I can remove outliers. e.g. originally most of my columns have a 'skewness' of 400! ...
303 views

### removing outliers in skewed data for xgboost

i have a couple of columns in my data which are postively skewed. they are non-normal from the hist plots. plotting a qq plot further cinfirms this. i should remove outliers from my data for xgboost. ...