Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

When is it admissible to combine two categories?

In a data mining project, I have a categorical variable $C$ which takes possible values $c_i$ and the outcome variable is continous. When I plot the conditional probability distribution of the outcome ...
0
votes
1answer
34 views

Find all possible clusterizations

I need help to find all possible clusterizations via the k-means method in Python. Let's assume for simplicity that I have the following table: height | weight | country of origin (X/Y/Z) | flag (1/0) ...
0
votes
1answer
14 views

the ratio of validation set and test set should be equal?

I always heard that the common ratio of the train:validation:test is 70:15:15 or 80:10:10 or 60:20:20, sounds like the validation set and test set should be equal size. Assuming that I wanna use 5 ...
0
votes
0answers
19 views

Machine Learning for choosing the right cover box to contain smaller item boxes

I am studying on choosing the right box to contain item boxes. When customers order items, items have their own cases and ordered items are packed with the right cover box to contain them to deliver. ...
0
votes
0answers
14 views

How can I do one class learning for outlier detection?

I understand I can use various sampling techniques when dealing with imbalanced datasets. However, I wonder how I can build a classification model from the training dataset only including data that ...
2
votes
1answer
63 views

Why does my model produce unrealistic output?

I am trying to run a binary classification problem on people with diabetes and non-diabetes. For labeling my datasets, I followed a simple rule. If a person has T2DM...
0
votes
0answers
17 views

Solution to a single feature logistic regression problem [duplicate]

So I'm having a hard time conceptualizing how to make mathematical representation of my solution for a simple logistic regression problem. I understand what is happening conceptually and have ...
1
vote
0answers
16 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
0
votes
0answers
33 views

How to replace 2 different values in the dataset using Filler Node in IBM SPSS

Anyone can help me out? In my dataset there is two different incorrect encoding as shown in the screenshot of which I want to replace, however, I can't seem to get the filler node to work for two ...
0
votes
0answers
5 views

Weighting Comments Against Other Parameters to Predict Likes

I've got a dataset of around 5.5k social media entries. I am looking for an algortithm that'll use the text of all entries and try to predict how many likes each post got (this can be both predicting ...
1
vote
0answers
21 views

A space of functions and their Fourier Transforms?

Conjugate variables and the Fourier transform are often used to analyze different states of a single object. For example in Quantum Mechanics it can be used to describe changing information about ...
0
votes
0answers
18 views

Understanding SON algorithm - partition based Frequent itemsets mining

There are several variants of the Apriori algorithm that focus on improving efficiency. For example, the base idea of the SON ...
2
votes
2answers
41 views

Understanding the determination of principal components

The idea of PCA is to find the directions (in high dimensional space) in which the essential structures (with regard to large variance, scatter) of the data lie. The assumption is that original ...
0
votes
1answer
18 views

How can i cite Sequential Forward Feature Selection (SFFS)?

I've seen many papers/books about this technique but none cite its author. Is it ok to cite any machine learning theory book that explains it? Thanks.
0
votes
0answers
10 views

How to do data mining that consider all possible variables specification?

First of all, I know the drawback associated with datamining in modelling, but this case is very specific, and my model don't need any replication. I just need to overfit the results of my database. ...
0
votes
0answers
12 views

Adaboost - Is it (really) necessary to plug sample weights into cost function?

I implemented Adaboost using the SAMME algorithm (for multiclass) with Multilayer Perceptron networks as weak learners. For the MLP, i am using ...
0
votes
1answer
16 views

What is the threshold for “frequent” in the Apriori algorithm?

I'm reading about the Apriori algorithm using the textbook Introduction to Machine Learning (Ethem Alpaydin) and had a question. I've noticed that the textbook and many other resources I find online ...
0
votes
1answer
28 views

Using my models with NGBoost?

I've come across this new tool of NGBoost from the Machine Learning group of Stanford, I was curious if peopel have started using it yet. They say that one can have a Base learner such as a regression ...
0
votes
0answers
17 views

How to measure the effects of multiple shorter time-series on one longer time-series?

I have a question pertains to time-series analysis. For example, I have a time-series data about the downloads of an app for a month. And I want to know that the effects on the downloads time-series ...
1
vote
1answer
36 views

Is it appropriate to use regularised regression for low-dimensional N>>p variable selection problems?

I am currently examining which of sixteen variables are the most important in predicting a binary outcome. There are 907 observations, so obviously $N$ is much larger than $p$ In the last six months ...
0
votes
0answers
23 views

Universal binary model

I have a set of 70 explanatory variables (not independent, numeric/boolean type) and one explained variable called is_bought (binary, yes/no). Data is coming from 10 userforms, each userform has 5-15 ...
1
vote
0answers
12 views

Neural network for generating interesting sets

I'm facing the following problem: There are $n$ items $U=\{i_1,...,1_n\}$. I have many examples $X$ where $X\subseteq P[U]$ (the power set of $U$). The goal is to create a system that given these ...
0
votes
0answers
11 views

Are there any advantages of k-means over Gaussian Mixture (Expectation minimization)?

In other words, if I already included Gaussian mixture into the analysis, does it make sense to add also k-means, as GM clustering is a generalization of k-means?
0
votes
1answer
42 views

Need help finding the right vocabulary so I can go research more

Looking for some vocabulary to help me refine my research so I can tackle this problem. Here's an overview of the problem statement I'm working on. At my company we manufacture various products, ...
1
vote
0answers
7 views

How the pattern ranking performed in this SDM paper?

I am following this SDM paper "Diversified Trajectory Pattern Ranking in Geo-Tagged Social Media" that I found very intersting and inspiring. However due to my limited knowledge in mathematical ...
0
votes
0answers
14 views

Patient Services Probabilities Prediction

I have a dataset in which I have a patient's diagnosis and service corresponding to those diagnoses. I want to apply machine learning in such a way that. When I enter the patient diagnosis to system ...
1
vote
0answers
22 views

How to compare the behaviour of different pretrained RL models?

I have a number of pretrained RL models (PPO2, ACER, ACKTR, ...) and I want to compare their behaviour in the environment. This includes their performance in respect of the episode-reward as well as ...
0
votes
1answer
173 views

Yeo-Johnson does not increase normality

I have used Box-Cox Yeo-Johnson transformation to make my skewed data columns less skewed and more normal so that I can remove outliers. e.g. originally most of my columns have a 'skewness' of 400! ...
0
votes
0answers
144 views

removing outliers in skewed data for xgboost

i have a couple of columns in my data which are postively skewed. they are non-normal from the hist plots. plotting a qq plot further cinfirms this. i should remove outliers from my data for xgboost. ...
1
vote
0answers
21 views

Is K-medoids / partitioning around medoids (PAM) appropriate for clustering data with many zero values?

I need to cluster a matrix which contains zero values. I am clustering three separate sets of 24 values. The first two are non-zero and represent hourly ambient temperature (in K) and electrical ...
0
votes
0answers
49 views

Which machine learning algorithm is best to use for prediction/ranking in this dataset?

Each row in the training set depicts co-occurring symbols (one training sample). The sequence doesn't matter, only the co-occurrence does. a, b, d, e b, c, e, z z, a b, f, g, i, s, u .... .... ...
1
vote
1answer
123 views

Interpreting the results of p-value and Cramer's V value

For my assignment I am working on a data set with the sample size n = 4.000.000 and about 450 columns. I have 4 rows in my cross table that have the values winter, spring, summer and fall. So my cross ...
0
votes
0answers
24 views

why k-means is better in clustering than topic modelling algorithms like LDA?

I want to know about the advantages of K-means in clustering essays to discover their topics. There are a lot of algorithms to do it such as K-medoid, x-means, LDA, LSA, etc. Please give me a full ...
1
vote
0answers
83 views

Mahalanobis distance between high dimensional arrays

As we know, the Mahalanobis distance (MD) is one of the distance metrics for measuring two points in multivariate space. In practice, I can compute Mahalanobis distance between two 1D arrays using ...
24
votes
6answers
3k views

What are the disadvantages of using mean for missing values?

I have an assignment (Data Mining course) and there is a part which asks: "What are the disadvantages of using mean for missing values?" in Missing Value section. ...
0
votes
0answers
10 views

new features selection

I am in a project in which I have a specific description of a certain binary profile for which I have about 200 positive examples and another 200 negative. This description is given from about 60 ...
0
votes
0answers
19 views

Stacking model: is finding best threshold necessary?

After performing a stacking model (with rstudio), is it necessary to choose the best threshold for it? In general after finding the best model among all the fitted models , you have to choose the ...
1
vote
1answer
27 views

How to measure smoothness of inputs over outputs?

I know similar questions have been asked for time series data. But my question is a little bit different. Consider that we have input dataset $X \in R^{N \times M}$, where $M$ is the dimension of ...
0
votes
0answers
8 views

Different normalization techniques (mean and min-max) on different columns in a data frame

Is it possible to have different normalization techniques (mean and min-max) applied on different columns in a data frame?
0
votes
0answers
18 views

Predict for an outcome within a time window

I have a dataset which has around 10K records. My objective is to predict whether the customer will churn or not. Binary classification problem with each class representing around 55:45 proportion ...
0
votes
0answers
38 views

Compute correlation between pairs of attributes for each class label

I am working on collecting some dataset characteristics for binary classification tasks, and I want to calculate the correlation following the measure proposed by this quote: The correlations ...
0
votes
0answers
12 views

What approach or unsupervised methods can be used to pick out patterns in noise?

This is a hypothetical situation. Let's say you have access to a lot of human behaviors and characteristics (features). Let's say you have a sample of 10000 humans. You know that within this sample, ...
0
votes
0answers
29 views

Low sample size with independent observations

I am looking at sports team level data (summarized by average in each season) over several seasons and would like to predict/classify the winner of the championship. In a single season, the data has ...
1
vote
0answers
27 views

How is domain knowledge developed?

Most sources I've read state that domain knowledge is crucial for making good inferences. As an example, if I'm conducting a study to assess the importance of a new biomarker for heart attacks, the ...
0
votes
0answers
9 views

Predicting 3-value class label with binary decision tree

Is binary decision tree working better while predicting class label that is binary as well (2 possible values) or it doesn't have any impact? If it does, would it be a better idea to use ternary ...
3
votes
2answers
108 views

What does the true level of significance mean in data mining?

There is a formula α*=1-(1-α)^c/k, a* - true level of significance. a - nominal level of significance. c - the number of candidate regressors. k - the number of finally selected regressors. I ...
2
votes
1answer
55 views

How to select best feature set?

I am currently using feature selection approaches like filter, wrapper, embedded etc. All these methods give different set of features and I rank them based on their frequency of occurrence in other ...
1
vote
2answers
1k views

How to adjust confounders in Logistic regression?

I have a binary classification problem where I apply logistic regression. I have a set of features that are found significant. But I understand that Logistic regression doesn't consider feature ...
1
vote
1answer
43 views

How to split training data when learning DNN for unknown test data?

I'm designing a CNN model for a data mining competition in which we are provided with N sample of training data. We do not know the test size, but presumably it is from the same distribution as ...
8
votes
4answers
1k views

One hot encoding of a binary feature when using XGBoost

I already asked this question is SO; however, I realized that this may be a better place for this type of question. I am well aware that when using categorical features with tree based models such as ...

1
2 3 4 5
24