Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
2
votes
3answers
62 views

One hot encoding of a binary feature when using XGBoost

I already asked this question is SO; however, I realized that this may be a better place for this type of question. I am well aware that when using categorical features with tree based models such as ...
0
votes
0answers
8 views

How to combine ranked features for feature selection? [closed]

I'm working on feature selection based on distance measures, I'm using Matlab software. first I compute the average distance between each feature and the rest of features in the data, so I compute a ...
0
votes
1answer
36 views

Number of Parameters to be learned in k Guassian Mixture model?

How many parameters do we need to learn for a k-Gaussian mixture model, where all mixtures have the same spherical radius?
1
vote
2answers
55 views

Event Identification in Series

I've just posted this question on Data Science SE which asks about machine learning methods to identify "events" in series (time-series or otherwise). I'm wondering if I should consider regressions ...
0
votes
0answers
18 views

How to identify important features in deep learning models

I am mostly familiar with traditional hand-crafted feature setting where we use ML algorithm such as SVM to analyse these features. In this way, we can identify what were the most important features ...
0
votes
0answers
21 views

Stratification of sample data is lowering my accuracy?

So I've got this trainingset, it has a bunch of stuff yada yada.. Main point is that there are two target variables that only occur once in the dataset. This means I can't stratify when sampling, I ...
0
votes
1answer
34 views

Data Mining/Statistical Methods to find trends, peaks, etc

currently I am working on a project for my final exam. The data is coming from a streaming plattform. The data I am using are some logging data (data when customers have problems with the streaming ...
0
votes
0answers
15 views

Statistical error in the approximation-estimation tradeoff

Show that $$E(g_\tau ^G(X)-g^* (X))^2 = E(X^T \hat{\beta}-X^T\beta^G)^2+E(X^T\beta^G-g^*(X))^2$$ where $g_\tau ^G(X) = X^T \hat{\beta}$ and $g^G(X) = X^T \beta^G$ where G is a class of linear ...
0
votes
1answer
16 views

Model Integration

I have two models one of them made by using a large dataset, the other one made by using a small dataset. I want to integrate these two datasets. The large dataset has more samples of rare instances. ...
0
votes
1answer
77 views

Why is the correlation of a variable and itself a histogram?

This post is visualizing the Wine dataset. You may have noticed that the figures along the diagonal look different. They are histograms of values of individual variables. We can see that the "Ash" ...
0
votes
1answer
42 views

What an input space exactly is in the context of machine learning?

I've been confused about various "space"s in machine learning for a long time. I've checked out this, this and this post. I am trying to get understanding through some concrete examples like this ...
1
vote
2answers
44 views

Should you take a sample when doing EDA?

Suppose i have a large dataset, such that python graphing libraries are unable to handle. Is it a good idea to take a random sample? Specifically if it's a classification task, and where the target is ...
0
votes
1answer
19 views

Can someone give a concrete example of exploitation in the context of Exploratory Data Analysis?

This post says Exploratory Data Analysis (EDA) consists of 2 steps exploration and exploitation. I know a little about exploration which uses some techniques such as data visualization to understand ...
0
votes
1answer
18 views

Interpret the clustering results of Weka to measure the performance [closed]

I'm having the Boston dataset, where it's class variable in the housing price. So I think regression is more suitable for this dataset, so we can predictions. I'm using Weka for this. I used several ...
0
votes
0answers
15 views

What are the state of the art algorithms for association rule mining?

I have done some work with the apriori algorithm to mine for association rules in a market basket dataset. However, the number of rules returned is enormous and a large part of the rules are just some ...
0
votes
0answers
32 views

Can anyone suggest the source to study basics needed for The Elements of Statistical Learning by Trevor Hastie Robert Tibshirani Jerome Friedman [duplicate]

anyone suggest the relevant study material to get the basics cleared for understanding the book The Elements of Statistical Learning by Trevor Hastie Robert Tibshirani Jerome Friedman
0
votes
1answer
48 views

What is first order difference in trend analysis?

I am following this paper: Measuring, Predicting and Visualizing Short-Term Change in Word Representation and Usage in VKontakte Social Network where in Differencing Statistics section they describe ...
1
vote
1answer
97 views

How to calculate tf-idf for a single term

I am following the tf-idf method described in this paper: Measuring, Predicting and Visualizing Short-Term Change in Word Representation and Usage in VKontakte ...
0
votes
0answers
9 views

How to edge weight the network to obtain shortest path and centrality measures

I am currently using networkx as follows to create a co-occurrence network where the weights represents the number of co-occurrences to calculate ...
1
vote
0answers
8 views

How to identify important categories in categorical variable to predict some binary variable?

Imagine that we have two sets of data. The first one contains information about websites visited by certain users: ...
0
votes
0answers
13 views

In a particular KNN with a particular dataset, what exactly the hypothesis space is?

Wiki gives this setting for KNN Suppose we have pairs ${\displaystyle (X_{1},Y_{1}),(X_{2},Y_{2}),\dots ,(X_{n},Y_{n})}$ taking values in ${\mathbb {R}}^{d}\times \{1,2\}$, where Y is the ...
1
vote
1answer
33 views

scikit-learn feature selection on k-fold loop

I am using the iterator of StratifiedKFold from sklearn and i've noticed that i must include a process of feature selection on ...
1
vote
2answers
68 views

Could anyone explain the terms “Hypothesis space” “sample space” “parameter space” "feature space in machine learning with one concrete example?

I am confused with these machine learning terms, and trying to distinguish them with one concrete example. for instance, use logistic regression to classify a bunch of cat images. assume there are 1,...
2
votes
1answer
116 views

Why is bagging stable classifiers not a good idea?

Citing Bagging Predictors, Section 4 (emphasis mine): Let each $(y, \mathbf{x})$ case in $\mathcal{L}$ be independently drawn from the distribution $P$. Suppose $y$ is numerical and $\phi(\mathbf{x}...
1
vote
1answer
122 views

Choosing the Right Data Mining Classification Technique

From the different Data Mining tasks, I want to train the Classification. For that, I: Took this dataset (can be used as an example for the answer). Got to know the data (data objects, attribute ...
0
votes
1answer
38 views

How to make a model to distinguish positions using data from sensors?

What I would like to do I need ideas about how to analyze data and make a model to distinguish positions where an object locates using data from receivers. I don't know of a similar case like my ...
0
votes
1answer
62 views

How can you test two datasets to determine if they originate from the same generating process?

I have two feature matrices X_i and X_j, with identical features but their own n - is there a statistical test to determine if the latent data generating process behind X_i is indistinguishable from ...
0
votes
0answers
14 views

How to determine the proper number of samples for development/test set in machine learning

I would like to know if there is a statistical/empirical method to determine the proper number of samples for development/test set used for testing the generalized ability of a machine learning model. ...
0
votes
0answers
15 views

Modeling both linear and non-linear relationship

Apology for being verbose and all the typos or mistakes. This problem has been bothering me for a while and I really hope you can help me with it. Let's say I want to model quarterly sales for a ...
0
votes
0answers
37 views

How to derive a ranking function from observations [duplicate]

I have a employee dataset with the following 5 details. projects completed customer ratings number of bugs reported customer complaints profit I want to rank ...
1
vote
1answer
61 views

How to convert classification features into a ranking function?

I am using 3 features (x1, x2, x3) for binary classification. All my feature values are in 0 to 1 range (unit range). I obtained how important each feature was in classification as follows (i.e. <...
0
votes
1answer
38 views

Which machine learning or deep learning model to use / make [closed]

I wanted to use machine learning / deep learning to predict profit of a , say, a shop using satellite imagery and previous data and I don't know what to use. I want to use close satellite imagery to ...
1
vote
1answer
209 views

K-means calculate MSE in Weka

I am doing some clustering analysis with Weka and decided to apply the k-means algorithm (the clusterer SimpleKMeans). On my first analysis I ran the algorithm with 2 clusters. Then, after finding ...
1
vote
1answer
81 views

Density Estimation and Data Normalization

Is there any problem to first normalize data (for example, min-max one) then use kernel density estimation to get pdf of each sample? Thanks.
0
votes
2answers
652 views

Splitting data into training, validation and test sets

I am currently comparing 3 classification methods on a data set (in R). To do so, I have run this over a loop (100 iterations). I have split my data into a training, validation and test set. The ...
0
votes
0answers
8 views

Testing Response Model

I'm thinking about strategies to examine how well a churn model performs. And, I know one question I'll receive is can we just target customers in the top decile of the churn model to evaluate model ...
1
vote
1answer
21 views

Tools in Data Science [closed]

I am a new one in Data Science and Machine Learning. I have some experience in Java, Python, SQL, Jupyter and most polular libraries like scikit-learn, numpy, pandas, tensorflow, keras. What tools ...
0
votes
0answers
12 views

Finding most related variables independent of variable type

Problem I am analysing a dataset containing variables of different types: continuous, ordinal and categorical. To prioritise in which order to analyse the variables, I would like to evaluate the ...
0
votes
0answers
28 views

Pointwise Mutual Information using spacy or just detailed explaination

So, I have been trying to play around with NLP recently and decided to work on a project involving Emotional Analysis. I have been following this particular research, http://www.cse.yorku.ca/~aan/...
0
votes
0answers
72 views

How to derive a ranking function by analysing feature correlations

I am analysing some employee details to find the efficiency of the employees. Ideally I want some rankings to rank them based on these features. My features include; current salary projects ...
0
votes
1answer
67 views

Comparing Strings Sequences of Different Length

What's the 'best' way to compare strings of characters of different lengths against an ideal sequential string of characters?
0
votes
0answers
21 views

determine suitable values for the parameters of the distance function for this graph

Hi I've been learning data mining and came across this question. I couldn't seem to figure it out myself. So we have a large single undirected graph(without attributes) G = (V,E) and want to detect ...
1
vote
0answers
17 views

How to use Machine Learning to discover important biomarkers in an unbalanced small data set

I have a project which I am just starting out, I am only just learning machine learning and statistics so I am somewhat unsure as to what approaches will be good to start off with, and I am sorry if ...
0
votes
0answers
12 views

Models for identifying anomalies in process data

We have a manufacturing production line with sensors collecting material weights at different locations along the line. The data structure looks like something below: We have noticed instances that ...
0
votes
1answer
23 views

What kind of analysis to use for continues value data set [closed]

I'm new in the data mining field and I'm using RStudio to do the analysis. I have a dataset and I don't know what could be the best method to use. I've tried using correlation and I don't know if ...
0
votes
0answers
21 views

Optimation Association Rule Using Genetic Algorithm

i have association rules that i got from FP-Growth algorithm and i want to optimize that association rules using Genetic Algorithm so i can get the strong rules. Does anyone know how to bring the ...
0
votes
2answers
77 views

Partitioning Around Medoids: Choosing a cluster number larger than the “optimal” one?

I asked a number of 71 'experts' to sort 92 different psychological constructs based on their similarity. Based on their answers, I constructed a dissimilarity matrix. Initially, I wanted to analyse ...
0
votes
1answer
81 views

High Precision and low recall score for TF-IDF when using KNN algorithm

I have twitter data which is labelled with the sentiment(Postive, Negative, Neutral) and I have evaluated the performance of Tf-Idf and Doc2Vec feature extractor using the KNN algorithm and logistic ...
0
votes
1answer
28 views

Can one reverse-engineer and deduce the underlying data for a constrained (max/min) optimization problem?

If one has the result, the constraints are known, but one does not have the input data.