Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

288 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5 votes
0 answers
195 views

Determining the effect of number of likes

Let's say I have marketing data and I need to determine how effective the marketing is. The marketing strategy is to publish facebook posts at inconsistent intervals. The goal is to see how the ...
user avatar
  • 443
5 votes
0 answers
85 views

(Nominal) raters with no gold standard

A friend of mine took a document and broke it up into parts, then asked 5 subject matter experts to classify each part into nominal category A, B, C, D, or E. (I'm not sure yet, but D may be "All of ...
user avatar
  • 20.2k
5 votes
1 answer
165 views

Regarding the size of training data for building classifier

When we build a classifier, like SVM or Naive Bayesian, are there any generic rules or theoretical derivations on the size of training data set? For example, to train a SVM-based classifier, what ...
user avatar
  • 2,647
5 votes
0 answers
1k views

What are some useful data visualisation or data mining techniques to investigate horse racing forms?

I have a dataset of 13k horse races from four different tracks with an average of 11 runners per race. In all, there are 26k unique runners in the data. For each race, I know who came first, second, ...
user avatar
4 votes
0 answers
3k views

Intuitive meaning behind support, confidence, lift and conviction

I'm learning about association rules and came across the common interestingness measures support, confidence, lift and conviction. I'm interested in the intuition behind your decision-making process ...
user avatar
4 votes
0 answers
393 views

How to test the result of cSpade

I'm new working with sequences rules and I am little lost about what is the next step. I already generate the rules but I'm not sure how to test them in a new dataset, in order to verify them. Do I ...
user avatar
  • 41
4 votes
0 answers
148 views

Distributions of eigenvalues of random matrices: what can they be used for in data mining?

I've accidentally come across some papers discussing distributions of principal components of the sample covariance matrices. An example of such a paper is Johnstone, 2001, On the distribution of the ...
user avatar
  • 571
4 votes
1 answer
114 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
user avatar
  • 141
4 votes
0 answers
178 views

Record linkage when sources have different fields

I have read a little about record linkage, but it seems to me that a requirement is that all fields in both sources can be compared. For example, with sources A and B, an assumption is that we can ...
user avatar
  • 141
4 votes
0 answers
2k views

SVM classifier (with soft-margin) implementation in R, gamma value and quadprog

I'm trying to implement a Support Vector Machine classifier in R and I have to solve the optimization problem using the quadprog R package which solves problems of the form : $$min_b \frac{1}{2} b^...
user avatar
  • 141
4 votes
0 answers
2k views

How do I interpret this Weibull plot?

I was exploring Weibull analysis for understanding reliability of two specific specimen. I used the R package, "weibulltoolkit" and "survival" to get the plot in question. The dataset is big so I am ...
user avatar
  • 4,272
3 votes
0 answers
1k views

Algorithm to detect time series anomalies (outliers) (using Apache Spark)

I am currently new to machine learning and I will be working on a project that involves using a Machine Learning library to detect and alert about possible anomalies. I will be using Apache Spark and ...
user avatar
3 votes
0 answers
64 views

Datamining: Why use algorithms for large datasets instead of using predictions based on a sample?

A growing amount of cluster algorithms have been developed for using large datasets such as CURE or BIRCH or filtering methods for common k-means. What is the advantage of using such an algorithm ...
user avatar
  • 73
3 votes
0 answers
683 views

How to compare growth rates

I'm analyzing the growth rate of a data set, and am unsure what is the best approach to make sure the growth is independent from another data set. Taking a simple (fictive) example, I'd like to ...
user avatar
  • 141
3 votes
0 answers
71 views

Several questions about using PCA on large data

I am using PCA on a square matrix of pairwise distances between 6000 elements, where the columns can be viewed as variables and rows as observations. Here are some of my questions and concerns: 1) ...
user avatar
  • 31
3 votes
0 answers
368 views

Is Mean Deviation better than Standard Deviation for data mining?

I just read this article and it says Mean Deviation(MD) is more efficient than Standard Deviation(SD) when there are some errors in observations. Like in real practise. I don't know what 'efficient' ...
user avatar
3 votes
0 answers
243 views

Hyperparameter optimization in 6-dimensional continuous space

I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do ...
user avatar
  • 5,901
3 votes
0 answers
433 views

Estimating probability distribution function of data stream

Although a similar question exists, I couldn't find my answer. I'm not a statistician hence please neglect if some terminologies aren't correct and let me know if I am interpreting something wrong. ...
user avatar
  • 83
3 votes
0 answers
100 views

Data Mining Methods with Nested Data

I have three general questions that I am really struggling to answer: a) When we have nested data (e.g. employees nested in departments which are nested in companies which are in specific ...
user avatar
  • 135
3 votes
0 answers
5k views

3-4-5 Rule How to partition the sets?

In Data Mining course, we are taking 3-4-5 Rule to segments the data uniformly. I'm trying to understand these lines below, and how they are linked to the graph below too. If an interval covers 3, 6, ...
user avatar
3 votes
0 answers
80 views

Which technique to build a model returning a vector of values (in R)

In my current project I need to build a model returning a vector of actions for each observation. I need a suggestion which statistical technique is used in general in such cases. In a project, I ...
user avatar
3 votes
0 answers
153 views

Parameter space exploration

I do realise this question is quite specific and practical, but I seek for some general help which helps me progress further in my analysis. Let $y(\boldsymbol{x})\in\mathbb{R}$ be the function I'd ...
user avatar
  • 264
3 votes
0 answers
343 views

Mutual information/pointwise mutual information for measuring prediction

I want to measure how well I predict a vector $Y$ (vector not a label) for observation $X$. Both $X$ and $Y$ have the same set of features ($1\times n$). For that, I thought of "scoring" the ...
user avatar
  • 131
3 votes
0 answers
676 views

Features selection by filter methods for multivariate time series?

I have a data set in which the samples are multivariate (about 30 variable/features) time series. These samples refer to two classes. I would like to select the variables more relevant to discriminate ...
user avatar
  • 509
3 votes
0 answers
235 views

Concerns about housing data mining

Can anyone suggest me good papers related to housing data mining. I have googled for a while and couldn't find any recent paper. I am assuming that people have stopped doing any mining related to ...
user avatar
  • 6,187
3 votes
0 answers
57 views

Do we need to apply the same transformation of predictors on a test dataset?

I first divide the dataset into training (75%) and test (25%). Then fit a logistic regression model on training data set. When fitting the model, I did some modification on independent variable, such ...
user avatar
  • 1,287
3 votes
0 answers
2k views

Weka J48 decision tree problem

I have a CSV dataset which contains mean (Numeric), spread (Numeric), review (string), ...
user avatar
  • 39
3 votes
0 answers
5k views

Validation error less than training error -- implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
user avatar
3 votes
0 answers
54 views

Can a different indexing approach improve the relevance and efficiency of search engine results?

I am creating a search engine, with the corpus consisting of websites crawled through a webcrawler (Apache Nutch). I need the query searches to be both fast and relevant. So far, I have been trying to ...
user avatar
3 votes
0 answers
368 views

Data prep / variable creation for predictive models

I was reading a couple of the write ups from a Kaggle challenge: Here is one and another and it got me wondering about variable creation in data mining and why there seems to exist so few texts or ...
user avatar
  • 7,770
3 votes
0 answers
89 views

Determine optimal (recurring) frequency of discrete values

So I'm trying to determine a semi-efficient way to calculate the optimal (recurring) frequency of a set of data. The data only exists at random time periods, but it is assumed that the data set is ...
user avatar
2 votes
1 answer
27 views

Visualizing shared instances of p-values<alpha across large numbers of treatments

Assume a data table that presents the p-values of a large number of independent runs of a statistical hypothesis test. Each run represents a single test with two possible hypotheses (i.e., null and ...
user avatar
2 votes
3 answers
374 views

Mahalanobis distance between high dimensional arrays

As we know, the Mahalanobis distance (MD) is one of the distance metrics for measuring two points in multivariate space. In practice, I can compute Mahalanobis distance between two 1D arrays using ...
user avatar
  • 4,732
2 votes
0 answers
255 views

Is there a well established algorithm to match two documents on a semantic level?

I have a set of documents from a wide variety of topics and I would like to retrieve the ones that are more similar to a new document provided. A search based on common words is not good enough, so ...
user avatar
2 votes
0 answers
98 views

Decision tree ,information gain and overfitting

If i use the information gain in order to evaluate the best split in a decision tree, why using a binomial split reduces the risk of overfitting ? Is the information gain test misleading if we have a ...
user avatar
  • 383
2 votes
0 answers
366 views

Is Multiple Correspondence Analysis applicable to Multi-valued Categorical Variables?

I have a data-set containing only Categorical Variables. I needed to do Principal Component Analysis on the data set. Eventually, I found Multiple Correspondence Analysis and learnt it. But, in MCA, ...
user avatar
2 votes
0 answers
302 views

Lift measure for a frequent itemset

I found it's possible to add the lift measure to the quality measures of a frequent itemset in R returned by the Eclat algorithm: ...
user avatar
  • 151
2 votes
0 answers
514 views

Euclidean vs Manhattan distance behaviour in high dimension - curse of dimensionality

I have compared different distance functions by computing the average tf/idf distance between documents. My results show a range between $10-15$ for the Manhattan and a range between $1-1.5$ for the ...
user avatar
2 votes
0 answers
388 views

Clustering of high dimensional data

I am having a data set with 54 independent variables .Most of them are having zeros it resembles like sparse matrix .How to cluster this kind of data and is there any data pre processing like Box cox ...
user avatar
2 votes
1 answer
383 views

How to check cross validation scores for market basket analysis?

If I have a large set of transactions where in each I buy a set of goods and I want to do market basket analysis using either A-priori or FP Growth or any other data mining method, you typically get ...
user avatar
2 votes
0 answers
537 views

Why StackingRegressor doesn't catch the trend?

I just reviewed very good example of fitting StackingRegressor from mlxtend package. ...
user avatar
2 votes
0 answers
534 views

How to do feature engineering of real time data?

I have made a good linear regression model with following step: Data Integration Data normalization/scaling(data preprocessing & feature engineering) Model Building(using linear regression with ...
user avatar
2 votes
0 answers
134 views

Can Resampling be used for estimation and goodness of fit test?

I am trying to compare my data with empirical distributions. But I don't have enough data to cut them to estimation data and validation data. I am trying a resampling approach and would like to see if ...
user avatar
2 votes
0 answers
315 views

Supervised cosine similarity

Suppose we have some samples, each sample is with two vectors and the corresponding label. That is, it looks like ($\mathbf{u}_i, \mathbf{v}_i, y_i$), where $y_i \in \{0, 1\}$ We can calculate the ...
user avatar
  • 21
2 votes
0 answers
47 views

Leads and Ideas on Learning from disconnected data

Problem of learning patterns from disconnected data. I have two independent discrete variables, D1 and D2, with information similar to tables in the images. I do NOT have the broken down data for A+...
user avatar
2 votes
0 answers
170 views

What is the difference between Gradient Boosting Machines and Gradient Boosted Regression Trees?

I would like to know if Gradient Boosting Machines and Gradient Boosted Regression Trees are the same algorithm or different. If they are different algorithms what is the main difference between them?
user avatar
2 votes
0 answers
147 views

Libraries to search sentences which include semantically similar phrases to the query sentence's

Could you recommend libraries to search sentences which include semantically similar phrases to the query sentence's? More semantically common phrases included, more scores. I think I should be able ...
user avatar
  • 21
2 votes
1 answer
35 views

How do I choose the appropriate numbers of customers to be considered for cluster analysis?

I am currently doing a customer segmentation project in SAS. I have identified 2700 customers who are have made a purchase in each of the 4 years I am analysing. For the cluster analysis the more ...
user avatar
  • 133
2 votes
0 answers
167 views

How to find the correct spatial scale in landscape ecology?

I am currently studying the effect of organic farming on honeybee colonies. I have calculated the percentage of organic land in several buffer areas around the hives (from 100m to 3000m in 100m steps) ...
user avatar
  • 959
2 votes
0 answers
242 views

Using entropy to imputing missing value based on grey relational analysis and clustering

This algorithm contain three techniques : 1-fuzzy c-mean clustering 2-Grey relational theory 3-Entropy multiple imputation The frame work of this algorithm is as follows : My questions are about ...
user avatar
  • 325

1
2 3 4 5 6