Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining ...

learn more… | top users | synonyms

1
vote
0answers
17 views

Determining the effect of number of likes

Let's say I have marketing data and I need to determine how effective the marketing is. The marketing strategy is to publish facebook posts at inconsistent intervals. The goal is to see how the ...
0
votes
0answers
5 views

Prediction - important features from a new element

On my data I did LSI and got a large matrix (>200000 samples, >6000 features, very sparse). I do SVD on it, keeping only 150 dimensions. When I get a new element, do a folding-in, calculate cosine ...
3
votes
0answers
42 views

What challenges can data science help with in small businesses

Data science can solve/help with many challenges. But it usually looked upon as something for multi-million dollar corporations. I'm looking for examples of challenges data science helped with in ...
1
vote
0answers
10 views

Artificial Data Generation based on Data Distribution

My data looks something this (for example): Salary | Age | Zip | Class 60000 | 35 | 5 | Yes 50000 | 52 | 4 | No 10000 | 25 | 3 | Yes 70000 | ...
0
votes
1answer
16 views

How to implement knn in r with missing values?

I have this data set from https://archive.ics.uci.edu/ml/machine-learning-databases/credit-screening/crx.names which gives a good summary of the attributes im using. Some of the observations are ...
2
votes
0answers
32 views

Estimating probability distribution function of data stream

Although a similar question exists, I couldn't find my answer. I'm not a statistician hence please neglect if some terminologies aren't correct and let me know if I am interpreting something wrong. ...
0
votes
0answers
8 views

Interpreting the lift curve

Suppose we have two classes: A and B. Suppose we use a logistic regression to assign each unit to A or B. The curve lift is calculated through this formula: $\frac{n_{22}/n_{.2}}{n_{2.}/n_{..}}$ ...
0
votes
0answers
7 views

How can I find Matching between a data set with given data set groups

I have a variability data new one, I would like to find the matching between given data set with group of data sets, which one among the data sets group has matched with the new data set. eg: A, B, ...
3
votes
1answer
40 views

Predict user behaviour with constantly changing input variables

How to work on building an engine for a website wherein we want to score/recommend stuff based on her different activities, like the music she rated or the article she read, or whether email ...
1
vote
2answers
57 views

How to Balance my Dataset?

I have 90% negative examples and 10% positive examples,(13,000 observations, 90 Variables). my model shows me that the miss classifications error is 0.1 but my confusion matrix shows me that the TP ...
0
votes
0answers
21 views

Carry out a 2-Sample t-test whether the means of two independent groups differ

I have carried out a paired t-test on the labor.arff data set and found out that one classification is not statistically better suited for this data set than the other. This is represented by the ...
0
votes
0answers
12 views

Changing the class file from nominal to numeric in order to carry out the linear regression [closed]

I am trying to perform the LinearRegression classification on the diabetes.arff and glass.arff data sets, with little luck. The option is greyed out and I cannot ...
2
votes
1answer
77 views

How can using Logistic Regression without regularization be better?

I'm using this Java machine learning library: https://sites.google.com/site/qianmingjie/home/toolkits/laml From the library I'm using Logistic Regression: ...
0
votes
1answer
27 views

Choose software tool for basic analysis [closed]

(I am pretty novice in data analysis) So, I have a set of elements (~105). All elements splitted for some classes of disjoint sets. Actually, there are two systems of sets. ...
1
vote
0answers
12 views

Standard deviation comparison for splitting clusters in ISODATA

I am currently implementing the ISODATA algorithm and I am new to cluster analysis as I just learnt about it. I got stuck at the step which I need to compute the standard deviation of each cluster, ...
7
votes
2answers
122 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
1
vote
2answers
43 views

Knn classifier for Online learning

Is Knn classifier suitable for online learning i.e. Is it effective to apply online learning approach for knn classifier?
0
votes
0answers
37 views

Which algorithm fits best to a transactional data classification problem?

I have transactional database - the data looks as follows: ID - COLUMN1 - COLUMN2 - COLUMN3 0 - A - B - C 0 - A - D - C 0 - E - B - C 1 - A - B - C 1 - A - B - C 2 - ...
0
votes
2answers
40 views

What is the best algorithm to find similar text documents?

I have many text documents and I would like to find similar documents to each document within my data set. Is Latent Dirichlet Allocation (LDA) the best way to do that, or are there other algorithms ...
1
vote
0answers
21 views

Feature selection in clustering

I am looking for a method for feature selection in Gaussian Mixture Models. I have a dataset with 2000 records and 40 variables. I tried to use the "clustvarsel" package in R, which use the BIC as ...
0
votes
0answers
21 views

How do I weight inputs to a regression model so that one figures into the model more than the other?

I have obtained a series of weights from a text mining algorithm. Unfortunately, my algorithm is not capable of doing certain tasks that are too similar without some sort of regression analysis, say ...
1
vote
0answers
18 views

Decision tree learning - gini impurity

Suppose we want to know whether Joe will play football today. We have a dataset containing information about weather such as outlook, wind, and humidity and the decision he made (to play or not to ...
3
votes
1answer
71 views

What's the meaning of dimensionality and what is it for this data?

I'm doing my assignment for my "Modeling and Optimization" course, and now I have doubts on the first question: What is the dimensionality of the data? What are the min, median, max, mean, ...
1
vote
1answer
26 views

What is the difference between A/B Testing and Randomized Control Trials?

The question is as the title says: what is the difference between A/B Testing and Randomized Control Trials?
-1
votes
2answers
63 views

Meaning of principal components

I have difficulty understanding the meaning of the Principal Components (PC) - On one hand, PC are computed by finding loading vectors that maximize the variance, but on the other hand I read ...
0
votes
0answers
19 views

Term importance from document with predefined weight

I am having a set of document with different value weight on them. I am trying to understand which term from the documents trigger the highest values. I have a theory on how to do it and I would like ...
1
vote
0answers
10 views

Practical applications of dimensionality reduction methods: Filtering, Wrapper, Embedded models

Filtering is basically sorting some features and picking top performing ones. Where wrapper in wrapper we go through unused/used features add/remove some of them and test performance over validation. ...
0
votes
0answers
23 views

k-means random initialization for very-large dataset, is it good enough?

I've got a question in clustering using random k-centers. I ran the k-means algorithm for 10 iteration, for some 100 rows taking 9 random initialization of centroids from the data set itself. The ...
-1
votes
1answer
31 views

Data mining or data visualization?

My basic doubt is what is difference between Data mining and Data visualization? does they have different algorithms? I have a machine data stored in database. I have to develop an app which will ...
2
votes
2answers
42 views

Is parametric equivalent to linear?

Some supervised learning techniques, such as GLM (e.g., logistic regression), are linear and parametric. On the other hand, one of the claimed advantages of nonparametric supervised learning ...
2
votes
1answer
53 views

Meaning of the Boosting algorithm for Regression Trees

I have a problem with understanding the concept of the Boosting Algorithm. ...
1
vote
1answer
41 views

out-of-bag error estimate for Boosted Trees

In Random Forest, each tree is grown in parallel on a unique boostrap sample of the data. Because each boostrap sample is expected to contain about 63% of unique observations, this lefts roughly 37% ...
1
vote
0answers
25 views

how to perform divisive hierarchical clustering

I've been trying for a long time to figure out how to perform (on paper) the divisive hierarchical clustering algorithem, however I'm not able to understand how to do it exactly. example: I need to ...
1
vote
0answers
34 views

Huge overfitting with Random Forests and Boosted Trees?

In the following picture, the boxplots represent a performance metric (the closer to 1, the better) recorded for 50 runs of cross-validation, and the black filled circles are the training values of ...
0
votes
0answers
4 views

How do we mine associations from sequences?

My data mining problem is a next web page prediction using the existing web data. For that I have a set of frequent sequences which are obtained using cspade algorithm in R. Now I am not sure how to ...
2
votes
1answer
79 views

R: Finding relationships between 2 variables to determine any patterns in data

I am working on finding relationships/patterns between 2 variables (Type_A, Type_B). ...
5
votes
1answer
93 views

Do CART trees capture interactions among predictors?

This paper claims that in CART, because a binary split is performed on a single covariate at each step, all splits are orthogonal and therefore interactions among covariates are not considered. ...
3
votes
2answers
78 views

Are Random Forests and Boosting parametric or non-parametric?

From this excellent paper by Breiman, we can seize all the difference between traditional statistical models (e.g., linear regression) and machine learning algorithms (e.g., Bagging, Random Forests, ...
1
vote
0answers
18 views

How could these two simple Bayesian algorithms be explained, simply? [closed]

count(this token in class) + 1 / count(all tokens in class) + count( all tokens ) and ...
5
votes
1answer
55 views

data mining methods/algorithms for fraud case

I recently got into a topic regarding fraudulent transactions. I am relatively new to data mining and just looking for some input for my case here. I started with a cluster analysis / anomaly ...
1
vote
1answer
18 views

Using Diebold-Mariano test to compare predictive errors in non-time-series?

I understand that the DM test is established for time series data, but could I still apply the test for non-time-series data? Could I simply replace the autocorvaiance part of the test statistics with ...
1
vote
2answers
46 views

Gini index - formal or heuristic?

Gini index is quite often used in constructing decision trees in data mining for attribute selection and attribute split point. Is Gini coefficient just a heuristic or can we formally explain why ...
2
votes
1answer
71 views

Simple way for histograms classification

I'm trying to classify a histogram. I have 4 classes and I generate 4 histograms (h1, h2, h3 and h4) for each class. Each histogram contains 10 bins (attributes describing an object) on the x-axis and ...
0
votes
0answers
37 views

Get common items when a column has a specific value?

I' have an excel sheet with n columns, these columns contain info about the students. For admission we have the score of a test from school which contains different areas(Math, biology, physics, ...
1
vote
0answers
45 views

Sample time series to equal interval

I have data with timestamp and associated values. time interval between two consecutive data is not constant. How to standardize the the time series and associated value ? eg- Input data is Timestamp ...
0
votes
0answers
20 views

PCA percentage calculation

Please kindly iam confused with PCA percentage calculation , according to the equation that provide , I divide each eigenvalue by the sum of them.then i multiply each one by 100. for example my ...
0
votes
0answers
7 views

How can be assesed that a given data representation is better than the other?

Given a classification dataset, suppose I learn many different data representation with Matrix Factorization, Clustering or with such approaches. At the end , how would I decide which is better than ...
-1
votes
1answer
41 views

Clustering Data of 8 dimensions

I am working on a data clustering and don't know how I can achieve it with R ! I am working on a data set of 50 observations each of 8 variables. What i want is to have clusters gathering the ...
1
vote
1answer
22 views

adjustment of lift measure

Lift is a measure widely used in many domains. However, it is known to have a problem for infrequent counts. What are the solutions for this type of problem? In frequent pattern mining hyper-lift was ...
1
vote
0answers
12 views

Parallel association rule mining

I am following papers about parallel association rule mining, in particular, this paper. I do not understand how conditional FP-Tree is generated in the paper, ...