Questions tagged [data-mining]

Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.

Filter by
Sorted by
Tagged with
3
votes
0answers
623 views

How to compare growth rates

I'm analyzing the growth rate of a data set, and am unsure what is the best approach to make sure the growth is independent from another data set. Taking a simple (fictive) example, I'd like to ...
4
votes
1answer
606 views

Is there ever an advantage to using raw data in supervised learning over normalized/standardized data?

I know sometimes it is necessary to normalize/standardize the predictors for supervised learning but is it fair to say that you can always do this without hurting performance? Or are there scenarios ...
3
votes
1answer
1k views

How does adding an irrelevant feature affect the accuracy of a model?

My main question is how could irrelevant features decrease the accuracy? Example: Lets say we have a feature and the data can be perfectly classified with 100% accuracy using SVM. How can the ...
0
votes
1answer
514 views

number of training data set and number of model parameters as well as under-fitting, over-fitting [closed]

In practice, there are scenarios such as Scenario 1: the number of training data is more than the number of model parameters; Scenario 2: the number of training data is less than the number of ...
2
votes
2answers
129 views

Which of these are the best Data Science courses for an recent graduate economist? [closed]

I have recently graduated from college in my country with a bachelor's degree in economics. In my country (Paraguay, a Latin American country) there is a big lack of data scientists/statiscians. I ...
2
votes
1answer
1k views

How to find the best set of parameters for XGboost

I am using XGboost as a learning engine and I am getting a good results with default configurations. Now, I want to improve the predication by tuning the parameters, however, the list of parameters is ...
0
votes
1answer
87 views

Mining Time to Event with no Censoring

I got a huge dataset, with 75 variables and over 800,000 observations. The target variable is "time to event", where the event is the withdrawal of the subject from a gambling website (days since ...
0
votes
1answer
100 views

Is there any Measure of Data similarity?

Lets say we have a data with 100 features of which only I have 10 features of uncorrelated data and rest 90 features are correlated to one of these 90 features(My problem is similar to these ...
2
votes
2answers
2k views

Extremely near centroids in k-means?

In the k-means algorithm, what happens if two of the initially chosen centroids are extremely near to each other? Say I have two centroids c1 and c2, and d(c1, c2) ~ 0, i.e. the distance between c1 ...
0
votes
1answer
253 views

Predicting profit drop and identifying patterns that lead to it

I have telecom line item (invoice) data. which looks like this: The data has a monthly granularity and in total there is 6 months of data. By summing up all the line items for a subscriber for every ...
0
votes
1answer
1k views

Data mining - trouble with computing error rate [closed]

.. Hi, guys. I have a problem at the uni with homework solutions that were presented to us. I tried solving the problem, but the solutions we got back from the teacher were in a way the exact ...
1
vote
1answer
135 views

Data Preparation

I faced a mixed data set which contained both continuous and categorical variables (totally more than 200 variables). Now I have chosen 60 variables out of them by the business specification. Then is ...
2
votes
2answers
133 views

How to deal with data where for most of the features, most of the rows have empty values

I am solving a classification problem. The final dataset has around 50 - 60 features and around 12 K - 15 K rows. Each distinct row representing distinct ID. Now the problem is that for each of these ...
4
votes
0answers
392 views

How to test the result of cSpade

I'm new working with sequences rules and I am little lost about what is the next step. I already generate the rules but I'm not sure how to test them in a new dataset, in order to verify them. Do I ...
1
vote
2answers
942 views

Can apriori algorithm be applied to an extremely small dataset effectively

I have a medical data set of just 50 samples and I need to study the relationships between several features. For this, I want to use the apriori algorithm but I am not sure whether it will be ...
2
votes
1answer
47 views

Measure the ability of variables to differentiate between two groups

I have binary, categorical, interval, and metric variables. For each of them, I would like to find a measure that helps me decide on how well the variable is able to help predicting a binary target ...
1
vote
1answer
208 views

Understanding the quality of the KMeans algorithm

After reading Unbalanced factor of KMeans, I am trying to understand how this works. I mean, from my examples, I can see that the less the value of the factor, the better the quality of KMeans, i.e. ...
1
vote
2answers
191 views

How to approachfor fit, train and test the Customer Lapsing Machine learning model

I was exploring the options to design the approach to train test and predict the customer lapsing model. Description: Lapsed => Customer didn't purchase an item in the financial year (ex: Customer A ...
2
votes
0answers
165 views

What is the difference between Gradient Boosting Machines and Gradient Boosted Regression Trees?

I would like to know if Gradient Boosting Machines and Gradient Boosted Regression Trees are the same algorithm or different. If they are different algorithms what is the main difference between them?
2
votes
0answers
143 views

Libraries to search sentences which include semantically similar phrases to the query sentence's

Could you recommend libraries to search sentences which include semantically similar phrases to the query sentence's? More semantically common phrases included, more scores. I think I should be able ...
0
votes
0answers
30 views

Diagnosis analytics

I am studying an analytic use case concerning exceeding energy equipments detection. As a beginner in data analytics I am not sure how to begin to resolve this problem. The use case is resumed as ...
1
vote
1answer
2k views

Is this big difference meaningful? 63% Rand index, but 0,004 Adjusted Rand Index

I have a data set n=175 and for 2 different clustering (A and B) I have 5 and 6 clusters. The table for similarity of clusterings is below. First I calculated the Rand Index both manually with Excel ...
2
votes
1answer
46 views

Can tools like SPSS find out which columns correspond to a certain data range

This is a beginner's questions about statistical tools. We have a spreadsheet with many columns from a survey. The first few columns describe each individual: Gender (discrete values Male, Female) ...
35
votes
5answers
5k views

Think like a bayesian, check like a frequentist: What does that mean?

I am looking at some lecture slides on a data science course which can be found here: https://github.com/cs109/2015/blob/master/Lectures/01-Introduction.pdf I, unfortunately, cannot see the video ...
6
votes
1answer
24k views

Association rules - support, confidence and lift

I am trying to mine association rules from my transaction dataset and I have questions regarding the support, confidence and lift of a rule. Assume we have rule like {X} -> {Y} I know that support ...
1
vote
1answer
497 views

Is there a way to automatically identify the best categorization of a continuous variable?

Say I want to model the income from people's age, and we know the effect of age is not continuous, despite it being a continuous variable, and an obvious thing to do is to categorize the age variable. ...
1
vote
1answer
34 views

what is the relationship between these two topics?

I would like to know what is the relationship between multivariate analysis and other topics such as: linear regression, neural networks or support vector machines. According to Wikipedia: ...
5
votes
2answers
1k views

Financial Slang and NLP for Sentiment Analysis

I am working on Sentiment-Analysis/Opinion-Mining of Tweets, focused on Finance related tweets. One of the biggest issues I am facing is the unability of my algorithm to detect equivalent entities (...
3
votes
0answers
65 views

Several questions about using PCA on large data

I am using PCA on a square matrix of pairwise distances between 6000 elements, where the columns can be viewed as variables and rows as observations. Here are some of my questions and concerns: 1) ...
0
votes
1answer
929 views

Plotting predicted MSE as a function of the training sample size for choosing $K$ in $K$-fold CV

From Principles and Theory for Data Mining and Machine Learning, Clarke et al.: One strategy for choosing $K$, if enough data are available, is to plot the predicted MSE as a function of the size ...
1
vote
0answers
515 views

Lift Charts in Multiple Linear Regression

When lift charts are generated in a Multiple Linear Regression model, for example, in predicting a continuous variable such as price of a car, how can they be explained in evaluating the performance ...
0
votes
0answers
44 views

Can I split a table through the correlation of its 2 columns?

I've got a 2 column table (X, Y). Every row of the table says that the X on that row use the Y specified in the same row. This are the sample data. https://jpst.it/L6I5 I'd like to cluster the table ...
0
votes
1answer
447 views

How do apriori based algorithms avoid finding spurious relationships in their rules?

I'm taking a class on data mining and of course, we went over the Apriori algorithm (Fast Algorithms for Mining Association Rules). Something that I want to know is how do apriori based algorithms ...
1
vote
0answers
51 views

statistics metrics or machine learning methods to quantify the relationship between a point and a group

There are several data groups: A1 = { A11, A12, A13, …, A1n} A2 = {A21, A22, A23, …, A2n} .. … … Am = {Am1, Am2, Am3, …, Amn} Here ...
1
vote
1answer
1k views

Clustering: In which cases would using single link, average link and complete link give me the same clusters?

When would I use each and when would they give me the same results?
3
votes
3answers
859 views

Neural network library for Python for Microsoft Windows [closed]

I have been having trouble with selecting a good library for Neural network algorithms in Python. TensorFlow isn't supported on windows. Theano is still in the beta phase of development. PyBrain too ...
1
vote
1answer
60 views

Using different prediction models/algorithms for different subsets of dataset

Is there practice in data mining and machine learning where different parts of dataset are predicted using different algorithms/models? The logic is that some data samples are better predicted with ...
1
vote
1answer
27 views

Merging information of several events

I'm working in a database related to endometrium ultrasound. My DB contains several columns that may describe one or more injuries (scar tissues) by dimensions and volume: Injury1Height, Injury1Width, ...
1
vote
1answer
56 views

Which graph/network format should be used when releasing a data set?

Under RENOIR (an EU funded project) I am working on releasing the data set of Slovenian Press Agency for researchers and other folk. As part of the data set we will be releasing some graphs with ...
0
votes
0answers
90 views

changepoint detection and its analysis

I applied changepoint detection provided by ecp package against a given time series. The time series plot marked with identified change points is shown as follows. ...
2
votes
1answer
33 views

How do I choose the appropriate numbers of customers to be considered for cluster analysis?

I am currently doing a customer segmentation project in SAS. I have identified 2700 customers who are have made a purchase in each of the 4 years I am analysing. For the cluster analysis the more ...
1
vote
0answers
304 views

How come “”mathematically“” that lasso shrinks the coefficient towards zero whereas ridge does not? [duplicate]

I understand them geometrically !
2
votes
1answer
86 views

How are the outcomes that generated from different predictive models combined to get more accurate predictions?

The simple average is commonly used to combine the predictions of different predictive models. Apart form the simple average, what are the other methods that can be used for combining the predictive ...
1
vote
1answer
320 views

Understanding Partial Correlation [duplicate]

I'm having some trouble fully understanding partial correlation and I was wondering if some of you can shred some light on my confusion. Let's consider the following scenario: It is a known fact that ...
4
votes
3answers
4k views

pattern recognition for sequence data

There are a lot of data sequences, I am trying to find pairs of sequences that are similar with other. Trivially, we can define some distance measure, and compare each pair of sequence in terms of ...
2
votes
1answer
8k views

Applying DBSCAN to a huge GIS dataset with a Haversine distance metric.

I have a training set (2GB) that contains GIS trajectory data for multiple taxi rides. I want to cluster the final destinations based on their spatial density and have therefore been trying to use the ...
-2
votes
1answer
164 views

Merge column vectors from the loop in to a single matrix *.mat in matlab [Solved] [closed]

I have an iteration of my algorithm and it gives me a column vector as a result (1899x1 double). I want to form a new matrix composed of these column vectors together sequentially (for example, the ...
1
vote
3answers
1k views

how to predict sales of an item changes based on a discount given to another item?

I am developing a system where the management of the supermarket can make decisions on past sales data. There I mainly want to focus on how to predict sales of an item changes based on a discount ...
3
votes
2answers
4k views

What is the difference between the classification and the pattern recognition?

Could somebody explain this difference just as simple as possible?
1
vote
1answer
66 views

Is the likelihood statistic applicable for model selection in machine learning?

Minimising the likelihood ratio statistic is often used as a criterion for model selection in connection with linear and related models and statistics such as as AIC are an extension of this practice, ...

1
5 6
7
8 9
24