Questions tagged [data-mining]
Data mining uses methods from artificial intelligence in a database context to discover previously unknown patterns. As such, the methods are usually unsupervised. It is closely related but not identical to machine learning. Key tasks of data-mining are cluster analysis, outlier detection and mining of association rules.
1,183
questions
0
votes
0
answers
81
views
Domain knowledge or data dredging?
I have a binary classification problem where I'm required to classify transactions as anomalous or normal (1 or 0 respectively), with anomalies being the rarer instance.
With what I know to be true ...
- 1
1
vote
0
answers
76
views
Data Science: Using Inferential Statistics to label train dataset
Lack of High Schools in remote areas is a problem for students in developing country. Students in some locations are better than that in other. So, I have to find those locations. Now, the main ...
- 13
2
votes
2
answers
11k
views
Machine Learning model on dataset with mainly zeros
I have a text field dataset. Each observation counts the number of appearances of that particular word, and the columns (variables) are most frequently appeared words. Within each column, zeros ...
- 97
0
votes
2
answers
2k
views
machine learning model true negative rate is too low while true positive rate is too high
I am using the tm package and h2o package to do text mining using neural network. Here I have a data frame of 100 most frequent words in the text. These variables only have values of non-negative ...
- 97
0
votes
2
answers
670
views
Normalization ,standardization, or do nothing
I have medical data with max value 500 along with values like age and binary values for sex (0 or 1).
I will use clustering to find the number of clusters.
Which is the best approach among three....
3
votes
1
answer
149
views
Data Science Interview Question - Build a model to Predict the Class of the Output [closed]
I recently attended an Interview and I was asked this question by the Interviewer.
Question : There are 100 books, 90 of the books belong to category A and 10 of the books belong to category B. ...
1
vote
1
answer
2k
views
guide for text classification using weka
I have a set of 2000 small texts (each less than 500 words) that I manually categorized. All the texts are in the same main subject, and I want to separate them into distinct groups based on their ...
- 11
5
votes
1
answer
5k
views
why don't we test error on autoencoders?
we are not suppose to use test set when we r fitting the model to our data.
but I noticed in this blog https://blog.keras.io/building-autoencoders-in-keras.html which is very established it uses
<...
- 151
0
votes
0
answers
21
views
Algorithm to handle positive rating if there is huge difference between the number of reviews
We have two products P1 and P2. P1 has total 1000 number of reviews and p2 has 200 reviews.
Positive rating of P1 is 85% (85% users are saying it's a good product) Positive rating of P2 is 90% (90% ...
- 101
1
vote
1
answer
304
views
Recommendations for textbooks covering current data mining/machine learning techniques for fraud detection?
I work in the health insurance field, but a general treatment of fraud detection methodologies would still be helpful.
So far I've discovered brief articles outlining particular techniques used in ...
- 5,342
0
votes
1
answer
2k
views
Designing a simple tag-based recommender system
I want to design a system for product recommendation, where I have the following:
~1000 products that are tagged with 1 to ~10 descriptive tags by professional product specialists.
~200000 users, ...
- 1
1
vote
4
answers
315
views
How to treat incomplete variable values
I'm trying to analyze some fairly sparse data on a recurrent medical symptom, and I don't know what to do with two entries where my data is incomplete.
My overall goal is a bit vague: it's to find a ...
1
vote
0
answers
106
views
Supervised Multi-Classification with small dataset
I've got 700 data rows with about 6 features and there are 300 different class labels. Most of the classes have got only 2 data rows (ca. 250 classes).
I know there are too many classes for this data ...
- 11
6
votes
2
answers
3k
views
Are XGBoost probability outputs based on the number of examples in a terminal leaf
I am trying to replace a c4.5 tree that someone else implemented with a boosted tree (XGBoost).
The data is extremely skewed and the company wants the new model to output similar distributions.
c4.5 ...
0
votes
0
answers
798
views
a question about splitting data into TRAIN/TEST set
Can we "trust" about this method when we have small data set ? Is there a minimum size required to partition the data ? e.g. N=200 ?
- 287
1
vote
0
answers
59
views
How can I visualize/analyze data that has percentages and numbers across multiple time periods?
I have a large dataset like this:
...
3
votes
0
answers
1k
views
Algorithm to detect time series anomalies (outliers) (using Apache Spark)
I am currently new to machine learning and I will be working on a project that involves using a Machine Learning library to detect and alert about possible anomalies. I will be using Apache Spark and ...
0
votes
1
answer
468
views
Extracting Part of Speech (Source and Destinations) using text mining/NLP?
I need to extract the source and destination terms from the text
documents using text mining/NLP/Information Retrieval ?
ex :
i am travelling from New York to London.
i am heading towards ...
- 111
1
vote
1
answer
78
views
Is it possible to forget data when computing sufficient statistics on a stream?
My understanding is when we are computing sufficient statistics on a stream, when a new instance arrives, the value of the new instance is added to the already computed sufficient stats. so there is ...
- 45
0
votes
0
answers
32
views
build a model capturing related features for multiple events
There is a data set, which has about 20000 data points. Each data point has about 100 features. At the same time, there is an instance set, including 60 different instances. Right now, there is a ...
- 5,034
3
votes
1
answer
2k
views
What is the confidence for rule for $\emptyset \implies A $?
I'm looking at a question for association rule mining and this comes up:
What is the confidence for $\emptyset \implies A $?
What is the confidence for $ A \implies \emptyset $?
Given:
$$\...
- 133
2
votes
1
answer
48
views
Privacy preserved record linkage
What is the best way for privacy preserved record linkage in data mining. I am final year university student. My thesis topic is related to data mining. I am working with data and I need to privacy ...
- 21
0
votes
0
answers
29
views
Alternatives to classification for natural text data
I'm working at a project that would like to do topic extraction / classification from data, consisting of various NL sources (tweets, social network updates, pastebin). Data are very diverse in ...
- 306
1
vote
1
answer
888
views
Multiway Array Aggregation
I'm having trouble understanding how the calculations for multiway array aggregations work using the example used in https://docs.google.com/file/d/0B5Ju2x50v6l5X0p3WDEwNkNTVW8/edit at page 165.
...
- 113
0
votes
0
answers
39
views
Building a Diagnostic System for Psychiatric Disorders
I am an undergraduate with near-infinite passion to the theoretical machine learning and ML applications. Inspired from my challenging mental disorders, I am really interested in building a ...
- 211
0
votes
1
answer
135
views
visualize a multi column matrix
I would like to visualize a 2000*100 matrix to find how different variables change based on values.
Can you suggest a way doing it using R or Matlab?
I have data from patients who use an ECG device ...
- 13
1
vote
1
answer
4k
views
Classifier for continuous data?
I am trying to classify a topographic cross section (profile) using a machine learning method. The classification consists of 2 main classes (scarp, no scarp) and 3 sub-classes (cls1, cls2, and cls3) ...
- 229
0
votes
0
answers
686
views
Goodness-of-fit vs maximum likelihood for logistic regression?
From what I understand, maximum likelihood is used to estimate a parameter alpha in a way that maximizes the probability P(Y=|x,alpha) for example. It is used for logistic regression in order to get ...
0
votes
1
answer
740
views
How do I calculate test error for adaboost?
I've calculated an adaboost algorithm for 20 iterations with a decision tree as my weak learner. I want to make a graph that plots the training error and the testing error. I have the training error,...
1
vote
0
answers
275
views
Why do we convert data from long-form to wide-form for analysis?
I convert data from long-form to wide-form for my thesis. It makes sense intuitively for me to do so as I can easily see the relationship between every variable. However I was wondering is there a ...
- 143
1
vote
1
answer
149
views
Becoming a data scientist using only StackExchange sites - what questions should I look at? [closed]
If I had a maths degree with a little foundation in statistics, what would be the top $100$ questions/posts on CrossValidated or MathStackExchange or MathsOverflow or Stack Overflow that I would have ...
1
vote
2
answers
296
views
Is it reasonable to use rules to label positive samples when doing fraud detection in machine learning?
We use supervised learning algorithm to detect fraud, but we have fewer or even no positive samples, is it reasonable if we use rules to label positive samples? if so, is the supervised learning model ...
- 11
6
votes
2
answers
11k
views
What's the difference between Leave-One-Out and K-Fold Cross validation?
As far as I know in K-fold cross validation the samples are split into k sets and at round k-1 of these are used for the training of the model and the last one is used for testing the model and ...
- 163
1
vote
1
answer
70
views
What kind of analysis can someone perform given a swap rate curve of a currency pair at different times?
The dataset that I have looks like the one attached - any ideas on how data like this can be used to extract anything meaningful?
I would appreciate any help or ideas you might have.
Thanks in ...
- 63
0
votes
1
answer
3k
views
How do weka classifiers deal with missing values? [closed]
I tried using a training set that has missing values. I applied filters (like replace missing data) and then after there were no more missing data I applied naive bayes, trees etc... I thought this ...
0
votes
0
answers
26
views
What Statistics Can I Get from This Grading Procedure
I am a mathematician, not a statistician. So, I am hoping this question is not too broad. I'll delete if the community wishes.
I recently read on Terence Tao's blog about a way to give partial ...
- 101
1
vote
1
answer
65
views
How can I rate the difficulty of a question in an adaptive learning platform?
I'm working on an adaptive learning platform with some friends for a project that helps students to prepare their university admission test.
We provide different kind of practice sessions and we have ...
0
votes
1
answer
147
views
How to determine recurring use numbers for my app
A few months ago, I wrote an Android App. I am getting two key pieces of data from this app regarding its usage. The first metric, is number of downloads per day. The second metric, is basically, ...
1
vote
3
answers
4k
views
binary and multiclass classifiers
I have a simple yes/no problem so I was naturally inclined towards using a binary classifier because I was reading the book, A Course in
Machine Learning by Hal Daumé III and I quote from it:
[ Binary ...
0
votes
1
answer
3k
views
Classification and mixed categorical and numeric variables
I've been working a little with weka and so far I haven't made my own database to apply a classifier but I've tried to look at the already existing files and from what I've seen there is absolutely no ...
3
votes
1
answer
2k
views
Naive Bayes and independence
In every example I see(spam, negative vs positive tweet , weather study...) there is always the assumption that the input features (or variables) are independent.
In order for me to be able to ...
1
vote
0
answers
133
views
What is the best way to make use of numerical variables together with text mining for classification purposes?
I have a document-term matrix and I was able to classify the documents into their respective classes using svm.
Now, I have numerical variables associated with the documents which I think might ...
- 11
0
votes
1
answer
280
views
what are mixed variables in data mining?
I read that neural networks, SVM and neuro-fuzzy don't support "mixed variables."
So what are those exactly? Does it refer to mixed types (numeric and non-numeric)?
And if so, does that mean the ...
2
votes
1
answer
260
views
What would be the final hypothesis like? if our unknown target is a distribution rather than a function?
The above picture is about building model, it seems a bit difficult to understand the meaning of "plus noise", and what would the final hypothesis look like? if the unknown target changes from ...
- 1,623
0
votes
0
answers
737
views
Classifiers for small data sets and low dimensional features?
I have a small data set (under 100 samples) and 5 input features. I often hear about how neural networks are prone to overfitting under such conditions and that naive Bayes is likely to underfit.
...
0
votes
1
answer
36
views
How do data mining classifiers behave when we add training samples to previously established training models?
I have yet to choose a classifier to get my model but I wanted to see which possible classifiers fit what I'm looking for.
In fact, I need an algorithm to establish offline training this way:
The ...
-1
votes
1
answer
82
views
How do big companies apply machine learning? [closed]
I was wondering how recommendation on youtube work for example? How are the algorithms applied, because every user gets different recommendations depending on his location, his past liked videos etc......
1
vote
1
answer
197
views
Classification vs regression machine learning?
While studying data mining methods I have come to understand that there are two main categories:
-Predictive methods:
classification
Regression
-Descriptive methods:
Clustering
Association rules
...
2
votes
0
answers
557
views
Why StackingRegressor doesn't catch the trend?
I just reviewed very good example of fitting StackingRegressor from mlxtend package.
...
- 243
-1
votes
1
answer
75
views
How to determine which columns/values in a dataset/dataframe can be Predicted which cannot?
I have a data-set/data-frame with columns Description, Department Name, Priority, Doctor name & Location.Description data comes as free text from from the UI.
Based on historical description's in ...
