# Tag Info

7

Bootstrapping is a concept in statistics of approximating the sampling distribution of a statistic by repeatedly sampling from a given sample of size $n$. We construct $B$ samples, each of size $n$, by sampling with replacement from the original sample. The statistic of interest is calculated for each of the $B$ samples. For sufficiently large $B$, we have a ...

3

In some sense I think this question is unanswerable. I say this because how well a particular unsupervised method performs will largely depend on why one is doing unsupervised learning in the first place, i.e., does the method perform well in the context of your end goal? Obviously this isn't completely true, people work on these problems and publish results ...

3

The symptoms you are describing sound like a classic case of overfitting. As described in the link, over time, researchers have managed to come up with various strategies such as pruning or early stopping to try to avoid or reduce the effects of overfitting. Depending on which specific learning algorithm you are using, and the details of how it was ...

3

Calibration and "data dregding" are not really differentiated by statistical tests, but by motives. If you are simply looking for patterns in an undirected manner, then you are data dredging. However, if you have a well formulated model that is known to capture the general features of a phenomenon, save for the specific values of some parameters, then what ...

3

Wikipedia has a good article on the topic, complete with formulas. The values in your matrix are the term frequencies. You just need to find the idf: (log((total documents)/(number of docs with the term)) and multiple the 2 values. In R, you could do so as follows: set.seed(42) d <- data.frame(w=sample(LETTERS, 50, replace=TRUE)) d <- ...

2

the point is to pick a value of the calibration parameter that gives optimum model correlation In that case, the question is about model optimization, which is related to, but usually not considered the same, as data dredging. Fortunately, there are "recipes" that you can follow to avoid the problems associated with the dredging. The big problem ...

2

Gini index here ($G$, say) just calculates diversity or heterogeneity (or uncertainty if you will) from the sum of squared category probabilities. If every value is in the same category, then the measure is $1 - 1^2 = 0$. If every value of $n$ values is in a distinct category, then the measure is $1 - n(1/n)^2 = 1 - 1/n$. The complement is in some ways ...

1

I think this is highly dependent on the classification approach that you chose. Many classifiers can get stuck in local optima that represent non-ideal parameter spaces. Another possibility is the scale of your normalization may be different between the feature sets thus causing issues with the classification.

1

Software The software you list (SPSS, SAS) are statistics packages. They are hardly suitable for actual data mining (kernel methods, neural nets, deep learning, ...). That said, you could easily replace both of those by R which works perfectly on any platform. In terms of data mining software, to my experience, you should be looking at things like Python, ...

1

I think your question should be split in two parts: Which OS should you use? This is a very controversial topic and it depends on the tools you will be using. I myself prefer to work on a Unix based system (i.e. Linux/Mac). However, since most of the software you mentioned are designed for MS Windows, I would recommend going for that option. ...

1

Cross validation is used to select your model. The out-of-sample error can be estimated from your validation error. Usually this validation error is the mean value of your ten validation errors. Please note that the model here not only means the feature number, but also refers to your function model (whether it is y=ax1+bx2+c or y=ax1^2+bx2+cx1x2+d...etc.), ...

1

You may want to read up on the APRIORI algorithm. It avoids unneccessary itemsets by clever pruning. {A} = 4 ; {B} = 2 ; {C} = 5 ; {D} = 4 ; {E} = 6 B is not frequent, remove. Construct and count two-itemsets (no magic yet, except that B is already out) {A,C} = 3; {A,D} = 3; {A,E} = 4; {C,D} = 3; {C,E} = 5; {D,E} = 3 All of these are frequent ...

1

I found a slightly extended definition in this source (which includes a good explanation). Here is a more reliable (published) source: CHARM: An efficient algorithm for closed itemset mining by Mohammed J. Zaki and Ching-jui Hsiao. According to this source: An itemset  is  closed  if  none  of  its  immediate  supersets  has   the  same  ...

1

In association rule learning, the confidence of a rule is defined as follows: $$conf(X\Rightarrow Y) = \frac{support(X\cup Y)}{support(X)}$$ The confidence is the amount of times a rule has been encountered in the data, conditional on the amount of times its left hand side was encountered. $100\%$ confidence implies that any record containing $X$ also ...

Only top voted, non community-wiki answers of a minimum length are eligible