Questions tagged [partitioning]
A partition is an assignment of every element of a set into 1 & only 1 subset w/ no empty subsets. A common instance of partitioning in statistics is the partitioning of sums of squares for F-tests.
123
questions
1
vote
1
answer
68
views
Pair confusion matrix for more than two cluster algorithms
How would one code the following issue:
The columns represent the k groupings from the k cluster algorithms. The rows are the N(N-1)/2 pairs of data points. If an algorithm groups a pair together (i,j)...
0
votes
1
answer
35
views
Find p-values from varpart table in vegan package
I have conducted a variance partitioning in R with the vegan package to test how weather variables partition in the variance of the green-up day.
...
- 416
0
votes
0
answers
52
views
How to perform a variation partitioning with random effects?
I would like to carry out a variation partitioning to see how weather and topography contribute to variation in the green-up day across states.
I would like to carry this out in R. My dataset is as ...
- 416
0
votes
1
answer
55
views
Annihilator matrix transforming all variables in deviations from their sample means?
In my econometrics notes (not a proper textbook) I find that given a partitioning of the sample matrix X into $$ X =(X_1 1),$$ where 1 is the (nx1) vector of all unity elements, then $$M_{[1]}=I - 1(1'...
0
votes
0
answers
74
views
Multicollinearity in polynomial regression in R
I fit a model to my data with the following formula after a stepwise selection (x1 to x5 stand for my variables):
lm(formula = outcome ~ x1+ x3 + x4 + x2+ x5+
poly(x4, 2) + poly(x2, 2) + poly(x5, 2) +
...
- 1
0
votes
1
answer
47
views
Partitioning survival data into prognostic groups
I have survival data (n ~ 1000) with ~40% censoring and ~10% missing data. I want to partition the subjects into 4 prognostic groups based on survival. Misganow B and Vidyasagar M (Exploiting Ordinal ...
- 103
1
vote
1
answer
16
views
How choose the number of classes of a series?
I am newbie in DS.
I'm doing a choropleth map with votes of a senator in several cities.
And I have two questions:
1 - How choose the correct number of classes to part the data?
At the time im using ...
1
vote
1
answer
48
views
fast distance metric between a new data entry and available clusters of data
Assume we divide a large data set D into m different partitions of data in a distributed learning case, using ...
- 125
2
votes
2
answers
258
views
Models for describing cluster change over time
Suppose I have a set of observations encoded as a matrix $X$ whose entries $X_{it} \in \mathcal{C} $ correspond to cluster labels of experimental unit $i$ with repeated measures at time $t$, where $\...
- 2,174
1
vote
0
answers
30
views
Which type of sum of squares corresponds to the partition of variance formula for two-way ANOVA?
Does anyone have a good reference for the formula for type I and type III sum of squares in two-way ANOVA?
I learned the following formula for partition of total sum of squares in two-way ANOVA. Is it ...
- 547
2
votes
0
answers
70
views
Partition a vector into equally sized groups
I have a vector of positive numbers x in R and I want to find a way to partition it into k groups so that all the groups have ...
- 123
1
vote
1
answer
161
views
Variation partitioning using a large matrix as a predictor
I'm trying to understand the amount of variance explained in a univariate response using multiple community matrices as predictors. The problem is set up as such: I have the measured rate of a process ...
3
votes
1
answer
311
views
Should the median intuitively split a dataset into equally sized halves - and should two medians split into equally sized fourths?
I cannot decide if this question is patently silly or actually deep, so figured that makes it a perfect question for Cross Validated.
We have a dataset with N ...
- 85
1
vote
1
answer
43
views
How to check whether a partitioning is more balanced?
For example consider the following two partitions that add approximately to 100 percent.
Partition 1 : 41% 36% 14% 5% 2% 1% <1% <1%
Partition 2 : 82% 10% 4% 3% <1% <1% <1% <1%
...
- 200
7
votes
1
answer
578
views
Sampling uniformly from the set of partitions of a set?
In this blogpost, the writer states "It’s easy to sample uniformly from the set of partitions of a set: you pick a number of bins using an appropriate exponential distribution, then randomly i.i....
- 233
0
votes
0
answers
196
views
Clustering a distance matrix with k-medoids
For a symmetric distance matrix that I want to cluster, I performed several cluster algorithms:
MDS into k-Means
DBSCAN
OPTICS
k-Medoids (the one I'm having trouble with)
Now, I would like to know ...
1
vote
1
answer
15
views
Bayesian evaluation if partitioning is justified for a dataset
I'd like to comparare whether partitioning of a dataset is justified. The data is categorical with two levels and the fitted parameter is the prevalence of positives for a certain condition in each ...
- 2,433
4
votes
1
answer
38
views
Learn how to bisect 2d data to arrive at 1d relationships
I want to find a way to fit a series of curves throught 2d data. I have a hypothsis of how this can be done (I will explain it below). I would appreciate if anyone could suggest whether this technique ...
- 408
1
vote
1
answer
16
views
A line through the origin that bounds a proportion of points in a scatterplot
Suppose I have a scatterplot in a box with 0 < x < x1 and y1 < y < y2.
Let 0 < prp < 1.
Is there an efficient way to find a line that passes through the origin, and that has prp ...
- 113
18
votes
3
answers
2k
views
Data partitioning for spatial data
I am constructing different configurations of a Random Forest in order to investigate the influence of well-design variables and location, on the first-year production volumes of shale oil wells, ...
- 443
4
votes
1
answer
135
views
Exact distribution of number of transitions in a binary vector of length $n$
I am trying to derive an analytic solution for the exact distribution of the number of transitions in a binary vector of length $n$ conditional on the observed number of $1$'s and $0$'s and under the ...
- 1,081
0
votes
0
answers
391
views
variation partitioning with a GAMM model including an auto-correlation structure in R
I would like to undertake variation partitioning in a GAM framework in R, as described here: https://r.789695.n4.nabble.com/variance-explained-by-each-term-in-a-GAM-td836513.html
However, my gam ...
- 43
2
votes
1
answer
161
views
Finding the probability of winning on or before n trials? [closed]
I have two teams A and B, the probability that A wins each hole is p, the probability that B wins each hole is q and the probability that neither wins is r. What is the probability of A winning on or ...
- 47
2
votes
1
answer
202
views
In this concrete example of applying sklearn knn (with kd_tree) on Iris Data Set, how many partitions are there?
The k-nearest neighbors algorithm (k-NN)
is a
non-parametric method used for classification and regression. In both
cases, the input consists of the k closest training examples in the
...
- 137
1
vote
0
answers
59
views
Recursive partitioning tree vs neural network model
I hope this question helps shed some light on trees vs neural models. I recently came across a model tree, or a recursive partitioning model. It is basically a decision tree that has linear regression ...
- 31
1
vote
2
answers
957
views
Does the multinomial coefficient create combinations or permutations?
I am looking at this formula
I just want to make sure I get it. For example, if I want to split a fifty-two card deck into all the possible combinations of four, thirteen card hands, then I get:
$$\...
- 317
0
votes
1
answer
283
views
default plot for mob object (glm tree) not returned; using party package [closed]
I'm trying to plot a glm tree using the package party. Per the reference guide, the default plot of the terminal node should be a spinogram as in this image but I ...
- 109
2
votes
1
answer
128
views
Optimization program to allocate time series in 2 groups
I am trying to dispatch $m$ time series into 2 groups. Then I want to sum the value for each time steps, to obtain a sum per group.
My goal is to minimize the standard deviation of the difference ...
- 1,519
0
votes
1
answer
71
views
Statistical test of fairness of partitioning
I have a question about a statistical test, but I lack the proper terminology so I'll try to explain in my own words.
Say I have a collection of $N$ objects. The collection is partitioned into $p$ ...
- 129
1
vote
0
answers
134
views
Variation partition analysis: morphology-climate correlation, controlling for geography
I'm working with three datasets with informations about individuals: morphological data, climatic data of the individual sampling site and geographical data.
My objective is to assess whether ...
- 195
1
vote
0
answers
33
views
Cluster/partition by time
I have a dataset of different events in time. I want to group/cluster/partition the data by datetime. A small example of the time of the data might be:
[19-09-2018 12:00, 19-09-2018 12:01, 19-09-2018 ...
- 11
0
votes
1
answer
34
views
Estimation of partitioning error in next-generation sequencing experiments
[Edited: explanation of the partitioning error]
I would like to estimate how the initial number of molecules (or the level of gene expression) affects reproducibility between technical replicates of ...
- 120
0
votes
1
answer
1k
views
Adjusted Rand Index with different size inputs
Forgive me if my terminology is off, this concept is somewhat new to me.
I'm trying to run an ARI calculation between two partitions which contain a different number of clusters, where each cluster ...
2
votes
1
answer
238
views
Appropriate partition region in a decision tree?
I have the following partition region for a decision tree, but I do not think it is a valid region:
$R=(\vec{X}=(X_1, X_2, ..., X_p)^T\in \mathbb{R}^p: X_1X_2 \geq 3)$
I feel that this is not a ...
4
votes
2
answers
449
views
Predictors in random Forest
I am building a random forest to predict a binary variable y.
I have several predictors named x1..n.
One predictor, lets say x1, is a very strong predictor of y but only in some cases (see below) ...
- 173
2
votes
0
answers
69
views
Estimates for k-d tree partition cells and for Voronoi partition cells
Let us have a $k$-d tree in a unit cube in ${\mathbb R}^k$. Let the points of the tree in the cube be i.i.d. with some density bounded away from 0. How can I estimate the volumes of partition ...
- 963
2
votes
1
answer
209
views
How to partition set of items into subsets with similar mean, variance and number of elements - looking for some help
Short version:
Given I have a list of 120 traits (happy, sad, etc.) evaluated on a number of scales ranging from -5 (for example highly negative) to +5 (for example highly positive), how can I group ...
- 517
2
votes
1
answer
678
views
Possible explanations for Imputation before train-test split?
I'm working on a real world data set containing missing information. I understand imputing missing values before data partitioning can lead to leakage of information. I'm using this R package MissMech ...
- 33
-1
votes
1
answer
97
views
Clustering restaurant dishes
I have data about restaurants, their dishes and customer's orders. Then I want to cluster dishes in restaurant's menu by customer order. So clusters will be dishes in order and criteria of similarity ...
- 1
1
vote
2
answers
117
views
Cluster non-vector data based on similarity rating
I have a list $S$ of $N$ non-vector elements (in this case words or concepts), and I'd like to partition them into k subsets of similar words.
For each pair $(a,b)$ with $a,b \in S$, I have a number ...
- 220
0
votes
1
answer
57
views
What statistical methods should i use to identify segments in a population according to a response variable?
Assuming I have $p$ uncorrelated explanatory variables $(X_i)_{i=1...p}$ and one binary response variable $Y$, such that : $$\log\left(\dfrac{P(Y=0|X)}{1-P(Y=0|X)}\right)=1_{\{X_1>0\}}\sum_{i=2}^p\...
- 11
-1
votes
1
answer
38
views
How is graph clustering performed? [closed]
When we do graph partitioning or graph clustering how does one give the graph as input? Is it by means of an adjacency matrix? What if the graph is very large?
- 1
3
votes
1
answer
1k
views
Is it compulsory to partition data in training and testing for time series modelling?
Is it compulsory to partition data in training and testing for time series modelling ?
The reason for asking this question:
Lets say training data= first 80% observation of actual time series data(...
1
vote
0
answers
54
views
Smart sampling to learn a space partitioning
Given a compact feature space, I want to partition the space according to some classification. The first method that comes to mind is to use SVMs. For the classification I have in mind, the space ...
- 111
0
votes
0
answers
70
views
Valid to supply unique features to each fold of k-fold Cross Validation?
I have a dataset of n = 55 samples with 300,000 - 600,000 features for each sample. I am trying to train a model (ksvm classification) to predict the class of each sample. N = 26 of the samples are ...
- 225
1
vote
1
answer
2k
views
Partitioning with cross validation?
I am new to data analytics having only started exploring the field this week. I have downloaded KNIME and am working with a single dataset to try out different classification algorithms.
I am ...
- 11
0
votes
2
answers
89
views
How to combine partitioned dataset to single dataset?
I have been doing some Machine learning research on Microarray dataset. So, I downloaded breast cancer dataset from "http://datam.i2r.a-star.edu.sg/datasets/krbd/BreastCancer". The problem here is , ...
- 123
2
votes
0
answers
65
views
How do loan companies set interest rate tiers?
What statistical or machine learning methods do companies like Lending Club use to segment their customer base into loan grades A1-G5? What would a reasonable partitioning method look like after ...
- 21
2
votes
1
answer
831
views
Calculating covariances
Currently I am preparing for my exam. Therefore, we have a sample exam with the answers included. Unfortunately, even with the answers it not all makes sense to me. This is the question with the ...
- 61
0
votes
0
answers
118
views
Partition Z-Scores Across Multiple Factors in R
Basic question
Is there a way to create z-scores across multiple factors quickly? And how would you do this without creating multiple subsets and rebinding them?
Details
Basically I am comparing ...
- 78