Recent Questions - Cross Validated most recent 30 from stats.stackexchange.com 2020-01-21T09:14:42Z https://stats.stackexchange.com/feeds https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/445748 0 Rolling Window Forecast and ARIMA endorphinus https://stats.stackexchange.com/users/267797 2020-01-21T09:10:10Z 2020-01-21T09:10:10Z <p>I want to do a rolling window forecast on a time series but it seems the series is white noise ARIMA (0,0,0) with non-zero mean. But when I difference the dataset and model it with an ARIMA(0,1,1) I get a really good forecast compared to the ARIMA (0,0,0). </p> <p>Why, and is the differenced forecast not valid?</p> <p><em>Red Line = ARIMA (0,1,1) and Blue line = ARIMA (0,0,0)</em></p> <p><a href="https://i.stack.imgur.com/4YNbi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4YNbi.png" alt="Red Line = ARIMA (0,1,1) and Blue line = ARIMA (0,0,0)"></a></p> https://stats.stackexchange.com/q/445746 0 Convergence of Diffusion Process Monte-Carlo MrMMS https://stats.stackexchange.com/users/267840 2020-01-21T08:31:49Z 2020-01-21T08:31:49Z <p>Let <span class="math-container">$X_t$</span> be a <span class="math-container">$d$</span>-dimensional diffusion process initialized at <span class="math-container">$x \in \mathbb{R}^d$</span>; given as the strong solution to the SDE <span class="math-container">$$X_t = x + \int_0^t a(t,X_t)dt + \int_0^t b(t,X_t)dW_t;$$</span> where <span class="math-container">$a$</span> and <span class="math-container">$b$</span> are predictable functions and <span class="math-container">$W_t$</span> is an adapted Brownian motion. Suppose further, that <span class="math-container">$f$</span> is a bounded continuous function from <span class="math-container">$\mathbb{R}^d$</span> to <span class="math-container">$\mathbb{R}$</span> and fix <span class="math-container">$T&gt;0$</span>. </p> <p>I would like to know the convergence rate of the naive Monte-Carlo estimate <span class="math-container">$$\mu^n \triangleq \frac1{n} \sum_{i=1}^n f(X_T(\omega_i)),$$</span> to the actual expectation <span class="math-container">$\mathbb{E}\left[f(X_T)\mid X_0=x\right]$</span>. </p> https://stats.stackexchange.com/q/445744 1 Excluding effect of control variables in assessment of logistic regression model humperderp https://stats.stackexchange.com/users/265139 2020-01-21T08:25:53Z 2020-01-21T08:25:53Z <p>I have a logistic regression model with ten independent variables of which two are included as controls. While this is necessary for correctly assessing the individual coefficients of the other variables, the two control variables are driving most of the result, and I would like to assess how much is explained by the other variables of the model in aggregate. </p> <p>One approach that I came across suggested first creating a model using only the controls, note the accuracy scores (AUC and Brier scores retrieved by means of bootstrap resampling in this case), then implement the full model and subtract the accuracy scores achieved using only the controls. The difference is then to give a measure of how much is explained by the remaining variables (one of the answers here <a href="https://www.researchgate.net/post/Questions_regarding_control_variables" rel="nofollow noreferrer">https://www.researchgate.net/post/Questions_regarding_control_variables</a>).</p> <p>Is this a valid approach?</p> https://stats.stackexchange.com/q/445743 0 Expected SARSA, SARSA and Q-learning Novak https://stats.stackexchange.com/users/223392 2020-01-21T08:18:48Z 2020-01-21T08:18:48Z <p>I would much appreciate if you could point me in the right direction regarding this question about <code>targets</code> for approximate <code>q-function</code> for SARSA, Expected SARSA, Q-learning (notation: <code>S</code> is the current state, <code>A</code> is the current action, <code>R</code> is the reward, <code>S’</code> is the next state and <code>A’</code> is the action chosen from that next state). I've written my thoughts next to each question/statement:</p> <ul> <li><p>Does the Q-learning target computation require the probability of current policy to select the <code>A'</code> (action that was actually made in the environment) in <code>S'</code>?</p> <ul> <li><em>I'm not sure what probability of policy means? Target in Q-learning doesn't depend on the action taken (<code>A'</code>). But, behavior policy does have some randomness (<code>epsilon</code> - probability of taking the random action) which determines which action will be taken in environment (random or action that maximizes Q-value). Does the question refer to that 'probability of current policy'? But generally, I think this is not correct since I don't see anything similar to probability in Q-function update (except that <code>epsilon</code>).</em></li> </ul></li> <li><p>Do we need an explicit policy for the Q-learning target to sample <code>A’</code> from. And for SARSA? </p> <ul> <li><em>I guess this is true for Q-learning since we need to get max Q-value which determines which action <code>A'</code> we'll use for update. For SARSA we update the <code>Q(S, A)</code> depending on which action was actually taken (no need for max)</em></li> </ul></li> <li><p>Is this statement true: <code>All methods (SARSA, Ex. SARSA, Q-learning) require R and S’ to perform updates.</code>? </p> <ul> <li><em>All methods require S, A, R, S'> In the statement, only a subset of required parameters is mentioned. Does it make it true or not since the FULL set of parameters is left out?</em></li> </ul></li> <li><p>Is the difference between SARSA and Q-learning targets only in how <code>A’</code> in <code>S’</code> is selected?</p> <ul> <li><em>I would say that this is not correct but I'm not entirely sure. Based on some code I've seen on github, both of them select the next action in the exactly same way, but they differ in how they update parameters (SARSA updates parameters based on action actually taken in environment and Q-learning on best possible action regardless of which action was actually taken).</em></li> </ul></li> </ul> https://stats.stackexchange.com/q/445742 0 How to compute the "cumulative" sample variance of multiple sample set generated by different seeds? Luigi2405 https://stats.stackexchange.com/users/271459 2020-01-21T08:18:08Z 2020-01-21T08:18:08Z <p>I simulated my system, recording some quantities over time. I did this several times but with different seeds, holding the same parameters (or iteration variables if you prefer), and varying ONLY the seed (in my system I simulated some arrivals with different seed per each iteration to add more randomness and get, hopefully, some more insights). This said, to show some statistics I need some "cumulative" quantity, in particular, sample mean and sample standard deviation. I's quite obvious that the sample mean is the mean of all the sample mean. But my question is: what about sample standard deviation? I hope I've been clear</p> https://stats.stackexchange.com/q/445741 0 Retraining of object detection CNN Martin Horák https://stats.stackexchange.com/users/227233 2020-01-21T08:07:11Z 2020-01-21T08:07:11Z <p>I am working on an object detection system that should detect UI elements (such as button, checkbox, radio button, etc..) in the photo of a touch screen of printer (not screenshots, but literally a photo of the screen).</p> <p>I have approximately <strong>900 images</strong> of screens which is most likely not enough even when I will use some image augmentation. So I found a dataset with screenshots of android apps (which I find pretty close to the printer screens). The datasets are huge <a href="https://zenodo.org/record/2530277#.XiV9mMhKiUk" rel="nofollow noreferrer">ReDraw</a> dataset has 19k unique app screens and <a href="http://interactionmining.org/rico" rel="nofollow noreferrer">Rico</a> dataset has 66k unique app screens. The problem is that the these datasets have quite a lot of mistakes such as: </p> <ul> <li><strong>invisible elements</strong> (that are hidden or not drawn), </li> <li><strong>inconsistent boundary boxes</strong> (bbox around checkboxes sometimes include the text that is next to the checkbox and sometimes just the checkbox or bboxes around text cover whole text area instead of just text) </li> <li><strong>inconsistency between classes</strong> (e.g. toggle button is sometimes annotated as checkbox or radio button or tabs are annotated as radio button class). </li> </ul> <p>My idea of a solution is to retrain one of the object detection CNN (R-CNN, YOLO, SSD), but to retrain the whole network (not just last few layers) you need a lot of data and it is not in my powers to hand-annotate thousands of images. So I am wondering whether it might be beneficial to use the android app dataset (with some wrong annotations) to train whole CNN and then lock the bottom layers and finish the training with 900 printer images extended by let's say another 1000 hand-annotated android screenshots. </p> https://stats.stackexchange.com/q/445740 0 Leave-One-Out CrossValidation with Weka Mr.R. https://stats.stackexchange.com/users/271456 2020-01-21T08:06:54Z 2020-01-21T08:29:00Z <p>I have a dataset consisting of 5 subjects (5 different virus names and 5 benign program names). The programs belong to the class "benign" and the viruses to the class "infected". There are 1000 instances in the dataset, each owning several attributes (besides class and subject).</p> <p>I now want to conduct an experiment where I take this dataset and measure the performance of different classifiers regarding unknown benign and infected instances. </p> <p>In order to do that I want to run a leave-one-out cross validation: in 5 runs a classifier shall be trained by 4 subjects of class benign and 4 subjects of class infected. Afterwards the classifier shall be tested on the remaining subject of class benign and the one from class infected. This I want to repeat 5 times with changing test subjects so that in the end every subject was a test-subject one time.</p> <p>I want to do this without preparing my datasets by hand. <strong>is there a way in weka 3.8 to automate this task in the knowledge flow?</strong></p> <p>Thanks in advance!</p> https://stats.stackexchange.com/q/445739 0 how to choose a sample? user45523 https://stats.stackexchange.com/users/45523 2020-01-21T07:51:57Z 2020-01-21T07:51:57Z <p>I need to review 600 exams with burbles by hand (I have a software to read them all). How many exams do I need to review in order to have 95% of chance of my software is working properly? </p> <p>I think this problem is very similar with the surveys in the elections when we take only a small sample to have an idea of the entire population (most of the time of more than 90% of certainty)</p> https://stats.stackexchange.com/q/445738 2 How to forecast low values in data more accurately than the higher values? Aditya C S https://stats.stackexchange.com/users/271448 2020-01-21T07:38:45Z 2020-01-21T07:38:45Z <p>I have a scenario where I have to forecast small values in data more accurately than the higher values. I have data set as below</p> <pre><code>start_time count 2019-12-14 10:00:00 20 2019-12-14 11:00:00 15 2019-12-14 12:00:00 3 2019-12-14 13:00:00 5 2019-12-14 14:00:00 10 2019-12-14 15:00:00 13 2019-12-14 16:00:00 7 2019-12-14 17:00:00 17 2019-12-14 18:00:00 5 2019-12-14 19:00:00 4 2019-12-14 20:00:00 10 2019-12-14 21:00:00 10 2019-12-14 22:00:00 10 2019-12-14 23:00:00 4 </code></pre> <p>I have to forecast low values like 3,4,5 etc... more closely/accurately than other higher values like 10,14,17 etc...</p> <p>I have tried ARIMA model with all different variations of P,D,Q values. However, I don't get the best match. Over all forecasting performs good but the models fail in predicting lower values or sudden drop in value. </p> <p>What model should I use to solve my problem?</p> <p>Thank you</p> https://stats.stackexchange.com/q/445737 0 Defining the number of multiple comparisons for a Bonferroni correction Ang Jit Wei Aaron https://stats.stackexchange.com/users/181953 2020-01-21T07:30:35Z 2020-01-21T07:30:35Z <p>I have a 4 (Time) x 3 (Size) x 3 (Type) Repeated Measures ANOVA. Since SPSS does not allow post-hoc analysis for repeated measures, I had to run individual t-tests one by one.</p> <p>Assuming that I am only strictly interested in looking at the time domain, do I also have to account for the other comparisons in the size and type domain for bonferroni comparisons? Or do I just need to assume the total number of comparisons to those just within time?</p> https://stats.stackexchange.com/q/445733 -2 meaning and use of 'abstract' in def (python) tib https://stats.stackexchange.com/users/270949 2020-01-21T06:52:29Z 2020-01-21T07:00:18Z <p>What does abstract mean in python please !! see figure bellow</p> <p><a href="https://i.stack.imgur.com/OEPdt.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OEPdt.jpg" alt="enter image description here"></a></p> <p>If there are any videos about this it will be better..</p> <p>Thanks in advance</p> https://stats.stackexchange.com/q/445732 1 time series model with additional, time-independent regressors? user3050269 https://stats.stackexchange.com/users/35483 2020-01-21T06:18:22Z 2020-01-21T07:26:03Z <p>How does one introduce time-independent regressors into a time-series model?</p> <p>Let's say that you want to model house prices based on mortgage valuations from the past 5 years AND based on additional regressors such as house size (time-independent) and population density in the area (time-dependent), as well as years since construction (time-dependent). </p> <p>Would such regressors be called external? Is this doable with AR/MA/etc models? If so, could you point me to the relevant approach? Thanks!</p> https://stats.stackexchange.com/q/445731 0 Derive that the residual u = 0 [duplicate] Carloose https://stats.stackexchange.com/users/271451 2020-01-21T06:10:43Z 2020-01-21T06:17:00Z <p><a href="https://i.stack.imgur.com/yD3q4.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/yD3q4.png" alt="Derive the following below"></a></p> <p>From the work I have done so far <a href="https://i.stack.imgur.com/m2txa.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m2txa.jpg" alt="enter image description here"></a></p> <p>not sure if the second line makes sense and if there's any methods to solve this without really giving the solution would be amazing</p> https://stats.stackexchange.com/q/445730 1 How to interpret output from RMSEP in R Tsu https://stats.stackexchange.com/users/0 2020-01-21T05:56:15Z 2020-01-21T06:06:12Z <p>I have a dataset with 15 columns and 500 rows. I have developed a plsr model as "plsr_model" and I have a testing dataset as "train.data". I want to find the Root mean square error of prediction (RMSE). As per the parameters required for the RMSE function, I have written as follows:</p> <p><code>RMSEP(plsr_model,Train.data,15,estimate = "all")</code></p> <p>I get the following output:</p> <pre><code>&gt; RMSEP(plsr_model,Train.data,15,estimate = "all") (Intercept) 15 comps train 0.9989 0.7599 CV 1.0011 0.8279 adjCV 1.0011 0.8238 test 0.9989 0.7599 </code></pre> <p>How do I interpret this? Is there any other Better way to calculate the error in prediction?</p> https://stats.stackexchange.com/q/445729 1 May each parameter's test statistics in regression be related to different distributions? Numbers https://stats.stackexchange.com/users/271077 2020-01-21T05:08:43Z 2020-01-21T05:08:43Z <p>When I run a linear regression, the parameters are all tested whether they're zero using t-test. In some other regression, all parameters are tested whether they're zero using z-test.</p> <p>Is there some kind of estimation algorithm for regression where one parameter can be tested whether it's actually zero using a z-test, another by t-test, some other by some other allowable distribution, etc? </p> <p>Is there a way to change via programming what distribution the test-statistic for each parameter is in order to better understand the mechanics? </p> <p>Is there a reason all the basic regression estimation techniques fix all the parameters' test-statistic to be related to the same distribution? </p> <p>By "related" I mean as in <span class="math-container">$\sim$</span>, i.e. x <span class="math-container">$\sim \chi^2$</span>.</p> https://stats.stackexchange.com/q/445727 1 Dimension reduction - doing a PCA on the coordinates of a MCA MercedesRD https://stats.stackexchange.com/users/58687 2020-01-21T02:58:57Z 2020-01-21T02:58:57Z <p>I have a dataset with 25 continuous variables and 2 categorical variables. I want to perform k-means clustering, so as a previous step I am performing a multiple correspondence analysis on the categorical variables as a way to transform them to continuous. The first 31 dimensions (components) keep 90 % of the variability, and the first 17 components 50 % of variability. Since I want to keep some balance in the influence of the the original continuous and categorical variables, I don't want to add 31 variables. So I was thinking of keeping just 17.</p> <p>But... Could it be an alternative to perform a PCA on the coordinates of the MCA? and this way, reduce even more the number of variables.</p> <p>thanks in advance.</p> https://stats.stackexchange.com/q/445725 0 How can I prove the simple random walk is a Markov process? Johnny Ton https://stats.stackexchange.com/users/264033 2020-01-21T02:30:46Z 2020-01-21T04:26:40Z <p>I know a simple random walk is defined as <span class="math-container">$X_t=X_{t-1}+w_t$</span>, but how can I modify this equation is show it is a Markov process?</p> https://stats.stackexchange.com/q/445723 2 What is the shape of the Benini distribution? Reinstate Monica https://stats.stackexchange.com/users/173082 2020-01-21T02:27:49Z 2020-01-21T09:08:36Z <p>The <a href="https://en.wikipedia.org/wiki/Benini_distribution" rel="nofollow noreferrer">Benini distribution</a> is a continuous univariate distribution that is used in actuarial applications. For all <span class="math-container">$x \geqslant \sigma$</span> it has density function:</p> <p><span class="math-container">$$\text{Benini}(x| \alpha, \beta, \sigma) = \frac{\alpha + 2 \beta \log (x/\sigma)}{x} \cdot \exp \Bigg( - \alpha \log \Big( \frac{x}{\sigma} \Big) - \beta \log^2 \Big( \frac{x}{\sigma} \Big) \Bigg),$$</span></p> <p>where we have the shape parameters <span class="math-container">$\alpha &gt; 0$</span> and <span class="math-container">$\beta &gt; 0$</span> and the scale parameter <span class="math-container">$\sigma &gt; 0$</span>. (The <span class="math-container">$\log^2$</span> notation is explained <a href="https://math.stackexchange.com/questions/150546/">here</a> for anyone unfamiliar with that notation.) I have plotted this density for a range of parameter values, and it seems to be strictly quasi-concave (with a single mode) in all cases, but monotonically decreasing (with a mode at its minimum value) in some cases. I would like to know the general "shape" of the distribution from the parameters.</p> <hr> <p><strong>Question:</strong> Is the Benini density function always unimodal? Is it always strictly quasi-concave? Under what conditions (if any) is the density function monotonically decreasing? What is the mode of the distribution as a function of the parameters?</p> https://stats.stackexchange.com/q/445700 0 What is the expanded representation, $\phi(X)$, required to obtain the RBF kernel? Nip https://stats.stackexchange.com/users/225266 2020-01-20T22:31:23Z 2020-01-21T08:54:49Z <p>For the two-dimensional case, where <span class="math-container">$\boldsymbol X=[x_1, x_2]$</span> and its corresponding expanded represetation <span class="math-container">$\boldsymbol\phi(X)= [1, \sqrt2 x_1, \sqrt2 x_2, x_1^2, x_2^2, \sqrt2x_1x_2]$</span>, we can derivate the polynomial function kernel <span class="math-container">$K(X,X^T)=&lt;\phi(X),\phi(X)^T&gt;=(1+&lt;X,X^T&gt;)^2$</span>. </p> <p>What is the expanded representation, <span class="math-container">$\boldsymbol \phi(X)$</span>, required to obtaing the RBF kernel <span class="math-container">$K(X,X^T)=&lt;\phi(X),\phi(X)^T&gt;=exp(-\sigma||X-X^T||^2)$</span>?</p> https://stats.stackexchange.com/q/445716 0 How to change (expand) the distribution of my data? Pe Dro https://stats.stackexchange.com/users/260706 2020-01-20T12:47:02Z 2020-01-21T06:38:39Z <p>I have to train a regression model that can approximate the function used to map the inputs (3) onto a single output (1). </p> <p>The data is made of rows of the input-output pair <strong>((X,Y,Z), SUM)</strong> where X,Y and Z are the input and SUM is the output. </p> <p>The values of X,Y and Z are in range 0-255. Now, when I obtain the results at the SUM node of the network, I get same results for cases like (0,0,255), (0,255,0) and (255,0,0). The following scatter plot describes the problem well. <strong>(STD = 209)</strong> <a href="https://i.stack.imgur.com/04ZsH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/04ZsH.png" alt="enter image description here"></a></p> <p>To avoid such similar results, I thought of changing the distribution (I don't know if standard deviation would define the problem well) and also change the range of these 3 variables by multiplying the R,B G values with some +ve and -ve values and obtained R', B', G'. </p> <p>Luckily, the <strong>SUM</strong>, now, was more distinct. Here's the scatter plot <strong>(STD = 331)</strong>:</p> <p><a href="https://i.stack.imgur.com/jUEEB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jUEEB.png" alt="enter image description here"></a> So, what I just did, kind of spread up the data and increased its STD... is it done well? Is it a thing? And what is a better method of doing so? </p> <hr> <p>Edit (21-01-2020): I read </p> <blockquote> <p>If you multiply the numbers on a list by any values (other than ±1), or if you raise the numbers on a list to a power, that always changes the standard deviation. Multiplying changes the spacing on the list. In particular, if you multiply each number by k, then you multiply the standard deviation by |k|.</p> </blockquote> https://stats.stackexchange.com/q/445578 11 How do DAGs help to reduce bias in causal inference? P Sellaz https://stats.stackexchange.com/users/6884 2020-01-20T08:00:21Z 2020-01-21T08:15:36Z <p>I have read in several places that the use of DAGs can help to reduce bias due to </p> <ul> <li>Confounding</li> <li>Differential Selection</li> <li>Mediation</li> <li>Conditioning on a collider</li> </ul> <p>I also see the term <em>“backdoor path”</em> a lot.</p> <p>How do we use DAGs to reduce these biases, and how does it relate to backdoor paths ? Extra points (I will award a bounty) for real world examples of the above</p> https://stats.stackexchange.com/q/396953 1 regression with z-scores as composite variables? Zara https://stats.stackexchange.com/users/240530 2019-03-11T22:00:04Z 2020-01-21T04:05:25Z <p>So I have 5 IV's- a,b,c,d,e and one DV. </p> <p>a is fine as is. b &amp; c measure the same concept and since their ranges are the same, I averaged the scores to create a composite variable. d &amp; e measure the same concept but their ranges are very different, so I averaged their z-scores to create a composite variable. </p> <p>MY questions are- 1. Is this an apprpriate way of creating composite variables in order to do regression? 2- If I use z-scores for one composite variable, should I do the same for the composite variable for b&amp;c and also use the z-score for a? 3- By using the averaged z-score, does it effect the interpretation of my regression results? </p> https://stats.stackexchange.com/q/357568 2 Statistical significance test for comparing two canonical correlation analyses Dikran Marsupial https://stats.stackexchange.com/users/887 2018-07-17T12:08:57Z 2020-01-21T03:03:25Z <p>I have a colleague who is comparing several different treatments of data via canonical correlation analysis. In other words, given some time-varying signal, $a(t)$, he extracting some vector of features $v_1(t)$. He then supposes that this is a predictor for some other vector $p(t)$. To check this he computes the [first] canonical correlation coefficient, $R_1 = \text{CCA}(v_1(t),p(t))$. And then he tries some new improved feature extractor, $v_2(t)$, and compares again $R_2 = \text{CCA}(v_2(t),p(t))$.</p> <p>I can find tests for the situation that $R_1$, and $R_2$ are different from zero. But what about the test that $R_1$ and $R_2$ are significantly different from each other? </p> <p>Asked on behalf of a colleague, but I'd also be interested in the answer.</p> https://stats.stackexchange.com/q/331712 1 spacetime R: How to handle missing data in a space-time-full data structure for spatio-temporal kriging purposes? Pigna https://stats.stackexchange.com/users/165961 2018-03-04T20:51:01Z 2020-01-21T07:01:00Z <p>I am using R and the spacetime package.</p> <p>I am having problems using <a href="https://www.rdocumentation.org/packages/spacetime/versions/1.1-5/topics/STFDF-class" rel="nofollow noreferrer">STFDF</a>.</p> <p>I want to use the STF data-structure since I have spacetime data with recurrent observations for fixed spatial coordinates. The problem is that there are some missing values here and there (both space and time wise). What should I do to fill the gaps? Can I put NAs in it?</p> <p>The data is composed by daily ozone mean measures for each station during a period of 1 year (2016, Jan to Dec). The stations are 30, however most of them don't have 366 records. Here is the distribution of stations by number of yearly measures:</p> <p><a href="https://i.stack.imgur.com/prCRR.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/prCRR.jpg" alt="enter image description here"></a></p> <p>How should I handle this? Do I need to add the records putting NAs / interpolating them? Can spacetime work around the NAs or automatically interpolate them?</p> https://stats.stackexchange.com/q/260439 1 Correct way to average data or to compare in another way shara https://stats.stackexchange.com/users/134541 2017-02-07T09:16:42Z 2020-01-21T08:03:05Z <p>I have data which is of similar type (though about a different subject) to that below and I would like to calculate averages per day so that I can compare between the different types of food. I would like to calculate the average weight of food eaten per day for different types of food.</p> <p>I have calculated mean, standard deviation etc. of (food eaten/animal)/day as in the example shown below. </p> <p>I also have the raw data, but I don't think I should just pool all the data together and take the average over all the days, as that would give more weight to results from certain days than others (whereas weather or other daily conditions changed from day to day and are likely to have had an effect). </p> <p>I have seen on the internet and it makes sense to me logically as well, that it's probably not correct just to average the average, but I'm not sure how to go about comparing the different foods?</p> <p>Food eaten</p> <p>APPLES</p> <p>10/12/2013: Mean = 1.8kg, s.d.= 0.5kg N=20 </p> <p>11/12/2013: Mean = 2.2kg, s.d. = 0.3kg N=6</p> <p>12/12/2013: Mean = 3.1kg, s.d. = 0.4kg N=12</p> <p>BANANAS</p> <p>10/09/2013: Mean = 2.3kg, s.d.= 0.5kg N=10</p> <p>11/09/2013: Mean = 2.2kg, s.d. = 0.3kg N=4</p> <p>12/09/2013: Mean = 1.1kg, s.d. = 0.5kg N=10</p> <p>CAKE</p> <p>20/01/2014: Mean = 1.0kg, s.d.= 0.2kg N=8</p> <p>21/01/2014: Mean = 2.0kg, s.d. = 0.3kg N=8</p> <p>22/01/2014: Mean = 2.0kg, s.d. = 0.3kg N=4</p> <p>etc.</p> <p>I want to compare the amount of cake eaten with apples and bananas and several other foods.</p> <p>I know there are a lot of other factors to consider here if I was really comparing food eaten. Actually my data is about something different I just don't want to put it on the website as it's owned by my company. Thanks!</p> https://stats.stackexchange.com/q/172395 1 which prediction algorithm can be used for multivalued attributes(columns) in R? Kamal Pradhan https://stats.stackexchange.com/users/89412 2015-09-14T07:13:01Z 2020-01-21T06:01:30Z <p>for example <a href="https://i.stack.imgur.com/5BIyz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5BIyz.png" alt="example data set"></a></p> <p>let we have to predict the combo_name using the meal_id but meal_id is a multivalued attribute. so in this case which prediction algorithms is to be used??</p> https://stats.stackexchange.com/q/138031 0 interpretation baseline corrected ANCOVA Jochem Jansen https://stats.stackexchange.com/users/53360 2015-02-17T11:14:33Z 2020-01-21T05:00:55Z <p>I hope you can help me with some statistic 'problem', or maybe check is more accurate =].</p> <p>I have data on an experiment with a 2 by 3 design, with session(first/second) and medication(a/b/c) as factors. The outcome measurement is a visual analogue scale, measuring emotion. Medication is only given after the first session.</p> <p>My problem is that the medication groups differ on the VAS at the first session and age, and therefore complicate the interpretation of the effect of medication. These differences are trend significant. My solution would be to use an ANCOVA with:</p> <ul> <li>DV : VAS score during second session </li> <li>Fixed factor: medication</li> <li>Covariates: VAS score during first session and Age</li> </ul> <p>My results show a significant interaction between session and medication, corrected for first session differences in VAS and age. The bonferroni corrected pairwise comparisons suggests that medication a and b have different effects on the VAS, but neither differs from c (placebo). This may be caused by a lack of power, because n=12 in all three medication groups.</p> <p>Would you consider this a sound analysis method and interpretation?</p> <p>many thanks in advance </p> https://stats.stackexchange.com/q/121952 1 In R: How do 'centralization' measures of the STATNET and iGraph package handle disconnected networks? wake_wake https://stats.stackexchange.com/users/59549 2014-10-29T18:02:21Z 2020-01-21T03:19:29Z <p>I am working with about 300 disconnected of different sizes. I calculate different graph-level centralization measures for these networks using the STATNET and iGraph packages in R. </p> <p>However, I find that the nodes in subgraphs of N=2 get assigned the highest value of 1 for the Eigenvector centrality measure with iGraph. As a result, networks with a lot of isolated dyads get very high graph-level Eigenvector centralization scores.</p> <p>In my networks this is not a valid result, because these networks are poorly connected and thus should, theoretically, have a low centralization score. </p> <p>Does anyone know how these measures handle disconnected graphs? And are there ways to deal with this? Also, are there other ways to assess the structure of these networks?</p> <p>Any help is welcome. Thank you!</p> https://stats.stackexchange.com/q/110201 5 ARIMA, adjustments and intervention analysis 1two3stats https://stats.stackexchange.com/users/53208 2014-07-31T21:31:33Z 2020-01-21T09:02:40Z <p>I have very little knowledge of time-series analysis (despite my stat master - didn't do anything else than an introductory course) but now I'm facing a statistical problem whose answer is this very kind of analysis - so would really need a helpful hand.</p> <p>In a nutshell, I have account monthly sales. In only <em>some</em> of them (1 to 10 or so), a marketing project has been conducted (I call them the Affected accounts); the other accounts remain unaffected (called the Control accounts). The question that I have to answer is basically "did the project have an impact in the Affected sales (but here's the part that annoys me) compared to the Control sales?" So, as I have a time-series analysis (and a seasonality) I guessed that I had to perform a proc ARIMA (I'm using SAS - and its fantastic feature that I discover and displays an automatically fitted model, Time Series Forecasting System). Here are my questioning:</p> <ul> <li><p>How is an adjustment variable accounted for, in an ARIMA model ? If I make an educated guess from the linear regression model, I assume I just have to enter the variable in the INPUT, as independent / explanatory variables ? Also, if I am to enter my variable "Control_Sales" like that, is my adjustment correct ? Shouldn't I do a preliminary transformation of my dependent variable, for example redefining it as (Affected_Sales/Control_Sales) or (Affected_Sales - Control_Sales) ?</p></li> <li><p>Besides, I have another series of questions around the Intervention analysis - because the conduct of the projects is never a straightforward, pulse intervention and I cannot really pin them down to a precise type. Here's the best explanation I can provide: The projects, as I already said, are not carried out in all accounts, but only in a very small portion of them. Moreover, the projects can be regarded as marketing campaigns that last several months. They have a start date and an implementation(=end) date. But even from my supervisors, there is disagreement on when we should be starting to assess an effect on the sales - sometimes, the sales start to increase <em>during</em> the campaign, most of the time, the effect is seen after the end of the project, but not straight after, perhaps from 2 to 9 months after... So I'm getting very confused when it comes to modelling the type of intervention (a ramp ? several pulses ?) and I am wondering if it wouldn't be better if I performed an Intervention Detection, to see where the impact actually took place, and in the discussion part of my analysis compare it to the reported, "theoretical" implementation date. I deeply believe this idea would be the most appealing, due to the fact that nobody is able to be clear about the Intervention start point (statistically speaking). What do you think ? How would you do ? But there comes the most challenging part: I have no clues whatsoever how to perform an Intervention Detection...</p></li> </ul> <p>I could really do with some piece of advice !</p> <p>PS. I also meant to say that I am happy to provide the data if it can help with the discussion.</p> <hr> <p><strong>Update</strong></p> <p><img src="https://i.stack.imgur.com/avcFk.png" alt="So here is the data"></p> <p>Here is my data, if that can help. Ideally I would stick to the ARIMA (or other, best-fitting time-series model) because I not comfortable at all with this field and this is something I am currently capable of doing (don't know anything about VAR or Granger causality).</p> <p>Moreover, I have only collected the sales from Jun-13 onwards, as I had fixed myself a 6-month limit for the "sale sbefore", but I can have figures back to 2011... I don't know however how relevant it is to include them in the analysis - and if that couldn't "dilute" the impact of the intervention ?</p> <hr> <p><strong>Update 2</strong></p> <p>Data from 2011 onwards</p> <pre><code> Date Sales_nat Sales_affected Jan-11 13535.04614 10564.2 Feb-11 12255.18701 6338.52 Mar-11 15504.88513 16902.72 Apr-11 13259.76914 14085.6 May-11 15967.85091 13381.32 Jun-11 15351.15898 9859.92 Jul-11 16001.81365 16902.72 Aug-11 20151.51071 23692.09 Sep-11 21533.29437 30507.47 Oct-11 21122.32893 19242.99 Nov-11 22350.66487 25579.51 Dec-11 21707.95193 15019.31 Jan-12 23225.30391 28394.63 Feb-12 22782.53005 23466.67 Mar-12 24346.6397 30030.61 Apr-12 23093.62005 21361.83 May-12 26336.53924 22530.96 Jun-12 22695.90797 18770.13 Jul-12 26825.00843 21824.68 Aug-12 26202.68137 23225.24 Sep-12 24917.01741 23929.52 Oct-12 30170.13777 20649.55 Nov-12 28223.8397 27215.49 Dec-12 28165.34954 19713.84 Jan-13 30716.72604 20182.69 Feb-13 26684.0364 28867.48 Mar-13 28277.98102 15019.31 Apr-13 30858.22655 28159.2 May-13 31845.99066 24871.22 Jun-13 29444.00066 31425.17 Jul-13 34896.64914 41989.37 Aug-13 30925.31929 23945.52 Sep-13 32861.02081 25577.51 Oct-13 34976.22452 10795.63 Nov-13 32547.14685 31668.6 Dec-13 38381.84523 35435.43 Jan-14 40211.59741 4221.68 Feb-14 31925.53772 29569.76 Mar-14 37865.2967 49251.59 Apr-14 39391.11061 28865.48 May-14 35614.36797 7042.8 Jun-14 41398.94482 37316.84 </code></pre> https://stats.stackexchange.com/q/57765 0 ARIMA model fitting and forecasting implementation Mavadu https://stats.stackexchange.com/users/25087 2013-05-01T01:11:09Z 2020-01-21T08:28:22Z <p>I am new to time series modelling and I am trying to build a simple time series model using ARIMA methodology and forecast. I could write an R program to do the same, but I am more interested in writing my own <code>arima(1,1,0)</code> and <code>predict()</code> functions that R provides in Java / C++ / Python. While I can go about reading the theory behind ARIMA modeling and then implementing it, I still face a learning curve there, due to time constraints, I would like to take an approach where I can see a time series data sample, and a write up which walks me through the steps to compute <code>arima()</code> and <code>predict()</code> so that I can implement an algorithm. If someone can point me to a sourcecode / psedocode / step-by-step example via data, it would really be helpful.</p>