2
$\begingroup$

I need to check if changes to a network architecture lead to an improvement in its performance, measured through the downlink traffic, in gigabytes (GB), and the user throughput, in megabits per second (Mbps). As shown in the chart below, the red points refer to the original architecture, while the green ones refer to the optimized architecture.

A visual inspection gives us hints that there is an improvement in the downlink traffic. The team responsible for the test measured such improvement using the following methodology: (a) calculated the linear regression for both data distributions (before and after); (b) for each traffic point of class "before", calculated the respective y-values by both regression equations; (c) weighted the difference between y-values and averaged the differences for all data points; (d) attributed this value as the percentage of improvement (~30%).

enter image description here

I particularly disagree with the effectiveness of this method since the $R^2$ (R squared or the coefficient of determination) is 0.51 for the before data (red line) and is 0.66 for the after data (green line), which means that both models are not able to represent the data distribution accordingly.

I have run statistical t-sample tests separately for each variable and only Traffic has a statistically significant difference. Does running statistical tests for each variable separately make sense?

What would be the best way to measure the distance between these both joint probability distribution?

As requested, I'm sharing the data I'm working with (exported using the dput R function):

structure(list(Traffic = c(358,634487359156, 358,081700458936, 
364,921330559789, 368,123672707711, 387,53643531514, 379,257232916016, 
362,713506327043, 359,82328383671, 381,305812425502, 407,339139784046, 
428,416632881068, 412,655144067482, 376,046166926732, 363,07104534918, 
358,815057027966, 282,870833084255, 361,876017556368, 345,019340467771, 
325,154738267302, 353,44020143376, 385,570663061749, 399,900057155374, 
380,75357941997, 358,444086059606, 385,568170535646, 414,561511115413, 
440,586668527286, 411,941173812588, 408,46216126721, 416,850480920185, 
370,807569426168, 317,450755503328, 401,90197103726, 357,792013536275, 
381,27631753328, 382,546398056533, 408,533475208496, 413,183559601361, 
374,583746472516, 377,47175338409, 382,34602665257, 424,172276348335, 
443,576869009092, 426,001928981756, 404,514830761776, 393,723023576954, 
380,731008211369, 341,872664736154, 396,601198857788, 385,935541188523, 
380,291215827827, 393,249582091019, 416,597766468055, 414,589898218255, 
391,348753989984, 370,381208988848, 379,26609523105, 414,334414292677, 
436,208269479671, 415,150716591475, 395,087681618456, 390,461553170917, 
351,186403941778, 301,609228907095, 297,733212342947, 306,126793995473, 
341,63476696697, 344,649754236136, 379,451234531048, 411,316242129047, 
365,386879046398, 354,853879155292, 362,501087713582, 402,304790950272, 
414,95713039746, 402,312960896944, 369,086203204564, 355,598175144442, 
328,091903332578, 287,513301425137, 399,676283700775, 396,413290084342, 
396,906256358086, 398,282269240724, 439,290693427295, 438,40016924009, 
385,792082463917, 389,526301987528, 397,595301351948, 445,665328936229, 
475,42858313454, 458,958109118267, 425,770816422521, 416,327327385858, 
364,511033068468, 317,879193045733, 417,47790512983, 417,448825658626, 
394,080008704402, 406,029455790487, 451,495485965286, 442,415628792346, 
416,22762634173, 417,579683279044, 423,085673441012, 449,121770273005, 
466,044776250682, 468,675222131592, 455,624078507892, 434,903847485663, 
384,542634517852, 303,480977536926, 419,506821377834, 400,548667836895, 
395,717182933187, 384,060330716882, 428,07003327905, 438,640836482722, 
416,389917485783, 418,902937692498, 417,281964883384, 431,67727244507, 
478,636602703031, 449,204300581756, 426,873759223187, 418,234663749496, 
376,283233853881, 327,534269853823, 410,116783178718, 404,865030679276, 
394,724049754734, 403,219686503819, 441,379153354432, 433,10756700729, 
403,84517208205, 395,899275812393, 421,981899798312, 455,844667068025, 
452,339206051219, 480,922249139665, 456,455059015964, 420,062516225176, 
408,752540558233, 348,543080009334, 431,440759412611, 410,560868246105, 
432,150990878351, 399,035981439598, 454,710706164742, 457,529199292771, 
425,425601557228, 405,04643119012, 418,493886464228, 447,466317519463, 
466,141707821362, 461,292359813914, 448,354626127909, 445,544302946551, 
405,518903360348, 372,891182774771), Throughput = c(11,5053501293553, 
11,9309758603423, 13,7779263824406, 14,3917297378668, 14,1145090734722, 
14,3385017873157, 15,5399892064479, 15,1220805389237, 12,7658038282802, 
11,0769541704585, 9,73121702659838, 11,1248395386123, 13,9785700324735, 
13,0841821644765, 16,8795922742222, 20,1712166512422, 9,85924113071211, 
11,0171963969773, 12,889078970263, 12,0520743946818, 12,787965651558, 
13,4985291104026, 13,6241787334508, 12,9393388194823, 10,6475688444511, 
8,94585741419127, 9,07309002461651, 10,5375512217505, 13,4126520452011, 
14,681020681011, 17,1688832173664, 20,390856149799, 10,9964196876543, 
12,0536573820587, 13,0703310249279, 13,1021886458897, 11,9937017351554, 
12,5364685320394, 12,8898704639515, 12,2924906025615, 11,7588259830951, 
11,2093314898589, 9,44153033660997, 10,2496453925612, 12,1389408269939, 
12,376784680386, 13,086556645542, 15,6967049567695, 10,1867216443261, 
10,5959238812769, 12,8493064124163, 12,3358248820065, 11,9731228992546, 
13,410871184402, 13,9815381338054, 13,9663018803019, 12,1306301432647, 
10,5494236270779, 9,84281763667588, 10,3733162813883, 12,5014449363236, 
13,8610332197323, 16,7369255368714, 20,1781422210165, 13,6150765560332, 
14,0442640086184, 14,163977429003, 14,6742929846588, 13,0740906199482, 
13,7559624325849, 13,5893530111571, 13,1089163422419, 11,5332502818746, 
9,11523706352876, 8,3805330471852, 9,73240426713112, 12,799046563197, 
14,8371428110662, 18,5229310449557, 20,435971290043, 12,0170507989659, 
12,9058982111435, 13,8481714472943, 14,7156485298826, 13,287200295575, 
13,5523506812201, 14,4592045748109, 14,2156223921772, 11,6292188916044, 
9,92968406898798, 9,45360061535948, 11,4089857727813, 14,4388236123322, 
15,909418885552, 18,1699248598878, 20,3621645035911, 11,7978070472534, 
13,1508655077321, 14,3343464454511, 14,0502002112821, 12,7616484864157, 
12,9440877816133, 14,2890334317849, 14,6643993135526, 13,2759215105139, 
11,2692871367623, 9,77474917946549, 11,5132650662402, 13,9352357530285, 
14,4083511053252, 16,6813231052547, 20,6856875487626, 11,6743340318485, 
13,2820555865998, 14,1556667452738, 13,7909860283007, 13,3631836896703, 
14,0503980847043, 14,9774350673516, 14,7544317206187, 12,8362467665561, 
10,9958260673879, 10,2391581011887, 10,852961456615, 14,1156963140049, 
15,1664041854793, 16,9983163274962, 20,2408680958295, 10,7083159850429, 
11,6856128169095, 14,1340985422624, 13,9089185878862, 12,713960991684, 
13,2529681935476, 14,5148070064275, 14,6163160719767, 13,1352335073843, 
11,5100990914862, 10,2308474174595, 10,3750971421874, 12,7921209934226, 
14,6289799709926, 16,1474606123663, 20,8216265897612, 11,4877393947863, 
11,3850430887043, 12,8635532988091, 13,4561841980682, 12,0020124188846, 
12,5788134443738, 13,8982334230915, 13,7057025833656, 12,9945455042547, 
11,5904357008683, 9,78207049608405, 10,6653774524422, 13,0885353797632, 
14,7293017960091, 15,5902490556672, 18,9172927752473), Target = structure(c(2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = c("After", 
"Before"), class = "factor")), .Names = c("Traffic", "Throughput", 
"Target"), row.names = c(NA, -160L), class = "data.frame")
Improve this question
$\endgroup$
9
  • $\begingroup$ Hi, welcome (+1). Couple of things: 1. Could you please elaborate on "coefficients [...] are 0.51 (Before) and 0.66 (After), which does not seem appropriate to represent the data." 2. Just eye-balling the plots I would say there is a statistically significant level shift between Before and After. 3. Therefore, could you perhaps post the regression output (and preferably also the data itself). $\endgroup$ – Jim Oct 11 '19 at 10:15
  • $\begingroup$ Thank you @Jim for the comments. (1) I make it explicit I was considering the $R^2$ coefficient. (2) Exactly, although there may be a statistically significant level between variables, I need to quantify it. (3) I'm not experienced in sharing data here, I hope it is fine using dput R function. $\endgroup$ – tbnsilveira Oct 14 '19 at 20:13
  • 1
    $\begingroup$ I'm a bit unclear on how performance relates to the two variables (throughput and traffic). Are they both measures of performance (e.g. would increase in traffic be considered an improvement)? Or, is performance measured by throughput, which is considered a function of traffic? $\endgroup$ – user20160 Oct 14 '19 at 20:40
  • 1
    $\begingroup$ Your question appears to be at odds with your stated objectives: neither $R^2$ nor some distance between distributions will measure the extent to which the optimization may have improved anything. Moreover, the plots strongly suggest the ordinary least squares models do a good job of describing the distributions of conditional throughputs. $\endgroup$ – whuber♦ Oct 14 '19 at 21:06
  • 1
    $\begingroup$ Why not? The graph you show provides convincing evidence of a systematic difference and will enable you to estimate that difference with reasonable precision. $\endgroup$ – whuber♦ Oct 15 '19 at 16:27
1
$\begingroup$

I'm not familiar with network stuff... but would this work?

$$T_i = \beta_0 + \beta_1 F_i + \beta_2 \mathbb{I}[\text{After}_i] + \varepsilon_i$$

Where $T_i$ is the throughput, $F_i$ is the traffic, and $\mathbb{I}[\text{After}_i]$ is the dummy whether the observation is before or after treatment. This would force the correlation between $T$ and $F$ to be the same ($\beta_1$).

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you @Art for the answer. However, my problem is not in finding a linear model that fits the data, but in finding a way to measure how distant both classes are from each other. In other words, how could I measure the improvement brought by the new network architecture? $\endgroup$ – tbnsilveira Oct 14 '19 at 20:17
  • 2
    $\begingroup$ The coefficient $\beta_2$ quantifies the expected improvement with a linear model, which based on the plot you posted, is a good approximation for how the average throughput varies with traffic, both before and after the network was changed. What you would get is essentially the shift of the line in your plot (because the slopes are almost the same even without restricting them to be), with confidence intervals. This is a very common way of measuring improvement. If you can say why you don't think the average improvement is the measure you want, we might be able to suggest another. $\endgroup$ – CloseToC Oct 14 '19 at 21:00
  • $\begingroup$ @CloseToC, I agree with you that in some cases, this methodology can be reasonable. However, it does not apply to my problem. I can't compare the results, in this case, through models that represent 51% and 66% of my measurements. It would imply a bias that I need to avoid. I mean, it is clear that there is an improvement in the traffic variable, but since I need to quantify this difference, I should find a model with higher accuracy. $\endgroup$ – tbnsilveira Oct 15 '19 at 16:24
  • $\begingroup$ Ah got it. It seems like you don't have the problem with the method... Just that R^2 is a bit too low for your application (liking?) In some fields, R^2 of 0.6 is quite substantial. Is finding other variables that might explain the error terms an option? It's hard to say what to do without having the field-specific knowledge... But with only T's and F's I'm not sure there's much more you could do. $\endgroup$ – Art Oct 15 '19 at 16:55
  • $\begingroup$ @tbnsilveira, R squared tells you how predictable throughput is with network traffic relative to using its mean as a predictor (in a linear model, but again, the plot shows that can barely be improved upon). This is irrelevant for your stated objective. If the lines were completely horizontal and the R squared was 0, you would still get a quantification of how much the optimisation of the network improved throughput with the regression model. $\endgroup$ – CloseToC Oct 15 '19 at 17:06
0
$\begingroup$

best indicator of change would be avg vertical distance between these two lines after all you want to know what is the improvement for given traffic value, so avg vertical distance would be avg improvement

I may be wrong but I would expect that this equality should be true (more less) for architecture that is fixed

$traffic*throughput = CONST$

so maybe hiperbolic regression would be better?

also if above equality is true then you would have to produce value $(1-\frac{CONST_{new}}{CONST_{old}}) * 100\%$ to have information about relative increase in performance

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.