5
$\begingroup$

I'm trying to analyze a series of request on our web server. I have the latency of each request with the associated timestamp that the request was made.

What I suspect is that higher latencies occur when requests are clustered in time, but I'm not sure about what the timescale even is. It could be requests made within a 5s, 1s, or even 1ms window.

I put together a rudimentary python script to use a sliding window to calculate the request density and compare that with the current latency but even as I tweaked the window size between a few milliseconds and several seconds I couldn't see any obvious correlation.

Crucially, I think the problem is that I don't know what the time period of the latency/request density is, and therefore how big to make my window.

Am I barking up the wrong tree? Or does my method sound roughly correct, suggesting there is no correlation?

Some people have asked for sample data, so I've uploaded some here

$\endgroup$
6
  • $\begingroup$ The method seems fine to me, but hopefully someone can come along with experience in this domain and provide a more authoritative answer. $\endgroup$ – shadowtalker Sep 4 '16 at 12:25
  • 1
    $\begingroup$ To get useful answers you have to provide example data, otherwise you are just getting confirmation of your own analysis $\endgroup$ – seanv507 Sep 6 '16 at 0:13
  • $\begingroup$ Your method sounds OK to me, though as above sample data would help. At some point you have to trust your data, and if you have tried your analysis with a variety of time windows you might genuinely have a null result. You might also consider recording, for each request at the time it was made, how many other requests were still outstanding as an alternative measure of request density. That gets around the time-window-sizing issue somewhat. $\endgroup$ – Upper_Case Sep 12 '16 at 17:14
  • $\begingroup$ Thanks for your inputs - I've uploaded sample data above $\endgroup$ – DaveBensonPhillips Sep 12 '16 at 17:26
  • $\begingroup$ Dave the sample data needs to be regenerated eg 16.607601s,1.47E+09 $\endgroup$ – seanv507 Sep 12 '16 at 18:24
1
$\begingroup$

Your methodology sounds OK, but it is entirely possible that no matter the size of your window, you may still find that there is no signal produced by the higher amount of requests. Some backgound:

Web servers often rely on a pool of backend available process threads that run very far under the surface of the application (to the point of not 100% transparent in logs). Often, web servers are so efficient and well-tuned that they rarely max their out under normal circumstances. This is a good thing - running a server with sub-optimal resources would be like driving your car on empty and only ever refueling to get to the next gas station! And as such, it's likely that you'll never be able to find any really strong evidence to prove a point by parsing your logs in this way...

What may be more effective is to use a service like Locust to load-test your app and then gather statistics this way. Loadtest is another available on NPM. These kinds of applications can help you find the numerical limits of your application in a sandboxed environment, which can be really nice for finding out where your run out of gas, keeping with the car analogy.

$\endgroup$
1
  • $\begingroup$ Thanks Derek. In our particular case the problem is (was) that we were trying to do run some pretty complex algorithms in the space of a web request. The heavy number crunching and potential for GIL-hogging/swapping (Python) I believe were to blame for overlapping requests becoming very slow. Have solved it since by moving this into a micro-service and using a more performant language. $\endgroup$ – DaveBensonPhillips Apr 2 '17 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.