I had been working on a project for a client where the security restrictions make it extremely difficult to use any new tool. Luckily we have almost all application layers and systems being indexed by Splunk.
A service oriented architecture is being followed where each layer is available as a service. A simple gui dashboard of the web application makes call to services that internally call to other services and this goes on till a depth of 4-5 layers.
With the help of our load test tool we are able to get the overall latency of a service call from the gui. With the help of Splunk we are able to get the latency of each layer, we are able to pin point which layer is taking the maximum time. Splunk is also proving helpful in tracking errors and exceptions, if any of the layers has any issue, it is very easy to find about which layer broke the flow. Had Splunk not been there it would have been very resource consuming and difficult to pin point the bad layer.