Finally I got some time to write something (useful I hope). Today I want to point out one of important HTTP server parameters - the keepAlive settings. This property is used for allowing persistent connections to IHS. By default this option is on and it has got couple of others parameters which tune the timeout of keepAlive or maximum keep alive requests.
The keepAlive options are good for example for apps where users stay logged in and often communicate with the server because IHS wont have to initialize the whole connection procedure from scratch so this acts something like a connection cache. In many other cases though, this is not a good option to use.
Couple of days ago I had a case of a client who had an ESB with IHS servers in the front and its performance was very poor, there were a lot of timeouts and the environment could work stable just for a couple of hours. One of important changes that solved the problem was turning off the keepAlive option.
On this chart you can see the traffic on IHS before the change.
You can notice, that the IHS used all threads for processing, they kept the threads instead of releasing them which cosed the degrading performance - the incoming requests had to wait longer and longer to process.
On the next chart you can see the traffic after the change (with the historic view of previous behavior).
IHS are now stable and the original Threads pool are big enough to process the request in the real time. Another interesting thing is, that the KA parameter doesn't really show that it causes the problem, because this value is quite low - but mainly the reason for that is that during big load all available connections are immediately occupied.
Documentation of IHS says that setting keepAlive will result in higher CPU utilization (which is logic because servers will have more work with creating new connections), but the effect that I measured was the opposite. NMON showed an average CPU utilization to be lower by about 5 to 10%. I can explain this because before the change IHS were heavily loaded with new requests which the server couldn't process - in the keepAlive off scenario IHS managed to offload the incoming traffic in real time and there were no repeating calls to it that it would have to handle.
All in all - the Apache HTTP Server on which IHS is based, has a real potential and though it runs very fast even with default settings, we have to remember that it will influence the rest of the system performance since it is one of the first components in the system architecture (in most cases) and the bigger gap is there, the slower whole system will run.