Effect on server when memory runs out and swapping starts
Archive - Originally posted on "The Horse's Mouth" - 2009-02-26 08:05:54 - Graham EllisHere's an interesting example from a web server showing what happens when real memory runs out and the system starts swapping ... it can be something of a downwards spiral in terms of performance, as the majority of the resources are essentially tied up by the swapping / keeping going at all once memory is full, and a hard to recover backlog of processes can build up.
First, an example where the system is running in memory:
And this is what can happen with a slightly higher arrival rate, and / or a slightly slower processing rate such that a queue builds up:
You'll notice by comparing these two (look at the purple highlights):
1. The load average has rocketed from 0.23 jobs waiting (average in last minute) to 22.59 jobs.
2. The cpu utilisation remains low - under 10% - for user processes
3. The cpu idle time has dropped from nearly 90% to zero, with all the extra resource being burned up by device wait time - 93.6% no less.
4. Swap space has only increased slightly in its use, but every process in this machine is active and the whole thing hsa started thrashing which means a queue has built up ...
5. There's been a rise from 115 to 191 total tasks (remember that the majority of the initial 115 will be operating system background processes, so that this is a truly massive increase)
6. The process table itself is showing details of just how / where the queue has built up.
See here for some tuning information for httpd if it's the web server that's consuming all the space (there are MySQL notes there too!)