Release It! - Chapter 10.4 Tune the Garbage Collector

In Java applications, garbage collection tuning is the quickest and easiest way to see some capacity improvements. An untuned application running at production volumes and traffic will probably spend 10% of its time collecting garbage. That should be reduced to 2% or less. You can get visibility into the garbage collector’s behavior by passing the -verbosegc argument to the JVM at start-up time. If you are using Java 5 or later, then you can use the jconsole tool that comes with the Java SDK. Once you can see the garbage collection patterns, tuning the garbage collector is largely a matter of ensuring sufficient heap size and adjusting the ratios that control the relative sizes of the generations. Perfectly tuned (if there is such a thing) settings for one release can be totally wrong for the next release so always retune for each release.

• Tune the garbage collector in production - User access patterns make a huge difference in the optimal settings, so you can’t tune the garbage collector in development or QA.
• Keep it up - You will need to tune the garbage collector after each major application release. If you have an annual demand cycle, you will also need to tune it at different times during the year, as user traffic shifts between features.
• Don’t pool ordinary objects - The only objects worth pooling are external connections and threads. For everything else, rely on the garbage collector.

 One question I have is how do you tune in a production environment without adversely impacting the business?  One idea I have is to observe one of the nodes in the cluster and use that data to tune all of the nodes' settings.