Release It! - Chapter 4.5 Blocked Threads

Modern programming languages can be used in large systems partly because of their ability to deal with concurrency.  Concurrency, unfortunately,
provides another way for a system to fail: Navel Gazing.  Navel Gazing is a term used to describe when all the threads are sitting around waiting for some impossible event, which means that despite that the runtime hasn't crashed, your system isn't doing any work.  There are four major issues around the problem:

• error conditions and exceptions create too many possible paths to test
• unexcpected interactions can introduce problems in previously safe code
• timing around thread interactions is crucial to manifesting this type of problem so you usually see it under times of high load where concurrent requests are more likely
• developers never test their code against 10,000 concurrent users

 From the business perspective, if the sytem can't generate revenue then the system is dead -- no arguing over whether it has crashed, hung or is just having a bad day.  Supplement internal monitoring (log files, port monitors, etc) with external monitoring (a client program sitting outside the data center running synthentic transactions -- an analog for a real live user).  The upshot is that it is difficult to get concurrent code right.   Your best shot is to use carefully crafted code, preferably proven libaries, especially the concurrency objects in Java 5 and above. Rolling your own pooling or caching library is expensive to get right, so don't do it -- stand on the shoulders of giants instead.  Try to avoid synchronizing methods on your domain objects -- it is a bad design smell. Instead, ensure that each thread gets its own copy of the object.  This helps to ensure that your code will work in a clustered environment and reduce collisions between threads, which improves throughput.  In short, thread blocking issues are hard to spot.  Third-party client libraries are notorious for causing blocking issues and are, typically, opaque.  Write learning tests that try and break the library in various ways.  Tie up connections, run it concurrently on lots of threads, reduce the amount of available memory, etc.  Protect your code calling the library. Use timeouts if available.   Use a thread pool that runs the API so that you can control timeouts if the API doesn't provide it natively.  Be aware that if you start using worker threads to run API calls, you'll need a firm grasp of concurrency
in your programming language.   Try and beat up the vendor before insulating your Integration Point with worker threads.  Blocked or slow responding threads typically appear around Integration Points and can form a feedback loop that can quickly result in a cascading failure. 

Blocked threads are the cause for a high proportion of system failures.  Scrutinize resource pools and make sure they are configured for concurrent access. Blocked database connection pools can lead to blocked threads, incorrect exception handling and cascading failures.  Use timeouts so no thread waits for ever.  Don't use the no arg wait() method, use the form that accepts a timeout instead. Use proven libraries.  Writing correct concurrent code is hard and leverage the work others have done before you. Beware of code you cannot see -- test and review third-party client libraries because they will fail and it is best if you have an idea of how.

Release It! - Chapter 4.4 Users

Every user consumes system resources so you need to know how your system reacts to abnormally high demand.  Try and keep user sessions as small as possible.  It is probably better to re-execute an operation than it is to cache data for 30 minutes that is never accessed and puts the system at risk.  Using Java's SoftReferences can be a good compromise but requires more complex coding.  You have to be aware that some users are more expensive than others because they do more -- like buy stuff.  There is no direct defense against expensive users but you can test and ensure your system can handle double the number of current level of expensive users (which implies that you have a way of figuring out how many expensive users you have).   Web sessions are the weakness of web applications so be aware of them.

Some users are bad either by accident or by design.  Some users may deviate from the expected work flow and stress your system.  Some users are actually programs, such as spiders or robots, that can stress your system.  Legitimate search engines will honor a robots.txt file which can allow you to control access to your site.  Others don't so you have two choices:  add firewall rules to prevent access from unwanted ip blocks or create a "terms of service" agreement and sic lawyers on the offending parties. 

Remember that users consume memory so make sure that sessions are used as caches and not the "database of record" so that they may be safely purged when memory becomes scarce. Some users are weird and will do strange things which cannot be defended against.  Malicious users exists and you can help your cause by knowing your network design and keeping all your software patched and up to date.  Users gang up on you when your site is the focal point  of sudden interest, like when referenced by Slashdot.  Use special stress tests to ensure your system can handle the extra load.

Release It! - Chapter 4.3 Cascading Failures

A Cascading Failure is when a crack in one layer triggers a crack in a calling layer.  The failure jumps between the layers when bad behavior in the caller gets triggered by a problem in the called layer.  Resource pools often get exhaused in this scenario and Integration Points without timeouts is a sure way to cause a Cascading Failure.  Cascading Failure are crack accelerators so preventing them is very important.  Stop cracks from spanning layers and make sure calling layers can still function even after a lower layer goes dark.  Examine your resource pools to ensure they are safe.  Safe pools always limit the time a thread can wait to check out a resource.  Defend against Cascading Failures by using Circuit Breaker and Timeout  patterns.  Circuit Breakers prevent call outs to sick layers and Timeouts ensure that you can return from a call to a sick layer.

Release It! - Chapter 4.2 Chain Reactions

Horizontal scaling is scaling by adding additional servers.  Vertical scaling is scaling via larger boxes.  Horizontal scaling uses load balancers to provide fault tolerance.  Horizontally scaled systems can exhibit a failure mode known as Attack of Self Denial.  The essence of the failure is that when a single node goes dark, the extra load on the remaining servers cause them to fail, typically due to a resource leak or load related bug.  Since all servers have the same bug, they will eventually fail in the same load-related way forming a Chain Reaction.  As more and more servers fail, the remaining servers fail faster and faster until the entire layer is dead.  The only way to stop the cascade is to fix the leak.  You can attempt  to group your servers via the Bulkhead pattern and break the chain reactions into separate chain reactions, which might give you  enough time to bring up another set of servers.  Accept this: one down server places the remaining servers in jeopardy.  A dead layer will then endanger the dependent layers.  Hunting for resource leaks is important because resource leaks are a primary killer of systems. You must also hunt for obscure timing bugs by load testing your system.  If you use the Bulkhead pattern on the server side and the Circuit Breaker pattern on the client side, you can prevent chain reactions that can take out entire layers.

Release It! - Chapter 4.1 Integration Points

Chapter 4 is an interesting chapter and a very large one so I'll be breaking up my posts into sub-sections of the chapter.  In this chapter, some common scenarios are identified and labeled as Antipatterns.  Most times a solution is also recommended but we fall prey to the chicken-and-egg scenario: some of the solution patterns called out haven't been described yet so you have to guess as to what they might mean and come back later to see if things make sense in the context of the Antipattern.

One factor that affects a system's ability to be effectively operated in the field is known as Interactive Complexity.  Interactive Complexity comes from systems that have enough moving parts and internal dependences that operators cannot keep a complete or accurate mental model of the system.  Hidden linkages between components can lead to Problem Inflation where an operator turns a dial thinking that it will solve a problem but instead, due to the operator's failure to fully understand all the relationships between components, makes the situation worse.  Tight coupling between parts of the system allow for cracks to travel to other parts of the system, making diagnosis more difficult and increasing the effects of the crack. Several Antipatterns have been discovered which can be used  to identify situations which are likely to cause failures in the system.  Things will break and it is up to you to identify failure points and plan for them.

Integration Points with other systems is the number one stability killer.  Most integrations make use of sockets which can be the cause of many headaches if you don't understand some of the nuances and code defensively.  For example, the amount of time it takes to detect a connection timeout varies between OSes and usually is measured in terms of minutes.  Blocking application threads for minutes at a time waiting for a connection failure is a sure way to cripple a system.  One idea is to use to run tcpdump on the production box and analyze the capture file on a non-production box.  The tcpdump program is much lighter weight than Wireshark and provides the same information.  You can use Wireshark to analyze the capture file on a non-production machine.  Keeping a copy of The TCP/IP Guide or TCP/IP Illustrated around can help guide you through network related failures.  One thing to remember is that TCP/IP was created in a time before firewalls existed so there are classes of failures around the interaction between the firewall and the TCP/IP stack.  Idle connections and firewalls are a bad mix.  HTTP client libraries can also be a problem because HTTP sits atop TCP/IP, which means it also has timeout related issues.  Most HTTP libraries are not cynical and do not provide timeouts in their API, making it harder to code defensively.  The  Apache HTTP Client library does support timeouts and is one API that can be used in cynical programs.  Vendor APIs, such as database drivers, are another source of integration pain.  Many times the server code is rock solid and cynical but the client-side APIs are not, which eventually causes your integration threads to block. Since you have no direct control over the API, all you can do is to identify bugs and work with your vendors.  Patterns exists to help combat integration point failures, such as Circuit Breaker and Decoupling Middleware.  One tactic you can use is to write a test harness that will verify that your software is cynical enough.  Make sure it can simulate integration and network failures.  You'll also want to run your tests under load, using JMeter or some other load tester.  Every integration point will fail.  Count on it and be prepared.  Failures never come in the form of a nicely formatted error message complete with a cause and solution.  Instead, you see slowness in responses, hanging, etc.  In order to debug what is going on, you'll likely have to peel away some abstractions and look deeply into the integration using decompilers and network sniffers.  One sad fact is that a remote system's failure will become your problem if you are not defensive enough.

Release It! - Introducing Stability

Enterprise software must be cynical meaning that it trusts no one, including itself.  Cynical software expects bad things to happen and has plans to deal with others failing to hold up their part of the bargain.

"A System is the complete interdependent set of hardware, applications and services required to process transactions for users." Translation: a system is everything needed to provide a working solution for the users, right down to the power supply and network cables.  A stable system is resilient to transient impulses, persistent stresses or component failures. In short,  the user can still get work done despite failures in some parts of the system.  A stable architecture costs the same to implement as an unstable one so doesn't it make sense to chose the former? An Impulse is a rapid shock to the system, such as a million hits to your website because of an article on Slashdot.   A Stress is force applied over an extended period of time, such as ever increasing response time from your database server or a credit card processor who can't handle your transaction load.  A Strain is a change of the shape of the system due to Stress, such as higher RAM usage due to the slow processing of an external service.

An enterprise system is supposed to run for a long time.  What is the working definition of "long time"?  How about the time between code deployments.  If you deploy new code once a year, then long time for you is 12 months.

Longevity tests are typically not run but not doing so will guarantee that bugs will appear.  If possible, dedicate a machine that runs a load testing tool that hits the machine with medium level transactions but for an extended period of time.  Make sure to put in slack periods of time to simulate slow periods in the middle of the night.  If you can't set up an entire environment, test the important parts and stub the rest.  If you don't do your own longevity testing, your production system will become your longevity test and you won't be happy.

Some component of the system will fail before everything else does and that component is known as a Crack.  The original trigger, plus the way the Crack spreads to the rest of the system coupled with the result of the damage is known as a Failure Mode.  It is in your own best interest to try and identify the Failure Modes and create a system that can withstand them.

A Crackstopper is software crumple zone designed to absorb the impact of failure and keep the rest of the system safe.  This allows you to decide what parts of the system are critical and keep cracks away from  them.  A software shock absorber, if you will.  Cracks propagate so you must do your best to prevent that.  Examples in the case study  of possible Crackstoppers include, TCP timeouts, partitioning servers better, using HTTP instead of RMI or using an asynchronous messaging protocol to decouple things.

Each failure results in a chain of events and each event in that chain can accelerate, slow or stop a crack.  Thumbrule: tight coupling accelerates cracks so reduce coupling to retard the spreading of a crack. Brute-force analysis of every resource call, I/O operation or external API call in order to identify potential failure scenarios is impractical.  Luckily, patterns do exist that can help prevent cracks from propagating and will be presented in future chapters.

Release It! - Case Study: The Exception That Grounded An Airline

This was an interesting chapter in that it described a real scenario that involved the author.  An airline experienced a 3 hour downtime of their check-in system that, due to the need to involve more humans to deal with the backlog, impacted most of the airline for almost 9 hours.  One interesting point was to note the downstream effects of the outage which ranged from union mandated overtime, to SLA contract breaches and potential bonus money being lost.  The team was able to bring the system back but people wanted to know what happened so the author had to fly on-site to perform a post-mortem and determined what went wrong.  Due to strained relations between the software vendor and the end-user, source code was not made available so decompilation of the Java byte codes coupled with thread dumps and logs provided the answer: an EJB method had chosen to not deal with an SQL exception when closing a connection.  This one decision caused all of the server's thread to hang waiting for a database connection that was never going succeed.  The author's emphasis was not on better testing of the system in the hope of revealing that particular oversight but, instead, was at the integration of the multiple systems.  His view is that the death of one of the systems should not halt the entire solution and promises to provide some guidance on how to keep a complex system healthy during times of stress.