Release It! - Chapter 4.7 Scaling Effects

Point-to-Point communication is a primary area where you will see scaling effects.  The number of connections is the square of the number of instances and the number gets big quickly.  Testing out these failures is next to impossible.  You can try:

• UDP broadcasts - ineffecient since the whole network hears all messages
• TCP multi-casting - more efficient because only interested parties get messages
• publish/subscribe messaging - more infrastructure needed
•  message queues - more infrastructure needed

Do the simplest thing that will work.  

Shared resources are another area where Scaling Effects come into play.  If the service is redundant and non-exclusive then you are okay -- just add more servers if needed.  Exclusive access is problematic.  Request queues back up waiting for their turn and the situation gets worse as more transactions are attempted.  Eventually the backlog fills up and the requests are dropped at the TCP/IP layer and then things get really ugly.  Share-Nothing architectures make it more difficult to fail over -- somebody has to migrate the user's session to another server, which is likely a shared resource.  One compromise it to reduce the fan-in (number of servers calling into a shared resource), perhaps only having servers pair-up for fault tolerance instead of everybody knowing about everyone else.

• pay attention to the differences in QA and Production environments - things work fine on a small scale but melt in a large one
•  watch out for point-to-point communications -  scales badly but might work if you know the number of servers will remain small.
• watch out for shared resources - they bottleneck, restrain capacity and are a stability threat.  Stress test shared resources heavily and make sure clients will keep working if the resource slows or fails altogether.