Software Design Blog

We are liable for the reliability of our apps, which work when tested but fail in production environments due to unexpected temporary conditions. Errors can occur due to an intermittent service, infrastructure fault, network issue or explicit throttling. If the operation is retried a short time later (maybe a few milliseconds later) the operation may succeed.

A lazy brain can quickly solve this problem without a lot of planning, flexibility or configuration in mind as shown in this MSDN sample that performs a retry when the SMTP mail client fails.

  
            try
            {
                client.Send(message);
            }
            catch (SmtpFailedRecipientsException ex)
            {
                for (int i = 0; i < ex.InnerExceptions.Length; i++)
                {
                    SmtpStatusCode status = ex.InnerExceptions[i].StatusCode;
                    if (status == SmtpStatusCode.MailboxBusy ||
                        status == SmtpStatusCode.MailboxUnavailable)
                    {
                        System.Threading.Thread.Sleep(5000);
                        client.Send(message);
                    }
                }
            }

The problem is how do you retry a failed operation correctly?

The solution is to create an error detection strategy to determine if the error can be retried and use a retry policy to define the number of retries and frequency between retries.

This post extends the previous example by adding retry logic to the SMTP Mail Client. The aim is to illustrate how to improve the reliability of an app by making it more robust.

Retry Solution - Done Properly

Let’s get started by implementing clean retry code in 4 easy steps.

Step 1 – Add the transient fault handling NuGet package

The Microsoft transient fault handling NuGet package contains generic, reusable and testable components that we will use for solving the SMTP mail client retry problem.

Add the EnterpriseLibrary.TransientFaultHandling NuGet package to the Orders.Implementation project.

Step 2 – Create a transient error detection strategy

Transient (temporarily) faults are errors that can be retried. Here are a few examples of transient errors that may work if retried:

• The host is not responding or is unavailable
• The server is busy or the connection was dropped

Here are a few examples of permanent errors that won’t work even if retried:

• Authentication failed (invalid username or password)
• Bad request (invalid email address, attachment not found)

Let’s create a detection strategy that will detect SMTP Mail errors that can be retried.

  
    public class SmtpMailErrorDetectionStrategy : ITransientErrorDetectionStrategy
    {
        public bool IsTransient(Exception ex)
        {
            var smtpFailedException = ex as SmtpFailedRecipientsException;
            if (smtpFailedException == null) return false;

            return smtpFailedException.InnerExceptions
                    .Any(mailEx => mailEx.StatusCode == SmtpStatusCode.MailboxBusy ||
                                   mailEx.StatusCode == SmtpStatusCode.MailboxUnavailable);
        }
    }

Step 3 – Create the SMTP Retry Decorator

Based on to the single responsibility principle (SRP), the mail client should only be responsible for one thing, which is sending mail. Adding error detection and retry logic violates the SRP since it shouldn't be it's problem.

The decorator pattern is great for extending an existing class. Just imagine how large the class will become if we continually add cross-cutting functionality to it such as retries, logging and performance monitoring.

Here is the existing SMTP mail client used in the previous post:

We are going to decorate the SMTP mail client with a retry mail client decorator as shown below:

  
    public class SmtpMailClientRetryDecorator : IMailClient
    {
        private readonly IMailClient _next;
        private readonly RetryPolicy _retryPolicy;

        public SmtpMailClientRetryDecorator(IMailClient next, RetryPolicy retryPolicy)
        {
            if (next == null) throw new ArgumentNullException("next");
            if (retryPolicy == null) throw new ArgumentNullException("retryPolicy");
            _next = next;
            _retryPolicy = retryPolicy;
        }

        public void Send(string to, string subject, string body)
        {
            _retryPolicy.ExecuteAction(() => _next.Send(to, subject, body));
        }
    }

Step 4 – Compose the Solution

The transient fault handling library provides the following retry strategies:

Let’s register the fixed interval retry strategy in the Unity IoC container as shown below. If you are new to Dependency Injection (DI), read this post.

  
            // This should be defined in app settings
            const int maxRetries = 5;
            var retryInterval = TimeSpan.FromSeconds(2);

            _container.RegisterType<FixedInterval>(
                        new InjectionConstructor(maxRetries, retryInterval));
            _container.RegisterType<RetryPolicy<SmtpMailErrorDetectionStrategy>>(
                        new InjectionConstructor(new ResolvedParameter<FixedInterval>()));
            
            _container.RegisterType<IMailClient, SmtpMailClient>(typeof(SmtpMailClient).FullName);
            _container.RegisterType<IMailClient, SmtpMailClientRetryDecorator>(
                        new InjectionConstructor(
                               new ResolvedParameter<IMailClient>(typeof(SmtpMailClient).FullName),

And for those who may think this is over-engineered and too complicated then this is how you can construct the retry policy in the SMTP mailer class – but of course you will run in to the problems discussed in earlier posts!

  
const int maxRetries = 5;
var retryInterval = TimeSpan.FromSeconds(2);
var policy = new RetryPolicy<SmtpMailErrorDetectionStrategy>(
                    new FixedInterval(maxRetries, retryInterval));
policy.ExecuteAction(() => client.Send(mesage));

Summary

This post showed how to implement a flexible, configurable solution using the Microsoft Transient Fault Handling Application Block to retry transient failures in order to improve app reliability.

If we refer to the original MSDN code sample, we can see a similar pattern by evaluating specific transient error codes (transient error detection strategy), pausing for 5 seconds (retry policy - retry Interval) and retrying the send operation once (retry policy - retry count).

The problem is how can we measure the queue processing throughput? Or in a production environment, how can we add a performance counter without modifying our existing infrastructure?

The solution is to add a decorator pattern that will act as a proxy between the QueueMonitor class and the actual command processing class to intercept the communication and add instrumentation.

The previous post used the competing consumer pattern to process a queue concurrently. The message queue strategies post used a single processor to process a queue.

In this post, we will add instrumentation to the single queue processor and the competing consumer queue processor to compare performance.

Setup

The experiment will add 512KB messages to a queue to simulate a decent workload. The intention is to read the messages from the queue and call a File SMTP mailer server that will write the emails to disk as fast as possible.

The core components such as the Queue Monitor were covered in the Message Queue Delivery Strategies post. The queue monitor is responsible for reading messages from the queue and pushing the message content to a command to execute.

The classes that will be used in this experiment are listed below.

  
    public class FileSmtpMailer : IMailer
    {
        private readonly string _path;

        public FileSmtpMailer(string path)
        {
            if (path == null) throw new ArgumentNullException("path");
            _path = path;
        }

        public void Send(string message, string sender, string recipients, string subject)
        {
            using (var client = new SmtpClient())
            {
                // This can be configured in the app.config
                client.DeliveryMethod = SmtpDeliveryMethod.SpecifiedPickupDirectory;
                client.PickupDirectoryLocation = _path;

                using (var mailMessage = new MailMessage(sender, recipients, 
                                                         subject, message))
                {
                    mailMessage.IsBodyHtml = true;
                    client.Send(mailMessage);
                }
            }
        }
    }

    public class MailProcessorCommand : ICommand<OrderMessage>
    {
        private readonly IMailer _mailer;

        public MailProcessorCommand(IMailer mailer)
        {
            if (mailer == null) throw new ArgumentNullException("mailer");
            _mailer = mailer;
        }

        public void Execute(OrderMessage message)
        {
            if (message == null) throw new ArgumentNullException("message");
            _mailer.Send(message.Body, message.Sender, 
                         message.Recipients, message.Subject);
        }
    }

Instrumentation

The TimerCommandDecorator decorator will be placed in between the QueueMonitor and the MailProcessorComamnd in order to measure the performance throughput.

  
    public class TimerCommandDecorator<T> : ICommand<T>
    {
        private readonly ICommand<T> _next;
        private int _messages;
        private Stopwatch _stopwatch;

        public TimerCommandDecorator(ICommand<T> next)
        {
            if (next == null) throw new ArgumentNullException("next");
            _next = next;
        }

        public void Execute(T message)
        {
            if (_stopwatch == null) _stopwatch = Stopwatch.StartNew();

            _next.Execute(message);

            _messages += 1;
            Console.WriteLine("Processing {0} messages took {1} sec", 
                              _messages, _stopwatch.Elapsed.TotalSeconds);
        }
    }

The TimerCommandDecorator implementation can easily be swapped out with a perfmon version that will increment the performance counter for each message.

Single vs Competing Consumers

Let's run the experiment by observing the throughput difference between the single processor and competing consumer receivers by processing 1000 messages.

The single processor version:

  
            using (var queue = new MessageQueue(QueuePath))
            {
                queue.Formatter = new BinaryMessageFormatter();

                var message = new OrderMessage()
                {
                    Recipients = "[email protected]",
                    Sender = "[email protected]",
                    Subject = "Hello World",
                    Body = emailBody // loaded from a file
                };

                // Send messages to the queue
                for (var i = 0; i < 1000; i++)
                {
                    queue.Send(message);
                }               

                var fileMailer = new FileSmtpMailer(@"C:\temp");
                var command = new MailProcessorCommand(fileMailer);
                var timerCommand = new TimerCommandDecorator<OrderMessage>(command);
                var queueMonitor = new QueueMonitor<OrderMessage>(queue, timerCommand);
                queueMonitor.Start();
            }

Processing 1000 messages took 47.3004132 sec

The competing consumer processor version with 10 processors:

  
                // Create 10 processors
                var commands = new List<ICommand<OrderMessage>>();
                for (var i = 0; i < 10; i++)
                {
                    commands.Add(new MailProcessorCommand(fileMailer));
                }

                var command = new CompositeCompetingCommandProcessor<OrderMessage>(commands);
                var timerCommand = new TimerCommandDecorator<OrderMessage>(command);
                var queueMonitor = new QueueMonitor<OrderMessage>(queue, timerCommand);
                queueMonitor.Start();

Processing 1000 messages took 16.241723 sec

We could experiment using various numbers of concurrent receivers to determine how performance will be affected.

The competing consumer pattern is a great option to fully utilize resources but it may have a negative impact on other tenants of the system. For example, if we call an external API with too many requests, it may appear like a denial-of-service attack.

Summary

The advantage of the competing consumers pattern is the flexibility to configure the number of concurrent consumers to produce a balanced outcome. It is however difficult to determine the throughput and performance impact without instrumentation in place to measure resource utilization.

An approach was covered to add performance instrumentation. The performance metrics revealed that the queue processing throughput can be improved significantly using the competing consumer queue processor compared to the single queue processor.

Processing everything in real-time is easy but often makes an application tightly coupled, prone to failure and difficult to scale.

Let’s assume a user just clicked on the purchase button to kick off the following workflow:

1. Save the order
2. Send a confirmation email (mask the credit card number)
3. Update the user’s loyalty points in an external system

The problem is that non-critical workloads (step 2 and 3) can negatively impact an application's performance and recoverability. What happens when the mail server is down and the email confirmation fails? How do we monitor failure? Do we roll the transaction back or replay the workflow by taking a checkpoint after each successful step?

The solution is to ensure that application flow isn't impeded by waiting on non-critical workloads to complete. Queue based systems are effective at solving this problem.

This post will use a purchase order application that accepts XML requests. The application will sanitise the credit card details by masking most of the digits before sending the request to a mailbox.

Setup

The following XML document will be used as the purchase order. It is about 512KB to simulate a decent payload.

  


  
    Test User
    
123 Abc Road, Sydney, Australia
    
      Visa
      4111111111111111
    
  
  
    
      Gambardella, Matthew
      
    
   ...

  
    public interface IMailer
    {
        void Send(string message);
    }

    public class PurchaseOrderProcessor
    {
        private readonly IMailer _mailer;

        public PurchaseOrderProcessor(IMailer mailer)
        {
            if (mailer == null) throw new ArgumentNullException("mailer");
            _mailer = mailer;
        }

        public void Process(string xmlRequest)
        {
            var doc = new XmlDocument();
            doc.LoadXml(xmlRequest);

            // Process the order

            _mailer.Send(xmlRequest);
        }
    }

  
    public class FileSmtpMailer : IMailer
    {
        private readonly string _path;
        private readonly string _from;
        private readonly string _recipients;
        private readonly string _subject;

        public FileSmtpMailer(string path, string from, string recipients, string subject)
        {
            if (path == null) throw new ArgumentNullException("path");
            _path = path;
            _from = @from;
            _recipients = recipients;
            _subject = subject;
        }

        public void Send(string message)
        {
            using (var client = new SmtpClient())
            {
                // This can be configured in the app.config
                client.DeliveryMethod = SmtpDeliveryMethod.SpecifiedPickupDirectory;
                client.PickupDirectoryLocation = _path;

                using (var mailMessage = 
                          new MailMessage(_from, _recipients, _subject, message))
                {
                    client.IsBodyHtml = true;
                    client.Send(mailMessage);    
                }             
            }
        }
    }

  
    public class MaskedMailerDecorator : IMailer
    {
        private readonly Regex _validationRegex;
        private readonly IMailer _next;
        private const char MaskCharacter = '*';
        private const int MaskDigits = 4;

        public MaskedMailerDecorator(Regex regex, IMailer next)
        {
            if (regex == null) throw new ArgumentNullException("regex");
            if (next == null) throw new ArgumentNullException("next");
            _validationRegex = regex;
            _next = next;
        }

        public void Send(string message)
        {
            if (_validationRegex.IsMatch(message))
            {
                message = _validationRegex.Replace(message, 
                               match => MaskNumber(match.Value));
            }
            _next.Send(message);
        }

        private static string MaskNumber(string value)
        {
            return value.Length <= MaskDigits ?
               new string(MaskCharacter, value.Length) :
               string.Format("{0}{1}", 
                          new string(MaskCharacter, value.Length - MaskDigits),
                          value.Substring(value.Length - MaskDigits, MaskDigits));
        }
    }

Baseline

Let's establish a performance baseline by running the application with the Null Mailer that doesn't do anything. Refer to the reject the null checked object post if you are new to the null object pattern.

  
    public class NullMailer : IMailer
    {
        public void Send(string message)
        {
            // intentionally blank
        }
    }

    static void Main(string[] args)
    {
        var path = Path.Combine(Directory.GetCurrentDirectory(), "PurchaseOrder.xml");
        var request = File.ReadAllText(path); 

        var nullMailer = new NullMailer();
        var orderProcessor = new PurchaseOrderProcessor(nullMailer);

        var stopWatch = Stopwatch.StartNew();
        Parallel.For(0, 1000, i => orderProcessor.Process(request));
        stopWatch.Stop();

        Console.WriteLine("Seconds: {0}", stopWatch.Elapsed.TotalSeconds);
        Console.ReadLine();
    }

Seconds: 6.3404086

Real-Time Processing

  
            Directory.CreateDirectory(@"C:\temp");
            var ccRegEx = new Regex(@"(?:\b4[0-9]{12}(?:[0-9]{3})?\b
                                       |\b5[1-5][0-9]{14}\b)", RegexOptions.Compiled);

            var path = Path.Combine(Directory.GetCurrentDirectory(), "PurchaseOrder.xml");
            var request = File.ReadAllText(path);
            
            // Use Unity for doing the wiring          
            var fileMailer = new FileSmtpMailer(@"C:\temp", "[email protected]", "[email protected]", "Order");
            var maskedMailer = new MaskedMailerDecorator(ccRegEx, fileMailer);
            var orderProcessor = new PurchaseOrderProcessor(maskedMailer);

            var stopWatch = Stopwatch.StartNew();
            Parallel.For(0, 1000, i => orderProcessor.Process(request));
            stopWatch.Stop();

            Console.WriteLine("Seconds: {0}", stopWatch.Elapsed.TotalSeconds);
            Console.ReadLine();

Seconds: 32.0430142

Background Processing

Let's extend the solution by adding a memory queue to buffer the results. The queue effectively acts as an outbox for sending mail without overwhelming the mail server with parallel requests.

    public class QueuedMailerDecorator : IMailer
    {
        private readonly IMailer _next;
        private BlockingCollection<string> _messages;

        public QueuedMailerDecorator(IMailer next)
        {
            if (next == null) throw new ArgumentNullException("next");
            _next = next;
            _messages = new BlockingCollection<string>();

            Task.Factory.StartNew(() =>
            {
                try
                {
                    // Block the thread until a message becomes available
                    foreach (var message in _messages.GetConsumingEnumerable())
                    {
                        _next.Send(message);
                    }
                }
                finally
                {
                    _messages.Dispose();
                }
            }, TaskCreationOptions.LongRunning);
        }

        public void Send(string message)
        {
            if (_messages == null || _messages.IsAddingCompleted)
            {
                return;
            }
            try
            {
                _messages.TryAdd(message);
            }
            catch (ObjectDisposedException)
            {
                Trace.WriteLine("Add failed since the queue was disposed.");
            }
        }

        public void Dispose()
        {
            if (_messages != null && !_messages.IsAddingCompleted)
            {
                _messages.CompleteAdding();
            }
            GC.SuppressFinalize(this);
        }
    }

Here is a diagram that depicts the collection of decorated mailers to intercept the communication between the PurchaseOrderProcessor and the FileSmtpMailer.

             
            // Use Unity for doing the wiring          
            var fileMailer = new FileSmtpMailer(@"C:\temp", "[email protected]", "[email protected]", "Order");
            var maskedMailer = new MaskedMailerDecorator(creditCardRegEx, fileMailer);
            var queuedMailer = new QueuedMailerDecorator(maskedMailer);
            var orderProcessor = new PurchaseOrderProcessor(queuedMailer);

            var stopWatch = Stopwatch.StartNew();
            Parallel.For(0, 1000, i => orderProcessor.Process(request));
            stopWatch.Stop();

            Console.WriteLine("Seconds: {0}", stopWatch.Elapsed.TotalSeconds);
            Console.ReadLine();

Seconds: 6.3908034 

The drawback of the in-memory queue used in this post is that it requires memory to store messages temporarily before it is processed. It is possible to lose messages if the application crashes or stops unexpectedly before all of the requests have been processed.

These issues can be addressed with a locally persisted queue such as MSMQ or a cloud based queue such as Azure Service Bus. Queues provide many benefits that will be covered in the next post.

Summary