Software Design Blog - Design Patterns

A step-by-step guide to detect errors and retry operations

Tue, 26 Jan 2016 21:35:00 +1300

We are liable for the reliability of our apps, which work when tested but fail in production environments due to unexpected temporary conditions. Errors can occur due to an intermittent service, infrastructure fault, network issue or explicit throttling. If the operation is retried a short time later (maybe a few milliseconds later) the operation may succeed.

A lazy brain can quickly solve this problem without a lot of planning, flexibility or configuration in mind as shown in this MSDN sample that performs a retry when the SMTP mail client fails.

try
{
client.Send(message);
}
catch (SmtpFailedRecipientsException ex)
{
for (int i = 0; i < ex.InnerExceptions.Length; i++)
{
SmtpStatusCode status = ex.InnerExceptions[i].StatusCode;
if (status == SmtpStatusCode.MailboxBusy ||
status == SmtpStatusCode.MailboxUnavailable)
{
System.Threading.Thread.Sleep(5000);
client.Send(message);
}
}
}

Warning! Bad code, do not copy.

The problem is how do you retry a failed operation correctly?

The solution is to create an error detection strategy to determine if the error can be retried and use a retry policy to define the number of retries and frequency between retries.

This post extends the previous example by adding retry logic to the SMTP Mail Client. The aim is to illustrate how to improve the reliability of an app by making it more robust.

Note: The approach used in this post is generic and can be applied to database repositories, cloud services, web services and more.

Download Source Code

Retry Solution - Done Properly

Let�s get started by implementing clean retry code in 4 easy steps.

Step 1 � Add the transient fault handling NuGet package

The Microsoft transient fault handling NuGet package contains generic, reusable and testable components that we will use for solving the SMTP mail client retry problem.

Add the EnterpriseLibrary.TransientFaultHandling NuGet package to the Orders.Implementation project.

Step 2 � Create a transient error detection strategy

Transient (temporarily) faults are errors that can be retried. Here are a few examples of transient errors that may work if retried:

The host is not responding or is unavailable
The server is busy or the connection was dropped

Here are a few examples of permanent errors that won�t work even if retried:

Authentication failed (invalid username or password)
Bad request (invalid email address, attachment not found)

Let�s create a detection strategy that will detect SMTP Mail errors that can be retried.

public class SmtpMailErrorDetectionStrategy : ITransientErrorDetectionStrategy
{
public bool IsTransient(Exception ex)
{
var smtpFailedException = ex as SmtpFailedRecipientsException;
if (smtpFailedException == null) return false;
return smtpFailedException.InnerExceptions
.Any(mailEx => mailEx.StatusCode == SmtpStatusCode.MailboxBusy ||
mailEx.StatusCode == SmtpStatusCode.MailboxUnavailable);
}
}

Step 3 � Create the SMTP Retry Decorator

Based on to the single responsibility principle (SRP), the mail client should only be responsible for one thing, which is sending mail. Adding error detection and retry logic violates the SRP since it shouldn't be it's problem.

The decorator pattern is great for extending an existing class. Just imagine how large the class will become if we continually add cross-cutting functionality to it such as retries, logging and performance monitoring.

Here is the existing SMTP mail client used in the previous post:

We are going to decorate the SMTP mail client with a retry mail client decorator as shown below:

public class SmtpMailClientRetryDecorator : IMailClient
{
private readonly IMailClient _next;
private readonly RetryPolicy _retryPolicy;
public SmtpMailClientRetryDecorator(IMailClient next, RetryPolicy retryPolicy)
{
if (next == null) throw new ArgumentNullException("next");
if (retryPolicy == null) throw new ArgumentNullException("retryPolicy");
_next = next;
_retryPolicy = retryPolicy;
}
public void Send(string to, string subject, string body)
{
_retryPolicy.ExecuteAction(() => _next.Send(to, subject, body));
}
}

Step 4 � Compose the Solution

The transient fault handling library provides the following retry strategies:

Retry strategy	Example (intervals between retries in seconds)
Fixed interval	2,2,2,2,2,2
Incremental intervals	2,4,6,8,10,12
Random exponential back-off intervals	2, 3.755, 9.176, 14.306, 31.895

Let�s register the fixed interval retry strategy in the Unity IoC container as shown below. If you are new to Dependency Injection (DI), read this post.

// This should be defined in app settings
const int maxRetries = 5;
var retryInterval = TimeSpan.FromSeconds(2);
_container.RegisterType<FixedInterval>(
new InjectionConstructor(maxRetries, retryInterval));
_container.RegisterType<RetryPolicy<SmtpMailErrorDetectionStrategy>>(
new InjectionConstructor(new ResolvedParameter<FixedInterval>()));
_container.RegisterType<IMailClient, SmtpMailClient>(typeof(SmtpMailClient).FullName);
_container.RegisterType<IMailClient, SmtpMailClientRetryDecorator>(
new InjectionConstructor(
new ResolvedParameter<IMailClient>(typeof(SmtpMailClient).FullName),

Success! We have added a flexible retry solution to our app.

And for those who may think this is over-engineered and too complicated then this is how you can construct the retry policy in the SMTP mailer class � but of course you will run in to the problems discussed in earlier posts!

const int maxRetries = 5;
var retryInterval = TimeSpan.FromSeconds(2);
var policy = new RetryPolicy<SmtpMailErrorDetectionStrategy>(
new FixedInterval(maxRetries, retryInterval));
policy.ExecuteAction(() => client.Send(mesage));

Summary

This post showed how to implement a flexible, configurable solution using the Microsoft Transient Fault Handling Application Block to retry transient failures in order to improve app reliability.

If we refer to the original MSDN code sample, we can see a similar pattern by evaluating specific transient error codes (transient error detection strategy), pausing for 5 seconds (retry policy - retry Interval) and retrying the send operation once (retry policy - retry count).

Kill the switch with the strategy pattern

Mon, 11 Jan 2016 15:52:00 +1300

The previous post described the challenge of validating credit card numbers based on the credit card type such as visa or master card. Many if conditional checks were performed which was improved using the switch statement.

Why was the switch statement bad?

Let's say you have 5 cases and 10 lines each.

What if we had to add, remove or modify a case? The class has 7 reasons to change, which violates the open/close principle.
At least 5 tests are required to test each case. The class does 5 things, so it violates the single responsibility principle.
The class is at least 50 lines long without the possibility of reusing anything.

This post will provide an illustration of using the strategy design pattern with table control flow to replace the switch statement.

The problem is how we can define a family of algorithms (credit card validators), encapsulate each one, and make them interchangeable?

The solution is to use the strategy design pattern that allows the algorithm to vary independently from clients that use it.

Download Source Code

Setup

The following classes from the previous post will be reused.

public class CreditCard
{
public string Type { get; set; }
public string Name { get; set; }
public string Number { get; set; }
public string Expiry { get; set; }
}
public interface ICreditCardValidator
{
void Validate(CreditCard creditCard);
}

Strategy Pattern

Let's get started with the strategy pattern in 5 easy steps.

1. Specify the signature for the algorithm in an interface

We need to validate a credit card number so we need a method that takes a number as shown below.

public interface ICreditCardNumberValidator
{
void Validate(string number);
}

2. Bury the algorithm implementation details in derived classes

Each credit card validator implementation can be different. In this example, all credit card validators will use a regular expression so we only need one class.

public class RegExCreditCardNumberValidator : ICreditCardNumberValidator
{
private readonly Regex _regex;
public RegExCreditCardNumberValidator(Regex regex)
{
if (regex == null) throw new ArgumentNullException("regex");
_regex = regex;
}
public void Validate(string number)
{
if (!_regex.IsMatch(number)) throw new InvalidCreditCardException();
}
}

A new instance of the class will be created for each credit card type. For example:

var visaRegEx = new Regex("^4[0-9]{6,}$", RegexOptions.Compiled);
var visaValidator = new RegExCreditCardNumberValidator(visaRegEx);

3. Identify an algorithm that the client would prefer to access

The credit card validation algorithm is based on the credit card type. The credit card type is supplied in the credit card model/DTO. Therefore, we can perform an algorithm lookup based on the credit card type.

A dictionary is great way to perform a lookup, where the key represents the credit card type and the value represents the validator.

var strategies = new Dictionary<string, ICreditCardNumberValidator>
(StringComparer.OrdinalIgnoreCase);

Solved! The dictionary key lookup is set to case insensitive. The problem of comparing strings or calling .ToLower() in the Switch solution disappears.

The strategy validator is shown below.

public class CreditCardValidatorStrategy : ICreditCardValidator
{
private readonly IDictionary<string, ICreditCardNumberValidator> _strategies;
public CreditCardValidatorStrategy(
IDictionary<string, ICreditCardNumberValidator> strategies)
{
if (strategies == null) throw new ArgumentNullException("strategies");
_strategies = strategies;
}
public void Validate(CreditCard creditCard)
{
if (creditCard == null) throw new ArgumentNullException("creditCard");
if (string.IsNullOrWhiteSpace(creditCard.Type))
throw new ArgumentException(Resource.MissingCreditCardType);
if (!_strategies.ContainsKey(creditCard.Type))
throw new InvalidCardException(
string.Format(Resource.UnsupportedCreditCard, creditCard.Type));
_strategies[creditCard.Type].Validate(creditCard.Number);
}
}

Solved! The validator class is agnostic of the credit card validation implementation and doesn't need to be modified in order to support additional credit cards.

4. Building the table control lookup

We can build the dictionary lookup table based on a repository such as a database or a configuration file.

The credit card types and regular expressions are defined in the App.config as shown below.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<configSections>
<section name="creditcardnumberexpressions"
type="System.Configuration.DictionarySectionHandler" />
</configSections>
<creditcardnumberexpressions>
<add key="visa" value="^4[0-9]{6,}$" />
<add key="mastercard" value="^5[1-5][0-9]{5,}$" />
</creditcardnumberexpressions>
</configuration>

We can compose the dictionary of strategies from the App.config as shown below.

public class ConfigDictionaryLoader : ISettingsLoader
{
private readonly string _sectionName;
public ConfigDictionaryLoader(string sectionName)
{
if (sectionName == null) throw new ArgumentNullException("sectionName");
_sectionName = sectionName;
}
public IDictionary<string, string> Load()
{
var settings = ConfigurationManager.GetSection(_sectionName) as Hashtable;
if (settings == null) {
throw new Exception(string.Format(Resource.MissingConfig, _sectionName));
}
return settings.Cast<DictionaryEntry>()
.ToDictionary(n => n.Key.ToString(), n => n.Value.ToString());
}
}
public class CreditCardValidatorFactory : ICreditCardValidatorFactory
{
private readonly ISettingsLoader _loader;
public CreditCardValidatorFactory(ISettingsLoader loader)
{
if (loader == null) throw new ArgumentNullException("loader");
_loader = loader;
}
public IDictionary<string, ICreditCardNumberValidator> CreateStrategies()
{
var cardPairs = _loader.Load();
var strategies = new Dictionary<string, ICreditCardNumberValidator>
(StringComparer.OrdinalIgnoreCase);
foreach (var pair in cardPairs)
{
var regEx = new Regex(pair.Value, RegexOptions.Compiled);
strategies[pair.Key] = new RegExCreditCardNumberValidator(regEx);
}
return strategies;
}
}

Solved! The solution in the previous post required a new build to add, remove or modify an algorithm. This problem disappears since values are loaded at run time.

5. Running the solution

The entire solution is wired up as shown below.

var configLoader = new ConfigDictionaryLoader("creditcardnumberexpressions");
var creditCardFactory = new CreditCardValidatorFactory(configLoader);
var strategies = creditCardFactory.CreateStrategies();
var validator = new CreditCardValidatorStrategy(strategies);
var creditCard = new CreditCard()
{
Type = "ViSa",
Number = "4111111111111111"
};
validator.Validate(creditCard);

Summary

The strategy pattern provides a great alternative to the switch statement.

The advantages of the strategy design pattern are:

Reduces long lists of conditions such as If and Switch statements
Avoids duplicate code
Changes in one class does not require changes in other classes
The Open-close principle is achieved since the class is open for extension but closed for modification
The Single responsibility principle is achieved since the complexity is encapsulated per strategy
Unit testing is simplified since each strategy can be tested in isolation

Message Queue Delivery Strategies

Mon, 21 Dec 2015 23:33:00 +1300

The previous post focused on MSMQ fundamentals using a pull approach to retrieve messages from a queue.

This post is the start of a series that covers multiple strategies to push queued messages to clients. The intention of the push approach is to keep clients agnostic of being part of a message based architecture.

MSMQ technology is used but it is easy enough to change the implementation to use an alternative queuing solution such as Azure Service Bus.

Download Source Code

Setup

Setting up an MSQM was covered in the MSMQ fundamentals post.

The following code was used for placing 3 unique OrderModel messages in the queue.

[Serializable]
public class OrderModel
{
public int Id { get; set; }
public string Name { get; set; }
}
using (var queue = new MessageQueue(@".\Private$\Orders"))
{
queue.Formatter = new BinaryMessageFormatter();
for (var orderId = 1; orderId <= 3; orderId++)
{
var order = new OrderModel()
{
Id = orderId,
Name = string.Format("Order {0}", orderId)
};
queue.Send(order);
}
}

Command Pattern

The problem is how can a message infrastructure issue requests to objects without knowing anything about the operation being requested or the receiver of the request?

The solution is to use the command pattern to decouple senders from receivers. A command decouples the object that invokes the operation from the object that knows how to perform the operation.

The generic command interface is displayed below. The OrderProcessorCommand class implements the command interface to indicate that it can accept OrderModel messages, which it will use to simulate an order being processed.

public interface ICommand<in T>
{
void Execute(T message);
}
public class OrderProcessorCommand : ICommand<OrderModel>
{
private readonly int _id;
public OrderProcessorCommand(int id)
{
_id = id;
}
public void Execute(OrderModel message)
{
if (message == null) throw new ArgumentNullException("message");
var start = DateTime.Now;
// Simulate work being performed
Thread.Sleep(TimeSpan.FromSeconds(2));
Console.WriteLine("Processed {0} by worker {1} from {2} to {3}",
message.Name, _id, start.ToString("h:mm:ss"),
DateTime.Now.ToString("h:mm:ss"));
}
}

Note that a sleep was added on line 21 to simulate work being performed by the processor.

Queue Monitor

The queue monitor class acts as orchestrator, which is responsible for listening to the queue for incoming messages and calling a command to execute each message.

When the client calls the start method then the workflow outlined below will run consciously until the client calls the stop method:

The BeginReceive method will kick off the queue listing operation.
The OnReceiveComplete event will be raised when a message arrives.
The command will be executed by passing in the message content.

public interface IQueueMonitor : IDisposable
{
void Start();
void Stop();
}
public class QueueMonitor<T> : IQueueMonitor
{
private readonly MessageQueue _queue;
private readonly ICommand<T> _command;
private bool _continue = true;
public QueueMonitor(MessageQueue queue, ICommand<T> command)
{
if (queue == null) throw new ArgumentNullException("queue");
if (command == null) throw new ArgumentNullException("command");
_queue = queue;
_command = command;
_queue.ReceiveCompleted += OnReceiveCompleted;
}
private void OnReceiveCompleted(object sender,
ReceiveCompletedEventArgs receiveCompletedEventArgs)
{
var message = _queue.EndReceive(receiveCompletedEventArgs.AsyncResult);
_command.Execute((T)message.Body);
if (_continue) _queue.BeginReceive();
}
public void Start()
{
_continue = true;
_queue.BeginReceive();
}
public void Stop()
{
_continue = false;
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool isDisposing)
{
if (!isDisposing || _queue == null) return;
_queue.ReceiveCompleted -= OnReceiveCompleted;
_queue.Dispose();
}
}

Note: The push method will not overwhelm the client with requests. The queue monitor will be blocked on line 27 and will not retrieve the next message in the queue until the command has finished processing the current request.

Single Receiver

Let's see the gears ticking over by processing the messages on the queue.

using (var queue = new MessageQueue(@".\Private$\Orders"))
{
var command = new OrderProcessorCommand(1);
var queueMonitor = new QueueMonitor<OrderModel>(queue, command);
queueMonitor.Start();
}

Processed Order 1 by worker 1 from 6:20:11 to 6:20:13
Processed Order 2 by worker 1 from 6:20:13 to 6:20:15
Processed Order 3 by worker 1 from 6:20:15 to 6:20:17

The output above shows that the single receiver processed the messages in order, one at a time.

The drawback of the single receiver is the finite amount of throughput due to the constraint of processing one message at time.

Summary

This post demonstrated a generic approach to continually monitor a queue for new messages and pushing the message content to a command to execute.

The next post will describe how to broadcast a single message across multiple processors.