Handling RabbitMQ publisher failures

In a previous post I wrote about Splitting your app into smaller apps using RabbitMQ, talked about the basics, gave an example of code to create basic publisher and consumer with bunny and sneakers and how to connect all pieces together.

Today I want to talk about handling publisher failures for dashboard app, an example, when we experience a connection problem because broker for any reason is down, you need to handle this case because the bunny is synchronous and your app may get stuck. To solve this issue, I added background job: Resque in my case we are using Redis/Resque for background jobs, with a great plugins which called Resque retry and provides retry, delay and exponential backoff support for resque jobs and Resque scheduler for support of queueing items in the future.

Now let’s write some code - adding a wrapper called message_sender which runs publisher and if there some problem exist and we got an exception, job executes with a delay of 1 minute (configure this value for your needs).

Message Sender

  class MessageSender
    def self.deliver(message)
      begin
        Publishers::MessagePublisher.publish(message) # it's publisher's code from previous post
      rescue => e
        Resque.enqueue_in(1.minutes, Jobs::DeliverMessageJob, message.id)
      end
    end
  end

Deliver Message Job

  require 'resque-retry'

  class DeliverMessageJob
    extend Resque::Plugins::Retry
    @queue = :messages_delivery

    @retry_limit = 3
    @retry_delay = 60

    def self.perform(message_id)
      message = Message.find_by_id(message_id)

      if message
        MessageSender.deliver(message)
      end
    end
  end

In DeliverMessageJob we used ‘resque-retry’ plugin to retry job 3 times with a delay of 1 minute in case of failure (configure this value for your needs), during this time we might fix this problem.

In coming post I will talk about ‘Job Handling Strategies’ chosen for consumer.