Background workers in Rails: how to keep one failing job from taking the whole application down

One job, six hours, Black Friday

Let me tell you about the time a single background job took down an entire eCommerce platform for six hours on Black Friday. One job. Six hours. Millions in lost sales. The job was processing order confirmations, hit a malformed email address and raised an exception nobody handled. It was set up to retry indefinitely with no circuit breaker, and it blocked the entire queue. Every other job (shipping notifications, inventory updates, payment confirmations) just sat there waiting. This is not a hypothetical. It happened, and it is exactly why I am writing this.

Background workers handle everything that would otherwise leave your customers staring at a loading spinner: sending emails, processing images, syncing data, generating reports, charging cards. They are also one of the biggest sources of production incidents when they are not designed properly. I have been running Sidekiq in production for over a decade across dozens of applications, from small startups to systems processing millions of jobs a day. The patterns here come from real outages.

This article explains what a background job is, how Sidekiq and the new Rails 8 default, Solid Queue, run them, and the handful of rules that stop one bad job from taking everything else down with it.

Why background workers exist

When someone presses a button on your website, they expect something to happen straight away. Jakob Nielsen's long standing rule of thumb is that 0.1 seconds is about the limit for a system to feel instantaneous, and 1 second is about the limit for the user's flow of thought to stay uninterrupted.¹ I cover the psychology of waiting in more depth in conversion and UX psychology.

Some work simply takes longer than that. Sending an email means talking to a mail server. Resizing an uploaded image takes a second or two. A PDF report can take ten. A call to someone else's API takes as long as their servers feel like taking.

If you do that work synchronously (the user waits while it completes), the application feels sluggish. Worse, inside a web request you are tying up a server process that could be serving someone else. Under load that turns into request queuing, timeouts and eventually the whole application grinding to a halt.

A background worker moves the slow work out of the request. Instead of doing it now, the web process writes a small note into a queue ("send the confirmation for order 4812") and answers the user at once. A separate process picks up the note and does the work.

Worked example: placing an order at Eglinton Coffee

Eglinton Coffee is an invented coffee roaster in Galway that sells beans online at eglintoncoffee.example. A customer presses "Place order". In the request, the shop saves the order and puts four notes on the queue: charge the card, reduce the stock, tell the warehouse, send the confirmation. The customer sees "Order confirmed" in a few hundred milliseconds. The four pieces of work happen over the next few seconds in the worker process, and the customer never waits for any of them.

The diagram reads top to bottom. The web request ends at the third box, which is why the customer is not kept waiting. Everything below it happens in the worker, and a failure there sends that one job round again without touching the others.

The stack: Sidekiq, Redis and Valkey

In the Ruby world Sidekiq has been the most widely used job processor for well over a decade. It stores its queues in Redis, an in memory data store that is very fast at exactly what a queue needs: pushing a job on, popping a job off, and keeping track of state. Current versions of Sidekiq need Redis 7.0 or later, and support Valkey 7.2 or later and Dragonfly as alternatives. Its own changelog shows the current major version, 8.1.²³

A quick note on Valkey

In March 2024 Redis moved from the permissive BSD licence to a dual licence, the Redis Source Available License (RSALv2) and the Server Side Public License (SSPLv1), starting with Redis 7.4.⁴ Within days, the Linux Foundation launched Valkey, a community fork of Redis 7.2.4 that stays under the three clause BSD licence.⁵ In May 2025 Redis added the GNU Affero General Public License (AGPLv3) as a third option, from Redis 8.⁶

For Sidekiq, Valkey behaves as a drop in replacement: same protocol, same commands, same configuration. If you are starting a new project, or your organisation is wary of the Redis licences, Valkey is a sound choice.

# Gemfile
gem "sidekiq", "~> 8.1"
# Sidekiq talks to Redis or Valkey through the redis-client gem,
# which it installs itself. No separate redis gem is needed.

# config/initializers/sidekiq.rb
# The same settings work for Redis and for Valkey.
Sidekiq.configure_server do |config|
  config.redis = { url: ENV.fetch("REDIS_URL", "redis://localhost:6379/0") }
end

Sidekiq.configure_client do |config|
  config.redis = { url: ENV.fetch("REDIS_URL", "redis://localhost:6379/0") }
end

Rails 8 ships its own queue: Solid Queue

Since Rails 8.0, a new application uses Solid Queue as its default Active Job backend. Solid Queue keeps the jobs in your database rather than in Redis, and for high volume it relies on FOR UPDATE SKIP LOCKED, available in PostgreSQL from 9.5 and in MySQL from 8.0. It also works with SQLite.⁷ It supports delayed jobs, priorities, concurrency controls, recurring jobs and bulk enqueuing, and in a new app it is configured for production out of the box, on a separate queue database.⁸

Active Job is the common interface in front of both. You write a job once, as a class that inherits from ApplicationJob, and Active Job hands it to whichever backend you configure: Solid Queue by default, or Sidekiq, or another adapter.

So which one? Solid Queue means one less piece of infrastructure, and for a shop or a SaaS with modest job volumes that is a real saving. Sidekiq is the more mature choice for very high throughput and comes with a large ecosystem. Everything in the rest of this article, independence, idempotency, isolation and monitoring, applies to both. Where the code differs I show both versions.

Your first background job

A simple job that sends a welcome email, in Sidekiq's own interface:

# app/sidekiq/send_welcome_email_job.rb
class SendWelcomeEmailJob
  include Sidekiq::Job

  def perform(user_id)
    user = User.find(user_id)
    UserMailer.welcome(user).deliver_now
  end
end

And the enqueue from the controller:

# app/controllers/registrations_controller.rb
class RegistrationsController < ApplicationController
  def create
    @user = User.new(user_params)

    if @user.save
      # Writes the job to Redis and returns at once.
      SendWelcomeEmailJob.perform_async(@user.id)
      redirect_to dashboard_path, notice: "Welcome aboard."
    else
      render :new, status: :unprocessable_entity
    end
  end
end

The key is perform_async. It does not send the email. It writes the class name and the arguments to Redis as JSON and returns in milliseconds. A Sidekiq process running separately picks the job up and calls perform. If the email takes half a second to send, the user is already looking at the dashboard.

The problem: jobs are not islands

That simple job has flaws that can hurt the whole application.

Flaw one: the user may no longer exist

Between enqueueing and running, the user might have been deleted, or (with Redis) the transaction that created them may not have committed yet. User.find raises ActiveRecord::RecordNotFound, and Sidekiq retries. By default it retries 25 times over roughly three weeks before moving the job to the dead set.⁹ That is 25 exceptions in your error tracker for a job that can never succeed.

Flaw two: the email provider may be down

Mail servers go down, connections time out, rate limits get hit. Every one of those raises. If your provider has an outage, every email job fails at once, and they all land in the retry set together. Sidekiq adds a random amount to each retry delay, which spreads them out a little, but the provider still comes back to a wall of work.

Flaw three: one job doing four things

The nastiest problems come when one job does several things that can each fail:

# app/sidekiq/process_order_job.rb
# One job, four ways to fail. Can you spot the problem?
class ProcessOrderJob
  include Sidekiq::Job

  def perform(order_id)
    order = Order.find(order_id)
    return if order.processed?

    charge_payment(order)      # calls the payment provider
    update_inventory(order)    # database updates, can deadlock
    notify_warehouse(order)    # calls an outside API, can time out
    send_confirmation(order)   # sends email, can fail

    order.update!(processed: true)
  end
end

If notify_warehouse times out, the whole job fails and retries. But the card has already been charged, so the retry charges it again. And if update_inventory deadlocks with another order job updating the same product, both fail, both retry, and both can deadlock again.

Why one job can block a queue

Sidekiq's own retries do not hold up a queue: a failed job is moved to a separate retry set and the thread moves on. What blocks a queue is work that occupies the threads. A job that hangs on a call with no timeout, or a job that loops on its own retries inside perform, keeps a thread busy. When enough threads are busy with the same broken dependency, nothing else runs.

How many threads that takes follows from Little's law, a result from queuing theory that holds for any stable system:¹⁰

$L = \lambda W$

Here $L$ is the average number of jobs being worked on at once (so the number of threads they keep busy), $\lambda$ is the arrival rate of those jobs in jobs per second, and $W$ is the average time in seconds that each job holds a thread. The rate is the same idea as any other rate, events over time, explained in Foundations: ratios and rates.

Worked example: Eglinton Coffee's warehouse goes slow

The example's assumptions: on Black Friday Eglinton Coffee takes 1,800 orders an hour, each order enqueues four jobs, and one Sidekiq process runs 10 threads. A normal job takes 0.4 seconds.

First the normal day. 1,800 orders an hour times four jobs is 7,200 jobs an hour. Divided by 3,600 seconds, that is $\lambda = 2$ jobs per second. With $W = 0.4$ seconds, $L = 2 \times 0.4 = 0.8$. On average fewer than one of the 10 threads is busy. Plenty of headroom.

Now the warehouse API slows down and every call waits for a 30 second timeout. Only the warehouse jobs are affected: one per order, so 1,800 an hour, which is $\lambda = 0.5$ per second. With $W = 30$, $L = 0.5 \times 30 = 15$. The warehouse jobs alone want 15 threads, and there are only 10. Within a minute every thread is waiting on the warehouse, and the payment, stock and email jobs queue up behind them. Nothing has failed yet, and the shop has already stopped. Cut the timeout to five seconds and the same arithmetic gives $0.5 \times 5 = 2.5$ threads. Put the warehouse jobs in their own process and even a total warehouse outage cannot touch the payments. Both fixes follow.

The golden rule: every job is independent and idempotent

This is the single most important lesson in the article:

Every job must be independent. Every job must be idempotent. Every job must assume it will fail and be retried.

The Sidekiq documentation says the same thing in its best practices: keep job arguments small and simple, and make jobs idempotent and transactional.¹¹

Independence

A job should not depend on state left behind by another job, or on jobs running in a particular order. It fetches what it needs when it runs.

# BAD: assumes another job already set shipped_at
class SendShippingNotificationJob
  include Sidekiq::Job

  def perform(order_id)
    order = Order.find(order_id)
    raise "Not shipped yet" unless order.shipped_at
    OrderMailer.shipped(order).deliver_now
  end
end

# GOOD: checks the state and handles every case
class SendShippingNotificationJob
  include Sidekiq::Job

  def perform(order_id)
    order = Order.find_by(id: order_id)
    return unless order                               # order gone, nothing to do
    return unless order.shipped_at                    # not shipped, the shipping job enqueues us later
    return if order.shipping_notification_sent_at     # already sent, do not spam

    OrderMailer.shipped(order).deliver_now
    order.update!(shipping_notification_sent_at: Time.current)
  end
end

Idempotency

A job is idempotent if running it twice has the same effect as running it once. That matters because retries, network errors and worker crashes all mean a job can run more than once.

For payments the provider helps. Stripe accepts an idempotency key with each request: if a request is repeated with the same key, Stripe returns the saved result of the first request instead of performing the operation again. Keys can be up to 255 characters, should not contain personal data such as an email address, and can be pruned after 24 hours.¹² In the Ruby library the key goes in the request options, the second argument, not among the payment parameters.

# app/sidekiq/charge_order_job.rb
class ChargeOrderJob
  include Sidekiq::Job
  sidekiq_options queue: "critical", retry: 10

  def perform(order_id)
    order = Order.find_by(id: order_id)
    return unless order
    return if order.paid?

    intent = Stripe::PaymentIntent.create(
      {
        amount: order.total_cent,
        currency: "eur",
        customer: order.user.stripe_customer_id,
        payment_method: order.payment_method_id,
        off_session: true,
        confirm: true
      },
      { idempotency_key: "order-#{order.id}-charge" }
    )

    order.update!(paid: true, stripe_payment_intent_id: intent.id, paid_at: Time.current)
  rescue Stripe::CardError => e
    # A decline will not get better with a retry. Record it and tell the customer.
    order.update!(payment_failed: true, payment_error: e.message)
    PaymentFailedMailer.notify(order).deliver_later
  end
end

If the worker crashes after Stripe has taken the money but before update! runs, the retry sends the same key, gets the original result back, and the customer is charged once.¹³

Graceful failure

Some errors will never go away, and retrying them only fills the error tracker. Others are temporary and deserve a retry. A job should tell the two apart. In this example MailingList is the shop's own small wrapper around its newsletter provider, with its own error classes, which is the example's assumption.

class SyncUserToMailingListJob
  include Sidekiq::Job
  sidekiq_options retry: 5

  def perform(user_id)
    user = User.find_by(id: user_id)
    return unless user&.email.present?

    MailingList.subscribe(email: user.email, first_name: user.first_name)
  rescue MailingList::AlreadySubscribed
    Rails.logger.info("User #{user_id} already on the list")
  rescue MailingList::InvalidEmail => e
    # Permanent: this address will never work. Record it, do not retry.
    user.update!(mailing_list_error: e.message)
  rescue MailingList::RateLimited, MailingList::ServerError
    raise # temporary: let Sidekiq retry with backoff
  end
end

Breaking up a job that does too much

Back to the order job that did four things. Split it into four jobs that can each fail on their own:

# app/sidekiq/process_order_job.rb
# Only coordinates. Each step is its own job.
class ProcessOrderJob
  include Sidekiq::Job

  def perform(order_id)
    order = Order.find_by(id: order_id)
    return unless order
    return if order.processing_started_at

    order.update!(processing_started_at: Time.current)

    ChargeOrderJob.perform_async(order_id)
    UpdateInventoryJob.perform_async(order_id)
    NotifyWarehouseJob.perform_async(order_id)
    SendOrderConfirmationJob.perform_async(order_id)
  end
end

# app/sidekiq/update_inventory_job.rb
class UpdateInventoryJob
  include Sidekiq::Job
  sidekiq_options queue: "default", retry: 5

  def perform(order_id)
    order = Order.find_by(id: order_id)
    return unless order
    return if order.inventory_updated_at

    ActiveRecord::Base.transaction do
      # Lock products in id order, so two orders never wait on each other in a circle.
      order.line_items.includes(:product).sort_by(&:product_id).each do |item|
        item.product.lock!
        item.product.decrement!(:stock, item.quantity)
      end
      order.update!(inventory_updated_at: Time.current)
    end
  end
end

# app/sidekiq/notify_warehouse_job.rb
class NotifyWarehouseJob
  include Sidekiq::Job
  sidekiq_options queue: "external_apis", retry: 15

  def perform(order_id)
    order = Order.find_by(id: order_id)
    return unless order
    return if order.warehouse_notified_at

    shipment = WarehouseApi.new(timeout: 5).create_shipment(
      order_id: order.id,
      items: order.line_items.map(&:to_warehouse_format),
      address: order.shipping_address
    )
    order.update!(warehouse_notified_at: Time.current, warehouse_shipment_id: shipment.id)
  rescue WarehouseApi::InvalidAddress => e
    # Permanent: a person needs to look at this address.
    order.update!(warehouse_error: e.message)
    AdminAlertJob.perform_async("warehouse_address_error", order_id)
  end
end

The confirmation email job follows the same shape: find the order, return if it is gone or confirmation_sent_at is set, send, record the time.

Now, if the warehouse API is down, only NotifyWarehouseJob fails and retries. The payment goes through, the stock is reduced and the customer gets the confirmation. The warehouse catches up when its API recovers. If the email fails, the charge and the shipment are unaffected and the email can be looked at on its own. Failures are isolated. One broken thing no longer breaks everything.

Two details are worth pointing out. The warehouse client has a five second timeout, which is the fix from the Little's law example. And the inventory job locks products in a fixed order (by id), which removes the circular wait that causes a deadlock.

The same job, written for Solid Queue

With Active Job the retry rules move into declarations. A failed job is not retried unless you say so: retry_on retries (by default five attempts, three seconds apart) and discard_on drops the job.¹⁴ Since Rails 7.2 the backoff option is :polynomially_longer, the older :exponentially_longer having been removed.¹⁵ If a record passed to a job has been deleted, Active Job raises ActiveJob::DeserializationError, which is a good candidate to discard.¹⁴

# app/jobs/notify_warehouse_job.rb
class NotifyWarehouseJob < ApplicationJob
  queue_as :external_apis

  retry_on WarehouseApi::Unavailable, wait: :polynomially_longer, attempts: 15
  discard_on ActiveJob::DeserializationError

  def perform(order)
    return if order.warehouse_notified_at

    shipment = WarehouseApi.new(timeout: 5).create_shipment(order_id: order.id)
    order.update!(warehouse_notified_at: Time.current, warehouse_shipment_id: shipment.id)
  end
end

Active Job can take the order itself as an argument, because it serialises records as a GlobalID and looks them up again when the job runs.¹⁴ Sidekiq's own interface cannot, as the lessons further down show.

Queue design: keeping critical jobs moving

Not all jobs are equal. A password reset matters more than a weekly analytics rollup. A payment matters more than a CRM sync. Sidekiq lets you list several queues, each with an optional weight. A queue with weight two is checked twice as often as a queue with weight one. Without weights, queues are processed in strict order: a job in a later queue waits until the earlier queues are empty.¹⁶

# config/sidekiq.yml
:concurrency: 10
:queues:
  - [critical, 6]
  - [default, 3]
  - [mailers, 2]
  - [external_apis, 2]
  - [low, 1]

The weights do not reserve threads for a queue, and the order of the list does not matter once weights are set. They set how often each queue is the first one a free thread looks at.

Worked example: what the weights mean

The weights in that file add up to $6 + 3 + 2 + 2 + 1 = 14$. Each time a thread looks for work, critical comes first with a probability of $6 / 14$, about 43%, and low with a probability of $1 / 14$, about 7%. If the first queue it looks at is empty, it moves on to the others, so low priority work still makes progress. Probabilities as long run shares are explained in Foundations: probability.

What the weights cannot do is protect critical when every thread is stuck inside a slow external_apis job. For that you need separate processes, each with its own threads:

# Process one: payments and other critical work only
bundle exec sidekiq -q critical -c 5

# Process two: everyday work and email
bundle exec sidekiq -q default -q mailers -c 10

# Process three: slow outside APIs and bulk work
bundle exec sidekiq -q external_apis -q low -c 5

Now a slow third party can only ever fill the five threads of process three. Payments keep flowing.

In Solid Queue the equivalent lives in config/queue.yml. The order of queues in a worker's list sets the polling order, and a worker does not take jobs from a later queue until the earlier ones are empty.⁸ Separate workers for separate queues give the same isolation as separate Sidekiq processes. Solid Queue can also cap how many jobs of one kind run at once, with limits_concurrency, for example one export per account.¹⁷

# config/queue.yml
production:
  workers:
    - queues: [critical]
      threads: 5
    - queues: [default, mailers]
      threads: 10
    - queues: [external_apis, low]
      threads: 5

Retries and the dead set

Sidekiq retries a failed job with an exponential backoff. Its documentation gives the delay before each retry as:⁹

$\text{delay}(c) = c^4 + 15 + r \times (c + 1)$

Here $c$ is the retry count, starting at 0 for the first retry, $c^4$ is the count to the fourth power, $15$ is a fixed 15 seconds, and $r$ is a random whole number from 0 to 9 that spreads retries out so they do not all land at once. The result is in seconds. The random part is best thought of through its average, the expected value, explained in Foundations: expected value.

Worked example: where "about three weeks" comes from

Ignore the random part first. The first five delays are $0 + 15 = 15$, $1 + 15 = 16$, $16 + 15 = 31$, $81 + 15 = 96$ and $256 + 15 = 271$ seconds, exactly the sequence the Sidekiq documentation lists.

Now add up all 25 retries, for $c$ from 0 to 24. The fourth powers sum to 1,763,020 seconds. The fixed part is $25 \times 15 = 375$ seconds. The random number averages $4.5$, and $(c + 1)$ runs from 1 to 25, which sums to 325, so the random part averages $4.5 \times 325 = 1{,}462.5$ seconds. In total that is $1{,}763{,}020 + 375 + 1{,}462.5 = 1{,}764{,}857.5$ seconds. Divided by 86,400 seconds in a day, that is about 20.4 days. The Sidekiq wiki rounds it to about 20 days in one place and 21 in another.⁹ Either way, a job that can never succeed spends three weeks being retried unless you tell it otherwise.

You can tell it otherwise, per job:

class SomeJob
  include Sidekiq::Job
  sidekiq_options retry: 10, backtrace: 10

  sidekiq_retry_in do |count, exception|
    case exception
    when RateLimitError then 60 * (count + 1)   # one minute, two minutes, three...
    when PermanentError then :kill              # straight to the dead set
    end                                         # nil: the default backoff
  end
end

After the last retry, the job goes to the dead set. The Web UI lets you retry dead jobs by hand, and Sidekiq keeps them for six months before discarding them.⁹ Monitor the dead set. If it is filling up, something is wrong. A hook on retries exhausted is the place to alert:

# app/sidekiq/concerns/dead_job_alerting.rb
module DeadJobAlerting
  extend ActiveSupport::Concern

  included do
    sidekiq_retries_exhausted do |job, exception|
      Rails.logger.error("#{job['class']} died after #{job['retry_count']} retries: #{exception.message}")
      AlertNotifier.notify("Job died: #{job['class']}: #{exception.message}")
    end
  end
end

class ImportantJob
  include Sidekiq::Job
  include DeadJobAlerting
end

Include Sidekiq::Job before the concern, because sidekiq_retries_exhausted comes from it. AlertNotifier stands for whatever sends your alerts.

Monitoring and alerting

You cannot fix what you cannot see. Sidekiq ships a Web UI. Mount it behind authentication:

# config/routes.rb
require "sidekiq/web"

Rails.application.routes.draw do
  authenticate :user, ->(user) { user.admin? } do
    mount Sidekiq::Web => "/sidekiq"
  end
end

For Solid Queue, Mission Control Jobs is the equivalent: a Rails engine that shows queues and jobs, and lets you retry or discard failed ones.¹⁸

The UI is for looking. For production you want numbers and alerts. These four are the ones I watch:

• Queue latency: how long the oldest job has been waiting. If it climbs, you need more workers or your jobs are too slow. Sidekiq gives it per queue as Sidekiq::Queue.new("critical").latency.
• Dead jobs: should be close to zero. Any rise needs a look.
• Retry set size: a sustained high number means something is failing systematically.
• Queue depth: occasional spikes are fine. Sustained growth means you are not keeping up.

Lessons from production

Lesson one: pass ids, not objects

Sidekiq stores job arguments in Redis as JSON. A Ruby object does not survive that trip, and even if it did, it would be stale by the time the job runs. Sidekiq's advice is to pass simple identifiers and look the record up inside perform.¹¹ Since Sidekiq 7.0, passing arguments that are not native JSON types raises an error rather than a warning.¹⁹

SendEmailJob.perform_async(@user)      # raises in Sidekiq 7 and later
SendEmailJob.perform_async(@user.id)   # right

Active Job is the exception, because of GlobalID, as above.

Lesson two: enqueue in bulk

# Slow: one round trip to Redis per line item
@order.line_items.each { |item| UpdateStockJob.perform_async(item.id) }

# Better: one job that handles the whole order
UpdateInventoryJob.perform_async(@order.id)

# Or, when you really need one job per item, push them in bulk
UpdateStockJob.perform_bulk(@order.line_items.ids.map { |id| [id] })

perform_bulk pushes many jobs in one go, 1,000 to a batch by default.²⁰

Lesson three: watch your memory

Workers are long running processes, and loading too much at once grows them for good.

# BAD: loads every pending order into memory
Order.where(status: "pending").to_a.each { |order| process(order) }

# GOOD: works through them 100 at a time
Order.where(status: "pending").find_each(batch_size: 100) { |order| process(order) }

Lesson four: test your jobs

The welcome email job from the start is not safe to retry, and a test makes that obvious. A corrected version records that the email was sent:

class SendWelcomeEmailJob
  include Sidekiq::Job

  def perform(user_id)
    user = User.find_by(id: user_id)
    return unless user
    return if user.welcome_sent_at

    UserMailer.welcome(user).deliver_now
    user.update!(welcome_sent_at: Time.current)
  end
end

# spec/sidekiq/send_welcome_email_job_spec.rb
require "rails_helper"

RSpec.describe SendWelcomeEmailJob do
  let(:user) { create(:user) }

  it "sends the welcome email" do
    expect { described_class.new.perform(user.id) }
      .to change { ActionMailer::Base.deliveries.count }.by(1)
  end

  it "does nothing for a user who no longer exists" do
    expect { described_class.new.perform(0) }.not_to raise_error
  end

  it "sends only once when it runs twice" do
    described_class.new.perform(user.id)
    expect { described_class.new.perform(user.id) }
      .not_to change { ActionMailer::Base.deliveries.count }
  end
end

Calling perform directly tests the logic without Redis. To test that a controller enqueues the job, require "sidekiq/testing" puts Sidekiq in fake mode, where jobs collect in an array you can inspect instead of going to Redis.²¹ I cover the wider testing approach in TDD and BDD in Ruby on Rails.

Design the jobs before they design your outages

Background workers are not optional in a modern web application, and they are a real source of complexity. The difference between a system that holds up and a house of cards is how the jobs are designed.

• Jobs are independent. They never rely on another job having run first.
• Jobs are idempotent. Running twice has the same effect as running once.
• Jobs fail gracefully. Permanent errors are recorded, temporary ones are retried, and every outside call has a timeout.
• Queues are isolated. Slow outside work runs in its own process, away from payments.
• Everything is monitored. Latency, dead jobs, retries and depth.

The Black Friday outage I opened with was completely preventable. A few guards, sensible retry limits and proper queue isolation would have kept it running. Design your jobs properly from the start, and let someone else learn these lessons the hard way.

Footnotes

1. Jakob Nielsen, Response times: the 3 important limits, Nielsen Norman Group, 1 January 1993, https://www.nngroup.com/articles/response-times-3-important-limits/

2. Sidekiq, README (requirements), GitHub, https://github.com/sidekiq/sidekiq/blob/main/README.md

3. Sidekiq, Changes, GitHub, https://github.com/sidekiq/sidekiq/blob/main/Changes.md

4. Redis, Redis adopts dual source available licensing, 20 March 2024, https://redis.io/blog/redis-adopts-dual-source-available-licensing/

5. The Linux Foundation, Linux Foundation launches open source Valkey community, 28 March 2024, https://www.linuxfoundation.org/press/linux-foundation-launches-open-source-valkey-community

6. Rowan Trollope, Redis is now available under the AGPLv3 open source license, Redis, 1 May 2025, https://redis.io/blog/agplv3/

7. Ruby on Rails Guides, Ruby on Rails 8.0 release notes, Solid Queue, https://guides.rubyonrails.org/8_0_release_notes.html

8. Ruby on Rails Guides, Active Job basics, sections 6 (default backend: Solid Queue) and 6.3 (queue order and priority), https://guides.rubyonrails.org/active_job_basics.html

9. Sidekiq wiki, Error handling (automatic job retry, the dead set), GitHub, https://github.com/sidekiq/sidekiq/wiki/Error-Handling

10. John D. C. Little, Little's law as viewed on its 50th anniversary, Operations Research 59(3), 536 to 549, 2011, https://people.cs.umass.edu/~emery/classes/cmpsci691st/readings/OS/Littles-Law-50-Years-Later.pdf

11. Sidekiq wiki, Best practices, GitHub, https://github.com/sidekiq/sidekiq/wiki/Best-Practices

12. Stripe, API reference, Idempotent requests, https://docs.stripe.com/api/idempotent_requests

13. Stripe, API reference, Create a PaymentIntent, https://docs.stripe.com/api/payment_intents/create

14. Ruby on Rails Guides, Active Job basics, sections 2.3 (GlobalID), 8.3 (retrying or discarding failed jobs) and 8.4 (missing records), https://guides.rubyonrails.org/active_job_basics.html

15. Ruby on Rails Guides, Ruby on Rails 7.2 release notes, Active Job removals, https://guides.rubyonrails.org/7_2_release_notes.html

16. Sidekiq wiki, Advanced options (queues and weights), GitHub, https://github.com/sidekiq/sidekiq/wiki/Advanced-Options

17. Solid Queue, README (concurrency controls), GitHub, https://github.com/rails/solid_queue

18. Mission Control Jobs, README, GitHub, https://github.com/rails/mission_control-jobs

19. Sidekiq, Welcome to Sidekiq 7.0 (upgrade notes, strict arguments), GitHub, https://github.com/sidekiq/sidekiq/blob/main/docs/7.0-Upgrade.md

20. Sidekiq wiki, Bulk queueing, GitHub, https://github.com/sidekiq/sidekiq/wiki/Bulk-Queueing

21. Sidekiq wiki, Testing, GitHub, https://github.com/sidekiq/sidekiq/wiki/Testing