What is a Task Queue


A Task Queue is almost self explanatory. At it's simplest it is a list of tasks that need to be processed in some sort of order.

The queue itself does not do any of the work and relies on the external workers to process the tasks.

At a minimum each task should be in one of the following states:

  • Waiting to be Processed
  • In Progress
  • Completed
  • Err/Failed
simple-fifo-task-queue

Using a Simple Message Queue


Message Queues are often used as the underlying technology upon which Task Queues are implemented.

It's easy to see why this is so common as on it's surface a task queue is seemingly similar to a message queue. The tempation would be to store serialised task data in the message body and map the message state to task state.

  • Waiting to be Processed -> Message Queued
  • In Progress -> Delivered
  • Competed -> Acknowledged
  • Err/Failed - > Negative Acknowlegedment
simple-fifo-message-queue

Where the similarities start to break down


Message queues are typically designed to be fast first in first out (FIFO) systems. Importantly messages cannot be modified and the queues do not support insertion.

To resend any failed message (or task) the message has to be copied and reinserted onto the end of the queue and wait for preceding tasks to be cleared.

However, it's not uncommon to want a task queue to try and process its earliest queued tasks first. Basically First Queued First Processed (FQFP). Tasks can be updated (at least some of its metadata can) and it may even by possible to insert new tasks ahead of others.


Message Queue FIFO != FQFP Task Queue

retry-with-fifo-message-queue