celery is a python library to manage some concurrent workloads.

Celery tasks are called from regular python code. They go through a queue and get executed in a separate worker process. The results are sent back to a database available to the caller. Its simplest form looks like this:

result = task.delay(arg1, arg2, kwarg1='x', kwarg2='y')

with celery managing everything if configured properly. In celery it is possible to specify more complicated structures using workflow primitives.

groups execute tasks in parallel. chords call a final one when all done

chains execute tasks using the output of an entry as the input of the next one, returning the final output

map apply the same function to a set of inputs, and return the outputs of each of them

The limitation

Celery is one of the many options for concurrent execution in python. It is a bit more complicated that using threads or subprocess, but resistant to the main process restarting. Unlike asyncio it allows more than 1 cpu at a time. It is less complicated that using a database or stream processing where guarantees of execution are stronger.

But there is a weakness that after certain scale becomes problematic: There is no permanent task storage over time.

  • When a celery task runs, the task gets inserted in the queue.
  • When a worker is ready to run the task (or wants to prefetch it), the task gets removed from the queue.

481aacff61ff0165a19c061b0e1dd9d3

So for example, if the worker gets killed but the task isn’t finished, what happens? The worker is responsible to put the task back in the queue. In most circumstances it will be handled fine, but under pressure, the task can receive a SIGKILL or produce a exotic signal like SIGSEGV. Celery won’t handle this so the task will be lost forever.

It is possible to use acks_late as a property of the task. This informs celery not to acknowledge the task until it is fully done. The problem then is about running the task multiple times in case of failure. This is the exactly once problem that kafka in particular aims to solve, although any database with transactional guarantees offers the same.

There is a complication to this limitation: Remember the workflows I mentioned before? They live as tasks in the same system. So they are either in the queue or in the workers’ memory. Any complicated workflow will inevitably disappear and executed partially or get executed multiple times in these situations.

Alternatives

Some people suggest to use temporal.io . I think the lock in is too unpalatable so rolling out something with redis, postgres or kafka that has better guarantees could do.

This depends on the complexity of the workflows though. For very complicated workflows something like https://argoproj.github.io/workflows/ could work best.