A Rust implementation of Celery for producing and consuming asyncronous tasks with a distributed message queue.
Simply put, this is a Rust implementation of the Celery protocol for producing and consuming asyncronous tasks with a distributed message broker. It comes with an idiomatic async API driven by the performant tokio.rs, and above all an emphasis on safety.
Celery revolves around the concept of a task. A task is a unit of work that is requested by a producer to be completed by a consumer / worker.
For example, a social media service may need tasks to notify a user's followers when they post new content. When a user uploads their content to the service's website, the website's backend would act as the producer sending out the tasks to a set of workers - usually deployed on a separate server or cluster - via a distributed message broker.
Celery application instance is meant to serve as either the producer or the consumer. In this example, both the website backend and the worker applications would initialize a
Celery app in the same way, with the exact same configuration. The web backend would then call
Celery::send_task to produce a task which a worker would receive while it is consuming tasks through the
The Celery framework is a multiple producer, multiple consumer setup: any number of producer applications can send tasks to any number of workers. Naturally this allows seamless horizontal scaling.
Broker is an integral part in all of this, providing the channel through which producers communicate to consumers and distributing tasks among the available workers. As of writing this, the only officially supported broker is the
AMQPBroker which can be used with a RabbitMQ instance. The RabbitMQ instance would be the actual broker, while the
AMQPBroker struct provides the API that the
Celery app uses to communicate with it.
Rusty Celery is developed on GitHub as an open source community effort.