Introduction to Celery

What is Celery?

Celery is an open-source, simple, flexible, distributed system to process vast amounts of messages while providing operations with the tools required to maintain such a system. It’s a task queue with a focus on real-time processing, while also supporting task scheduling. Celery is written in Python, but it can be implemented in any language as the protocol is the same for all. It is a cross-platform software which means you can set up Celery in any Operation System.

What's a Task Queue?

Task queues are used as a mechanism to distribute work across threads or machines.

A task queue's input is a unit of work, called a task, dedicated worker processes then constantly monitor the queue for new work to perform.

Celery communicates via messages, usually using a broker to mediate between clients and workers. To initiate a task a client puts a message on the queue, the broker then delivers the message to a worker.

A Celery system can consist of multiple workers and brokers, giving way to high availability and horizontal scaling.

Celery is written in Python, but the protocol can be implemented in any language. In addition to Python there's node-celery_ for Node.js, a PHP client, gocelery for golang, and rusty-celery for Rust.

Language interoperability can also be achieved by using webhooks in such a way that the client enqueues an URL to be requested by a worker.

Why is Celery useful?

Task queues and the Celery implementation in particular are one of the trickier parts of a Python web application stack to understand.

If you are a junior developer it can be unclear why moving work outside the HTTP request-response cycle is important. In short, you want your WSGI server to respond to incoming requests as quickly as possible because each request ties up a worker process until the response is finished. Moving work off those workers by spinning up asynchronous jobs as tasks in a queue is a straightforward way to improve WSGI server response times.

Why use Celery?

One of the most difficult parts of the Python web application stack is understanding and implementing asynchronous task queues or job queues. Task Queues are very important to serve users with better functionalities. If users are not satisfied, all the effort put into the application will go in vain. So, it is better to use a task queue like Celery to avoid such scenarios in real life. As a developer, we want our application to be fast and responsive to all the requests that it gets. As we know that each request ties up to a worker process until the response is finally finished. In a normal application, we would have overloaded our process worker which eventually slows down our applications. But, we don't want that do we? of course not, so this is the main reason why we use Celery. Celery makes it easy to put tasks in a queue that can be executed later. By doing this we are improving our WSGI server response time and also perform some other important tasks while this executes.

Merits of using Celery?

The following are some of the merits of using Celery.

Open-Source: As explained above, Celery is Open-Source software written in Python. This enables small businesses to use such technology and even modify some changes in the official source code.

Broker Support: Celery supports multiple message brokers like RabbitMQ and Redis. Which are also the recommended ones. Also, we have some additional brokers like MongoDB, Amazon SQS, CouchDB, IronMQ, Django ORM are supported too.

Easy to Install: Celery is very easy to install and there are no workarounds needed. It is pretty straightforward though.

Web Framework Integrations: It also integrates itself with multiple web frameworks, including Django, Flask, Pyramid, Pylons, web2py, Tornado, and Tryton.

Celery is...

Simple: Celery is easy to use and maintain, and does not need configuration files.

Here's one of the simplest applications you can make:

    from celery import Celery

    app = Celery('hello', broker='amqp://guest@localhost//')

    @app.task
    def hello():
        return 'hello world'

Highly Available: Workers and clients will automatically retry in the event of connection loss or failure, and some brokers support HA in way of Primary/Primary or Primary/Replica replication.

Fast: A single Celery process can process millions of tasks a minute, with sub-millisecond round-trip latency (using RabbitMQ,py-librabbitmq, and optimized settings).

Flexible: Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, broker transports, and much more.

It supports...

• Message Transports

  • RabbitMQ, Redis, Amazon SQS

• Concurrency

  • Prefork, Eventlet, gevent, single threaded (solo)

• Result Stores

  • AMQP, Redis
  • memcached
  • SQLAlchemy, Django ORM
  • Apache Cassandra, IronCache, Elasticsearch

• Serialization

  • pickle, json, yaml, msgpack.
  • zlib, bzip2 compression.
  • Cryptographic message signing.

Framework Integration

Celery is easy to integrate with web frameworks, some of which even have integration packages:

Django	not needed
Pyramid	pyramid_celery
Pylons	celery-pylons
Flask	not needed
web2py	web2py-celery
Tornado	tornado-celery

Installation

Before installing Celery, install the following using the commands shown:

pip install redis pip install sqlalchemy

All set, Let’s get started with the installation of Celery. As explained above installing Celery is the easiest thing. Just type this command in your command line (or) terminal.

pip install celery

That’s it our installation is complete.

Bundles

Celery also defines a group of bundles that can be used to install Celery and the dependencies for a given feature.

You can specify these in your requirements or on the pip command-line by using brackets. Multiple bundles can be specified by separating them by commas.

$ pip install "celery[librabbitmq]"

$ pip install "celery[librabbitmq,redis,auth,msgpack]"

The following bundles are available:

Serializers

celery[auth]	for using the auth security serializer.
celery[msgpack]	for using the msgpack serializer.
celery[yaml]	for using the yaml serializer

Concurrency

celery[eventlet]	for using the eventlet pool.
celery[gevent]	for using the gevent pool.

Transports and Backends

celery[librabbitmq]	for using the librabbitmq C library.
celery[redis]	for using Redis as a message transport or as a result backend.
celery[sqs]	for using Amazon SQS as a message transport.
celery[tblib]	for using the task_remote_tracebacks feature.
celery[memcache]	for using Memcached as a result backend (using pylibmc)
celery[pymemcache]	for using Memcached as a result backend (pure-Python implementation).
celery[cassandra]	for using Apache Cassandra as a result backend with DataStax driver.
celery[azureblockblob]	for using Azure Storage as a result backend (using azure-storage)
celery[s3]	for using S3 Storage as a result backend.
celery[couchbase]	for using Couchbase as a result backend.
celery[arangodb]	for using ArangoDB as a result backend.
celery[elasticsearch]	for using Elasticsearch as a result backend.
celery[riak]	for using Riak as a result backend.
celery[cosmosdbsql]	for using Azure Cosmos DB as a result backend (using pydocumentdb)
celery[zookeeper]	for using Zookeeper as a message transport.
celery[sqlalchemy]	for using SQLAlchemy as a result backend (supported).
celery[pyro]	for using the Pyro4 message transport (experimental).
celery[slmq]	for using the SoftLayer Message Queue transport (experimental).
celery[consul]	for using the Consul.io Key/Value store as a message transport or result backend (experimental).
celery[django]	specifies the lowest version possible for Django support.

Downloading and installing from source

Download the latest version of Celery from PyPI:

https://pypi.org/project/celery/

You can install it by doing the following,:

$ tar xvfz celery-0.0.0.tar.gz
$ cd celery-0.0.0
$ python setup.py build
# python setup.py install

The last command must be executed as a privileged user if you aren't currently using a virtualenv.

Implementation of Celery

Open up your code editor and create a new python file called “tasks.py”. Now, import celery and create an app using celery and pass the broker URL and also specify the database backend. In this case, the broker URL is from Redis and the database backend is from SQLite3.

from celery import Celery

app = Celery(
  name= 'tasks',
  broker='redis://localhost:6379',
  backend = 'db+sqlite:///db.sqlite3')

@app.task
def absoluteSub(a, b):
    return abs(a-b)