Task Patterns

Asynchronous tasks are the backbone of our backend. We use Celery to manage call asynchronous tasks.

General task usage

For the standard functions we use the asynchronous call with delay() as described in the docs.

create_articles_per_tag.delay(tag.id)

The function has to be registered with @shared__tasks

@shared_task
def create_articles_per_tag(tag_id):
    ...

delay(), apply_async(), s(), si()

• delay() and apply_async() are very similar but with apply_async you can configure more attributes Calling Tasks
• s and si are used for chaining calls. s() takes additional attributes and si() ignore results from other calls. Canvas: Designing Work-flows

Chaining tasks

Chaining tasks

Build up a group of parallel tasks:

celery_group.append(extract_information_from_file.si(file.pk))

These groups were computed and after they are all successful the follow-up task will be computed:

chord(celery_group, compute_sql_statements.si('dynamic_tables.sql')).apply_async()

Time based events

You can schedule events in the database but also in the code. For example in {project_name}/celery.py:

app.conf.beat_schedule = {
    # Execute daily at 2 a.m.
    'nightly-logs-to-db': {
        'task': 'articles.tasks.log_to_db',
        'description': 'Searches the log file for user activity and writes it to the database.',
        'schedule': crontab(minute=0, hour=2),
        'args': (),
    },
}