Asynchronous Distributed Task Queues with Celery & Redis

1Concept

Celery offloads long-running, CPU-intensive, or scheduled background tasks (e.g. video transcoding, PDF generation) to distributed worker clusters backed by Redis or RabbitMQ as the message broker.

2Architecture Diagram

Web App (FastAPI) ---> task.delay(user_id) ---> [ Redis Message Queue ] ---> [ Celery Worker executes task ]

3Code Example

celery_example = '''
from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")

@app.task(bind=True, max_retries=3)
def process_report_export(self, user_id: int):
    try:
        print(f"Generating export for user {user_id}...")
        return {"status": "SUCCESS", "url": "https://storage.corp/report.pdf"}
    except Exception as exc:
        raise self.retry(exc=exc, countdown=10)
'''
print("=== Celery Distributed Task Pipeline ===")
print(celery_example.strip())

4Expected Output

=== Celery Distributed Task Pipeline ===
from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")

@app.task(bind=True, max_retries=3)
def process_report_export(self, user_id: int):
    try:
        print(f"Generating export for user {user_id}...")
        return {"status": "SUCCESS", "url": "https://storage.corp/report.pdf"}
    except Exception as exc:
        raise self.retry(exc=exc, countdown=10)

5Key Takeaways

• ✓Tasks must be idempotent because network failures may trigger automatic retries.
• ✓Run Celery with prefetch limits (`--prefetch-multiplier=1`) for long-running jobs.
• ✓Flower provides a real-time web dashboard for monitoring Celery clusters.