Running Celery at Scale in Production: A Practical Guide

I decided to document and blog my experiences of running Celery in production at scale. All of these are actual things that work and have been battle-tested at production scale. Celery is a very popular framework used by Python developers to run asynchronous tasks. Still, it comes with its own set of challenges, including running at scale and managing cloud infrastructure costs.

This was originally a talk at Pycon India 2024 in Bengaluru, India.

Substack

Slides can be found at GitHub

YouTube link for the talk

/r/django
https://redd.it/1o4nw6y

without formdata=None explicitly written it will create everything as a formdata? is that normal behaviour? when instantiating the form thing?

I was trying to figure out why a python list wouldn't be validated like a normal python list of strings and it turned out I had to write formdata=None like

form = PackingListForm(formdata=None,data=packing_list_form, meta={'csrf': False})

but this below here was silently giving me an empty list?

form = PackingListForm(data=packing_list_form, meta={'csrf': False})

but the documentation says if u dont pass formdata it will use data? but then it acted like it was formdata then? I dont get it. I know I sound confusing as hell but maybe. someone has an answer or idk if someone ever has this problem they now know what kinda weird obscure thing this is. or hell maybe I am just using it for the wrong purposes entirely. just spent 2 hours and a bunch of llms and documentations and stuff. idk. weird. weird.


/r/flask
https://redd.it/1o4tt81

Advice on logging libraries: Logfire, Loguru, or just Python's built-in logging?

Hey everyone,

I’m exploring different logging options for my projects (fastapi backend with langgraph) and I’d love some input.

So far I’ve looked at:

Python’s built-in `logging` module
Loguru
Logfire

I’m mostly interested in:

Clean and beautiful output (readability really matters)
Ease of use / developer experience
Flexibility for future scaling (e.g., larger apps, integrations)

Has anyone here done a serious comparison or has strong opinions on which one strikes the best balance?
Is there some hidden gem I should check out instead?

Thanks in advance!

/r/Python
https://redd.it/1o4uyrv

No Kernel source

https://redd.it/1o4v52d
@pythondaily

What is considered truly advanced in Django?

Hello community,

I've been working professionally with Django for 4 years, building real-world projects. I'm already comfortable with everything that's considered "advanced" in most online tutorials and guides: DRF, complex ORM usage, caching, deployment, etc.

But I feel like Django has deeper layers, those that there are very few tutorials around (djangocon and those kind of events have interesting stuff).

What do you consider the TOP tier of difficulty in Django?

Are there any concepts, patterns, or techniques that you consider truly separate a good developer from an expert?

/r/django
https://redd.it/1o52kon

Monday Daily Thread: Project ideas!

# Weekly Thread: Project Ideas 💡

Welcome to our weekly Project Ideas thread! Whether you're a newbie looking for a first project or an expert seeking a new challenge, this is the place for you.

## How it Works:

1. **Suggest a Project**: Comment your project idea—be it beginner-friendly or advanced.
2. **Build & Share**: If you complete a project, reply to the original comment, share your experience, and attach your source code.
3. **Explore**: Looking for ideas? Check out Al Sweigart's ["The Big Book of Small Python Projects"](https://www.amazon.com/Big-Book-Small-Python-Programming/dp/1718501242) for inspiration.

## Guidelines:

* Clearly state the difficulty level.
* Provide a brief description and, if possible, outline the tech stack.
* Feel free to link to tutorials or resources that might help.

# Example Submissions:

## Project Idea: Chatbot

**Difficulty**: Intermediate

**Tech Stack**: Python, NLP, Flask/FastAPI/Litestar

**Description**: Create a chatbot that can answer FAQs for a website.

**Resources**: [Building a Chatbot with Python](https://www.youtube.com/watch?v=a37BL0stIuM)

# Project Idea: Weather Dashboard

**Difficulty**: Beginner

**Tech Stack**: HTML, CSS, JavaScript, API

**Description**: Build a dashboard that displays real-time weather information using a weather API.

**Resources**: [Weather API Tutorial](https://www.youtube.com/watch?v=9P5MY_2i7K8)

## Project Idea: File Organizer

**Difficulty**: Beginner

**Tech Stack**: Python, File I/O

**Description**: Create a script that organizes files in a directory into sub-folders based on file type.

**Resources**: [Automate the Boring Stuff: Organizing Files](https://automatetheboringstuff.com/2e/chapter9/)

Let's help each other grow. Happy

/r/Python
https://redd.it/1o54sdj

What's the highest # of open source libraries you've ever packaged into a single application?

Hey everyone,

I was going down a dependency rabbit hole this morning for a new project and it got me curious. We all know the node_modules memes, but I think we sometimes lose sight of just how much incredible, free work our applications are built on.

I just ran a check on the open-source AI agent I've been building, and the number that came back was genuinely mind-boggling to me: nearly 250 open-source libraries.
It's an autonomous agent that does self-tuning for LLMs, so it needs a whole stack to work: vector databases, search indexes, SLM inference, observability and tracing, the web framework, etc. But seeing it all laid out... it's a humbling reminder that my "project" is really just a thin layer of orchestration on top of decades of work from thousands of developers I'll never meet.

It really reinforces the whole "standing on the shoulders of giants" thing. It feels like a responsibility to contribute back.
So, it made me wonder: what's your number? What's the deepest you've ever gone down the dependency rabbit hole, and what kind of project was it?

/r/Python
https://redd.it/1o4vn5m

rovr v0.4.0: an update to the modern terminal file explorer

source code: [https://github.com/nspc911/rovr](https://github.com/nspc911/rovr)

what my project does:

* it's a file manager in the terminal, made with the textual framework

comparison:

* as a python project, it cannot compete in performance with yazi at all, nor can it compete with an ncurses-focused ranger. superfile is also catching up, with its async-based preview that was just released.
* the main point of rovr was to make it a nice experience in the terminal, and also to have touch support, something that lacked, or just felt weird, when using other file explorers.

hey everyone, this follow-up on [https://www.reddit.com/r/Python/comments/1mx7zzj/rovr\_a\_modern\_customizable\_and\_aesthetically/](https://www.reddit.com/r/Python/comments/1mx7zzj/rovr_a_modern_customizable_and_aesthetically/) that I released about a month ago, and during the month, there have been quite a lot of changes! A shortcut list was added in #71 that can be spawned with `?`, so if you are confused about any commands, just press the question mark! You can also search for any keybinds if necessary. rovr also integrates with [fd](https://github.com/sharkdp/fd), so you can simply enable the `finder` plugin and press `f` to start searching! yazi/spf style `--chooser-file` flag has also been added. An extra flag `--cwd-file` Also exists to allow you to grab the file if necessary (I'm planning to remove cd on quit to favour this instead) cases where opening

/r/Python
https://redd.it/1o4q4vt

🚀 Blinter The Linter - A Cross Platform Batch Script Linter

Yes, it's 2025. Yes, people still write batch scripts. No, they shouldn't crash.

## What It Does
✅ 158 rules across Error/Warning/Style/Security/Performance
✅ Catches the nasty stuff: Command injection, path traversal, unsafe temp files
✅ Handles the weird stuff: Variable expansion, FOR loops, multilevel escaping
✅ 10MB+ files? No problem. Unicode? Got it. Thread-safe? Always.

## Get It Now

pip install Blinter

Or grab the standalone .exe from GitHub Releases

## One Command

python -m blinter script.bat

That's it. No config needed. No ceremony. Just point it at your .bat or .cmd files.

---

The first professional-grade linter for Windows batch files.
Because your automation scripts shouldn't be held together with duct tape.

📦 PyPI • ⚙️ GitHub

What My Project Does
A cross platform linter for batch scripts.

Target Audience
Developers, primarily Windows based.

Comparison
There is no comparison, it's the only batch linter so theres nothing to compare it to.

/r/Python
https://redd.it/1o4sswc

Built this Django-Unfold showcase — thinking to extend it into a CRM project

/r/django
https://redd.it/1o4psg7

Livestream with django

Hello, to give you some context: in the app I am developing, there is a service called "Events and Meetings." This service has different functionalities, one of which is that the user should be able to create an online event. My question is, besides django-channels, what other package can help achieve livestreaming for more than 10 or 20 users?

I should mention that I am developing the API using Django REST Framework.

/r/django
https://redd.it/1o4wqlr

P Adapting Karpathy’s baby GPT into a character-level discrete diffusion model

Hi everyone,

I've been exploring how discrete diffusion models can be applied to text generation and put together a single annotated Jupyter Notebook that implements a character-level discrete diffusion GPT.

It's based on Andrej Karpathy’s baby GPT from his nanoGPT repo, but instead of generating text autoregressively (left-to-right), it learns to denoise corrupted text sequences in parallel.

Discrete diffusion model in action

The notebook walks through the math, introduces what adding noise for discrete tokens means, builds discrete diffusion model from baby GPT, and trains it on Shakespeare's text using Score-Entropy based objective.

Access it on GitHub (notebook + README):
https://github.com/ash80/diffusion-gpt
or run it directly on Google Colab:
https://colab.research.google.com/github/ash80/diffusion-gpt/blob/master/The\_Annotated\_Discrete\_Diffusion\_Models.ipynb

I'd appreciate any feedback, corrections, and suggestions, especially from anyone experimenting with discrete diffusion models.

/r/MachineLearning
https://redd.it/1o4qu0h

My first medium blog on GIL

Hi everyone, today I tried my first attempt at writing a tech blog on GIL basics like what is it, why it is needed as recent 3.14 gil removal created a lot of buzz around it. Please give it a read. Only a 5 min read. Please suggest if anything wrong or any improvements needed.

**GIL in Python: The Lock That Makes and Breaks It**

PS: I wrote it by myself based on my understanding. Only used llm as proof readers so it may appear unpolished here and there.

/r/Python
https://redd.it/1o4oozb

Need advice on simulating real time bus movement and eta predictions

Hello Everyone,

I'm currently studying in college and for semester project i have selected project which can simulate real time bus movement and can predict at what bus will arrive that the certain destination.

What I have:

1. Bus departure time from station
2. Distance between each bus stop
3. Bus stop map coordinates

What I'm trying to achive:

1. Simulating bus moving on real map
2. Variable speeds, dwell times, traffic variation.
3. Estimate arrival time per stop using distance and speed.
4. Live dashboard predicting at what time will reach certain stop based upon traffic flow,speed

Help I need:

1. How to simulate it on real map (showing bus is actually moving along the route)
2. What are the best tools for this project
3. How to model traffic flow

Thanks



/r/Python
https://redd.it/1o5gurz

My first RTC app using Django channels finally went live. I need your feedback.

Hey folks, I just finished the first version of my real-time chat app built with Django, Django Channels, and WebSockets. I also used React for the frontend (which I actually learned while building this project).

It’s still missing some important stuff like testing, better error handling, and a few production-level optimizations, but it’s functional, users can register, log in, and chat in real time with real typing indicators and live presence tracker. I’d really appreciate any backend-focused feedback.

Tech stack:

Django + Django Channels
Redis for message brokering
PostgreSQL
React (frontend)

Live demo / GitHub repos:
here are the live version, frontend and the backend

Login with these accounts to explore:

email: guest1@djgram.com, password:1234, email: guest2@djgram.com, password: 1234

I know it’s far from perfect, still no tests or CI/CD setup but I wanted to get some real feedback before adding more features.
Any feedback (even brutal honesty) is super welcome.

/r/djangolearning
https://redd.it/1o4ulh1

Trying to use Google Drive to Store Media Files, But Getting "Service Accounts do not have storage quota" error when uploading

I'm building a Django app and I'm trying to use Google Drive as storage for media files via a service account, but I'm encountering a storage quota error.

# What I've Done

Set up a project in Google Cloud Console
Created a service account and downloaded the JSON key file
Implemented a custom Django storage backend using the Google Drive API v3
Configured GOOGLE_DRIVE_ROOT_FOLDER_ID in my settings

# The Error

When trying to upload files, I get:

HttpError 403: "Service Accounts do not have storage quota. Leverage shared drives
(https://developers.google.com/workspace/drive/api/guides/about-shareddrives),
or use OAuth delegation instead."

# What I've Tried

1. Created a folder in my personal Google Drive (regular Gmail account)
2. Shared it with the service account email (the client_email from the JSON file) with Editor permissions
3. Set the folder ID as GOOGLE_DRIVE_ROOT_FOLDER_ID in my Django settings

This is the code of the storage class:

```

# The original version of the code
# https://github.com/torre76/django-googledrive-storage/blob/master/gdstorage/storage.py
Copyright (c) 2014, Gian Luca Dalla Torre
All rights reserved.
"""

import enum
import json


/r/django
https://redd.it/1o5f3tr

gRPC: Client side vs Server side load balancing, which one to choose?

Hello everyone,
My setup: Two FastAPI apps calling gRPC ML services (layout analysis + table detection). Need to scale both the services.

Question: For GPU-based ML inference over gRPC, does NGINX load balancing significantly hurt performance vs client-side load balancing?

Main concerns:

* Losing HTTP/2 multiplexing benefits
* Extra latency (though probably negligible vs 2-5s processing time)
* Need priority handling for time-critical clients

Current thinking: NGINX seems simpler operationally, but want to make sure I'm not shooting myself in the foot performance-wise.

Experience with gRPC + NGINX? Client-side LB worth the complexity for this use case?

/r/Python
https://redd.it/1o5kwve

Parsegument! - Argument Parsing and function routing

Project Source code: https://github.com/RyanStudioo/Parsegument

Project Docs: https://www.ryanstudio.dev/docs/parsegument/

# What My Project Does
Parsegument allows you to easily define Command structures with Commands and CommandGroups.
Parsegument also automatically parses arguments, converts them to your desired type, then executes functions automatically, all with just one method call and a string.

# Target Audience
Parsegument is targetted for people who would like to simplify making CLIs. I started this project as I was annoyed at having to use lines and lines of switch case statements for another project I was working on

# Comparison
Compared to python's built in argparse, Parsegument has a more intuitive syntax, and makes it more convenient to route and execute functions.

This project is still super early in development, I aim to add other features like aliases, annotations, and more suggestions from you guys!

/r/Python
https://redd.it/1o5jkb9