DataCompose: Bringing shadcn's copy-to-own pattern to data engineering


# What My Project Does

DataCompose provides 100+ battle-tested PySpark primitives for cleaning contact data (addresses, emails, phones). Instead of importing a library, you copy the source code directly into your project (like Shadcn). Think of it as production-ready snippets that you own and can modify.

## Target Audience

- **Data Engineers** working in restricted environments (banks, healthcare, government)
- **Teams** tired of writing the same address/email/phone cleaning logic at every job
- **Anyone** who needs to customize data cleaning for their specific edge cases
- **Spark users** who want composable transformations without external dependencies

## Comparisons

- **vs pandas/polars**: Built specifically for PySpark at scale, not local dataframes
- **vs Great Expectations**: We clean data, and have validation functions
- **vs dbt**: We handle the messy cleaning before your dbt transformations
- **vs custom UDFs**: Pre-tested, composable, and includes 336 tests

This is the architecture that I went with, there is (supposed) a registry that lives where the datacompose cli copies and pastes the primitive (the data cleaner that you want) to your repo, and you can fully edit the code, and extend anyway you want. And then you can delete the cli and the code still works.

## Code Example

```bash
datacompose init

# Generate email cleaning primitives
datacompose add clean_emails --target

/r/Python
https://redd.it/1mq0v3w

React + Django Channels Help (groupadd() and groupsend() not working)

Hi,

I am learning how to write a project using Django REST Framework + Django Channels + React for the first time and noticed that I for some reason methods like .groupadd() and .groupsend() do not seem to be working in my project, preventing me from joining the Django Channels room and sending messages between the room's members, respectively.

I have tried using ChatGPT to help me debug the problem but it keeps sending me around in circles by asking me to change the "type" field from "chat.join" to "chatjoin" (the name of my async def chatjoin() method), putting print statements before and behind groupadd() and groupsend() to see if those statements appear in the terminal and thus if execution has stopped during the execution of groupadd()/groupsend(), as well as wrapping the chatjoin() method's implementation in try-catch statements to see if it is silently failing, verifying if my Redis database is up and running at the right address (it is), and even overriding the init() and dispatch methods of the my Consumer class to verify if the "chatjoin" method is being received by my consumer (it is).

None of these have worked and it seems like my entire Consumer class is working (I

/r/djangolearning
https://redd.it/1mk8i2j

Swizzle: flexible multi-attribute access in Python

Ever wished you could just do `obj.yxz` and grab all three at once? I got a bit obsessed playing around with `__getattr__` and `__setattr__`, and somehow it turned into a tiny library.

# What my Project Does

Swizzle lets you grab or assign multiple attributes at once, and it works with regular classes, dataclasses, Enums, etc. By default, swizzled attributes return a `swizzledtuple` (like an enhanced `namedtuple`) that keeps the original class name and allows continuous swizzling.

import swizzle

# Example with custom separator
@swizzle(sep='_', setter=True)
class Person:
def __init__(self, name, age, city, country):
self.name = name
self.age = age
self.city = city
self.country = country

p = Person("Jane", 30, "Berlin", "Germany")

# Get multiple attributes with separator


/r/Python
https://redd.it/1muhw70

I created a 5 hour free tutorial on creating your own git from scratch using Python

Ever wondered what happens when you run git init, git add, or git commit? In this 5 hour tutorial, we’ll build a minimal version of Git from scratch in Python. By the end, you’ll understand the internals - objects, commits, branches and more because you’ll have written them yourself!

Watch the video here - https://www.youtube.com/watch?v=g2cfjDENSyw

/r/Python
https://redd.it/1myvz9i

How To Implement A Recursive Formset/Form

Hello! New Django dev here. How would one implement a recursive formset?

For example, I want to nest `FormsetA` inside another instance of `FormsetA` as a formset.

I tried adding it dynamically within the `__init__` such that there is a base condition (i.e., `if self.depth >= self.max_depth: return`), but it still goes on and on. Is there a sample online I could use as reference to make this work?

Any help is much appreciated!

/r/django
https://redd.it/1n6cw9x

Gmail SMTP on Railway suddenly failing with Errno 101 Network is unreachable + site slowdown when sending emails

Hey all,

I’ve had a Django app running on Railway for \~5 months without email issues. I’m using Gmail Workspace SMTP with django.core.mail.backends.smtp.EmailBackend and an app password. A few days ago, outgoing emails started failing and any view that triggers an email slows the site to a crawl.

Symptoms:

Email sends started failing out of nowhere.
Any request that sends mail hangs and degrades performance.
Sometimes seeing worker timeouts.
Swapping to Resend works, but I prefer Gmail Workspace so messages appear in our “Sent” mailbox.

Error (logs):

Traceback (most recent call last):
File "/app/users/emails.py", line 143, in sendinternalconfirmation
msg.send()
File "/opt/venv/lib/python3.13/site-packages/django/core/mail/message.py", line 301, in send
return self.getconnection(failsilently).sendmessages([self])
File "/opt/venv/lib/python3.13/site-packages/django/core/mail/backends/smtp.py", line 128, in sendmessages
newconncreated = self.open()
File "/opt/venv/lib/python3.13/site-packages/django/core/mail/backends/smtp.py", line 86, in open
self.connection = self.connectionclass(self.host, self.port, **connectionparams)
File "/root/.nix-profile/lib/python3.13/smtplib.py", line 255, in init
(code, msg) = self.connect(host, port)


/r/django
https://redd.it/1n7ozeo

I built a Python library to simplify complex SQLAlchemy queries with a clean architecture.

Hey r/Python,

Like many of you, I've spent countless hours writing boilerplate code for web APIs that use SQLAlchemy. Handling dynamic query parameters for filtering on nested relationships, sorting, full-text search, and pagination always felt repetitive and prone to errors.

To solve this, I created fastapi-query-builder.

Don't let the name fool you! While it was born from a FastAPI project, it's fundamentally a powerful, structured way to handle SQLAlchemy queries that can be adapted to any Python framework (Flask, Django Ninja, etc.).

The most unique part is its installation, inspired by shadcn/ui. Instead of being just another black-box package, you run query-builder init, and it copies the entire source code into your project. This gives you full ownership to customize, extend, or fix anything you need.

GitHub Repo: **https://github.com/Pedroffda/fastapi-query-builder**

# How it Works: A Clean Architecture

The library encourages a clean, three-layer architecture to separate concerns:

1. BaseService: The data access layer. It talks to the database using SQLAlchemy and the core QueryBuilder. It only deals with SQLAlchemy models.
2. BaseMapper: The presentation layer. It's responsible for transforming SQLAlchemy models into Pydantic schemas, intelligently handling relationship loading and field selection (select_fields).
3. BaseUseCase: The business logic layer. It coordinates the service and the mapper. Your API endpoint talks to this layer, keeping your routes incredibly clean.

# A Quick,

/r/Python
https://redd.it/1n8b41e

Class type parameters that actually do something

I was bored, so I made type parameters for python classes that are accessible within your class and contribute to behaviour . Check them out:

[https://github.com/arikheinss/ParametricTypes.py](https://github.com/arikheinss/ParametricTypes.py)

T = TypeVar("T")

class wrapper[T](metaclass = ParametricClass):
"silly wrapper class with a type restriction"

def __init__(self, x: T):
self.set(x)

def set(self, v: T):
if not isinstance(v, T):
raise TypeError(f"wrapper of type ({T}) got value of type {type(v)}")
self.data = v

def get(self) -> T:
return self.data
# =============================================

w_int = wrapper[int](2)


/r/Python
https://redd.it/1nazumb

Visual Studio Code Error: Extremely Slow Terminal and Missing Database File with Flask and GitHub.

Hey everyone,

I'm hoping to get some help with a problem I'm having with my Python/Flask project in Visual Studio Code. I've tried a few things, but I haven't been able to solve it, and I'm a bit stuck.

Background

I was previously using GitHub Desktop to manage my repositories. All of a sudden, I started getting an error that said it couldn't find the local repository, even though the files were still on my computer.

My temporary fix was to re-clone the repository. This worked and GitHub Desktop now works fine, but it seems to have caused a new issue in Visual Studio Code.

The Current Problem

Extremely Slow Terminal: When I use the Visual Studio Code terminal to run commands like flask db init or flask run, the process is incredibly slow. It eventually tells me the process was successful, but the wait time is unusually long.

Database File Isn't Visible: Even though the terminal indicates that the flask db init command ran correctly, I can't see the database file (usually a .db file) in the Visual Studio Code file explorer. It's as if the file isn't being created or is being created in the wrong location, even though it doesn't throw any errors.

What I've

/r/flask
https://redd.it/1nbuqim

NOW - LMS: A flask based learning platform

<tl-dr>


# Python >= 3.11
# Sources:https://github.com/bmosoluciones/now-lms
# License: Apache 2
python3 -m venv venv
venv/bin/pip install now_lms
venv/bin/lmsctl database init
venv/bin/lmsctl serve
# Visit `http://127.0.0.1:8080/` in your browser, default admin user and password are `lms-admin`.

</tl-dr>


Hello, this is a project I have been working to release a online learning plataform for my sister use and my use.

NOW - LMS is designed to be simple yet powerful. Here are its key features:

* **Clean codebase**: Python and HTML5.
* **Compatible with multiple databases**: SQLite, PostgreSQL, and MySQL.
* **Complete course creation functionality**, allowing full curriculum setup.
* **Courses are organized into sections**, which group resources in a logical manner.
* **Flexible resource types** within a course section:
* YouTube videos
* PDFs
* Images
* Audio files
* Rich text content
* External HTML pages
* Slide presentations
* External resource links
* **Course types**:
* Free or paid
* Self-paced, synchronous (with tutor), or time-limited
* **Paid courses support an audit mode**, allowing limited

/r/flask
https://redd.it/1ngwce6

stop wrong ai answers in your django app before they show up: one tiny middleware + grandma clinic (beginner, mit)

hi folks, last time i posted about “semantic firewalls” and it was too abstract. this is the ultra simple django version that you can paste in 5 minutes.

**what this does in one line**
instead of fixing bad llm answers after users see them, we check the payload before returning the response. if there’s no evidence, we block it politely.

**before vs after**

* before: view returns a fluent answer with zero proof, users see it, you fix later
* after: view includes small evidence, middleware checks it, only stable answers go out

below is a minimal copy-paste. it works with any provider or local model because it’s just json discipline.

---

### 1) middleware: block ungrounded answers

`core/middleware.py`

```python
# core/middleware.py
import json
from typing import Callable
from django.http import HttpRequest, HttpResponse, JsonResponse

class SemanticFirewall:
"""
minimal 'evidence-first' guard for AI responses.
contract we expect from the view:
{ "answer": "...", "refs": [...], "coverage_ok": true }
if refs is empty or coverage_ok is false or missing, we return 422.
"""

def __init__(self, get_response: Callable):
self.get_response = get_response

def __call__(self, request: HttpRequest)

/r/django
https://redd.it/1nj9mmu

Error running app

Hello everyone, I am currently trying to implement Oauth with google in a flask app for a learning project. Building the normal auth modules with email and username work fine, however as I refactor the code to work with oauth using the python oauthlib and requests modules, I am getting this error:

```bash
(.venv)daagi@fedora:~/Desktop/sandbox/oauth-primer$ python app.py
Usage: app.py [OPTIONS]
Try 'app.py --help' for help.

Error: While importing 'app', an ImportError was raised:

Traceback (most recent call last):
File "/home/daagi/Desktop/sandbox/oauth-primer/.venv/lib64/python3.13/site-packages/flask/cli.py", line 245, in locate_app
__import__(module_name)
~~~~~~~~~~^^^^^^^^^^^^^
File "/home/daagi/Desktop/sandbox/oauth-primer/app.py", line 1, in <module>
from website import create_app
ImportError: cannot import name 'create_app' from 'website' (consider renaming '/home/daagi/Desktop/sandbox/oauth-primer/website/__init__.py' if it has the same name as a library you intended to import)```

This is my file hierachy structure:

```bash
.
├── app.py
├── LICENSE
├── oauth.log
├── __pycache__
│   └── app.cpython-313.pyc
├── README.md
├── requirements.txt
├── TODO.md
└── website
├── auth.py
├── database
│   └── db.sql
├── db.py
├── __init__.py
├── models.py
├── oauth.py
├── __pycache__
├── static
│   ├── style
│   │   └── style.css


/r/flask
https://redd.it/1nk2otw

python-cq — Lightweight CQRS package for async Python projects

# What My Project Does

[`python-cq`](https://github.com/100nm/python-cq) is a package that helps apply CQRS principles (Command Query Responsibility Segregation) in async Python projects.

The core idea of CQRS is to separate:

* **Commands** → actions that change the state of the system.
* **Queries** → operations that only read data, without side effects.
* **Events** → facts that describe something that happened, usually triggered by commands.

With `python-cq`, handlers for commands, queries, and events are just regular Python classes decorated with `@command_handler`, `@query_handler`, or `@event_handler`.
The framework automatically detects which message type is being handled based on type hints, no need to inherit from base classes or write boilerplate.

It also integrates with dependency injection through [`python-injection`](https://github.com/100nm/python-injection), which makes it easier to manage dependencies between handlers.

Example:

```python
from dataclasses import dataclass
from injection import inject
from cq import CommandBus, RelatedEvents, command_handler, event_handler

@dataclass
class UserRegistrationCommand:
email: str
password: str

@dataclass
class UserRegistered:
user_id: int
email: str

@command_handler
class UserRegistrationHandler:
def __init__(self, events: RelatedEvents):
self.events = events

async def handle(self, command: UserRegistrationCommand):
""" register the user """
user_id = ...


/r/Python
https://redd.it/1nnimms

Django Forms Lifecycle

class ContractItemForm(forms.ModelForm):
product = forms.ModelChoiceField(
queryset=Product.objects.none(), # start with no choices
required=True,
)

class Meta:
model = ContractItem
fields = ['product']

def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)

# default: empty queryset
self.fields['product'].queryset = Product.objects.none()

# if editing an existing instance, allow that one product
if self.instance and self.instance.pk and self.instance.product_id:


/r/django
https://redd.it/1nw7763

Show & Tell PyClue/Cluedo-style deduction game in Python (pygame)

What My Project Does
I built a small Clue/Cluedo-style deduction game in Python using pygame. It’s a scene-based desktop game with clean, portable asset handling. You can run it from source or as a single Windows .exe (PyInstaller one-file). The repo is meant to be a practical reference for packaging pygame apps reliably.

Source code (GitHub):
https://github.com/rozsit/112\_PyClue\_Game

(Windows build is in the GitHub Release — see “Downloads” below.)

Target Audience

Python devs interested in pygame architecture and packaging to .exe.
Learners who want a small, readable codebase (scenes, UI, audio, animations).
Casual players who just want to double-click an `.exe` and try a Clue-like game.

Comparison
Compared with other “pygame Clue clones” or small hobby games, this repo focuses on robust distribution and developer ergonomics:

Works the same in dev and frozen modes (PyInstaller).
Global hooks route string paths for `pygame.image.load`, `pygame.mixer.Sound`, and `pygame.mixer.music.load` → fewer path bugs after packaging.
Audio init on Windows is hardened (ensure_audio() tries multiple drivers/buffer sizes).
Animated GIF support via Pillow (e.g., winner screen fireworks → frames + per-frame duration).
Comes with a one-command build script (PowerShell) and a SHA-256 file for integrity checks.

How Python Is Used

pygame for windowing, scenes, input, and rendering.
Pillow to

/r/Python
https://redd.it/1nx3hkg

Rant: Python imports are convoluted and easy to get wrong

Inspired by the famous "module 'matplotlib' has no attribute 'pyplot'" error, but let's consider another example: numpy.

This works:

from numpy import ma, ndindex, typing
ma.getmask
ndindex.ndincr
typing.NDArray

But this doesn't:

import numpy
numpy.ma.getmask
numpy.ndindex.ndincr
numpy.typing.NDArray # AttributeError

And this doesn't:

import numpy.ma, numpy.typing
numpy.ma.getmask
numpy.typing.NDArray
import numpy.ndindex # ModuleNotFoundError

And this doesn't either:

from numpy.ma import getmask
from numpy.typing import NDArray
from numpy.ndindex import ndincr # ModuleNotFoundError

There are explanations behind this (numpy.ndindex is not a module, numpy.typing has never been imported so the attribute doesn't exist yet, numpy.ma is a module and has been imported by numpy's __init__.py so everything works), but they don't convince me. I see no reason why import A.B should only work when B is a module. And I see no reason why using a not-yet-imported submodule shouldn't just import it implicitly, clearly you were going to import it anyway. All those subtle inconsistencies where you can't be sure whether something

/r/Python
https://redd.it/1o9gyxa

QuickScale v0.60.0 - Deploy Django to Railway in 5 minutes with one command

Just shipped QuickScale v0.60.0, and I'm excited about the deployment automation we built.

What's New

One command handles the entire deployment workflow:

quickscale deploy railway

This automatically:

Provisions PostgreSQL database \- No manual Railway dashboard clicking
Generates SECRET_KEY \- Uses Django's secure random generation
Configures environment \- Interactive prompts for ALLOWED\_HOSTS, DEBUG, etc.
Runs migrations \- Executes `manage.py` migrate on Railway
Collects static files \- Handles `collectstatic` for production
Sets up HTTPS \- Railway auto-provisions SSL certificates

The entire process takes under 5 minutes.

# Quick Start

# Install QuickScale
pip install quickscale

# Create new project
quickscale init my-saas-app
cd my-saas-app

# Deploy to Railway
quickscale deploy railway
The CLI guides you through the process with clear prompts.

# Looking for Feedback, would love to hear yout thoughts.

GitHub: github.com/Experto-AI/quickscale

Railway deployment docs: Railway Guide

/r/django
https://redd.it/1oea4p0

🌟 Myfy: a modular Python framework with a built-in frontend

What It Does

Tired of gluing FastAPI + Next.js together, I built **Myfy** — a modular Python framework that ships with a frontend by default.

Run:

myfy frontend init

and you instantly get:

📝 Jinja2 templates
🎨 DaisyUI 5 + Tailwind 4 + Vite + HMR
🌗 Dark mode
🚀 Zero config that works out of the box

Target Audience

For Python devs who love backend work but want a frontend without touching JS.
Perfect for side projects, internal tools, or fast prototypes.

Comparison

Unlike FastAPI + Next.js or Flask + React, Myfy gives you a full-stack Python experience with plain HTML + modern CSS.

Repo → github.com/psincraian/myfy
If it sounds cool, drop a ⭐ and tell me what you think!

/r/Python
https://redd.it/1olyidq

I divided up models.py into different files in flask and I getting circular import errors. Does anyone have any suggestions?

I used chatgpt to create a flask file tree with blueprints.

My code was very similar to this and it was working except the templates are there own folder not nested within the other folders like auth













myapp/

│

├── app.py

├── config.py

├── extensions.py

│

├── blueprints/

│ ├── __init__.py

│ ├── main/

│ │ ├── __init__.py

│ │ ├── routes.py

│ │ ├── models.py

│ │ └── templates/

│ │ └── main/

│ │ └── index.html

│ │

│ └── auth/

│ ├── __init__.py

│ ├── routes.py

│ ├── models.py

│ └── templates/

│ └── auth/

│ └── login.html

│

└── templates/

└── base.html





The problem is I changed the blueprints

/r/flask
https://redd.it/1oydkld

Hi all I'm making my first large scale app and I'm struggling to organize the code!

Right now it's a monolith, I've got everything separated into models->feature->model.py etc. it's ok but it's making development a nightmare when I have to search for the model serializer signal service url view to develop.

I was thinking about separating every feature into Django apps But they all really depend on one another and will require loads of inter app foreign keys, so I was thinking of leaving it in a monolith but separating into app style folders ie. api.feature.models - views - serializers - admin - urls etc. but I was wondering if this would slow down the app as it would all be linked via . Init

/r/django
https://redd.it/1p8siw6