Added live streaming on my Django project
I recently added live-streaming to my Django project would like for folks to check it out. Has live comment updating and everything.
Uses:
Aws IVS
Vastvp video player
Vastcomments
Once you create an account go to this link and start a livestream:
https://vastvids.com/live/create_livestream/
/r/django
https://redd.it/1kcf05s
I recently added live-streaming to my Django project would like for folks to check it out. Has live comment updating and everything.
Uses:
Aws IVS
Vastvp video player
Vastcomments
Once you create an account go to this link and start a livestream:
https://vastvids.com/live/create_livestream/
/r/django
https://redd.it/1kcf05s
Frontend Help
I am super new to Django (and web dev overall) and keep being told that Django is backend only. There's advice on plenty of different front end options to incorporate, but sounds like I would need to learn a whole new language (Javascript) and some specific framework (Vue,React, etc).
If my goal is to merely display bar graphs and charts for some sport stats- what is the simplest frontend I would need? I know there's Django templates but my research shows that wouldn't be great for graphs (especially if I wanted them to be interactive) - Is that a correct assumption? Has anyone used Django templates for interactive graphs and charts?
I feel very confident with my overall python skills so I think I can figure out the backend portion of things, but don't know what the line of demarcation is for needing a frontend or what is a super low level frontend framework that could.
I've seen options such as ChartJs and Bootstrap- are these good implementation options? Would it be better to just have straight HTML files in the template folders and have Javascript and CSS to compliment them? Or are all those band aid solutions and its
/r/djangolearning
https://redd.it/1kbtlvn
I am super new to Django (and web dev overall) and keep being told that Django is backend only. There's advice on plenty of different front end options to incorporate, but sounds like I would need to learn a whole new language (Javascript) and some specific framework (Vue,React, etc).
If my goal is to merely display bar graphs and charts for some sport stats- what is the simplest frontend I would need? I know there's Django templates but my research shows that wouldn't be great for graphs (especially if I wanted them to be interactive) - Is that a correct assumption? Has anyone used Django templates for interactive graphs and charts?
I feel very confident with my overall python skills so I think I can figure out the backend portion of things, but don't know what the line of demarcation is for needing a frontend or what is a super low level frontend framework that could.
I've seen options such as ChartJs and Bootstrap- are these good implementation options? Would it be better to just have straight HTML files in the template folders and have Javascript and CSS to compliment them? Or are all those band aid solutions and its
/r/djangolearning
https://redd.it/1kbtlvn
Reddit
From the djangolearning community on Reddit
Explore this post and more from the djangolearning community
For multi-model fetch and pandas resample
I'm relatively new to Django, and I will admit, I've been struggling on how to get this to work a while. Currently, I have left this feature out of the dashboard out till a future version, but it still bugs me.
class Palworldplayermetrics(
models
.
Model
):
id = models.BigAutoField(primary_key=True)
player = models.ForeignKey('Palworldplayers', models.DO_NOTHING, related_name='playerinfo', blank=True, null=True)
palplayermetrictype = models.TextField(blank=True, null=True) # ALWAYS PING
data = models.FloatField(blank=True, null=True)
insert_time = models.DateTimeField(blank=True, null=True)
server = models.ForeignKey(Palwordservers, models.DO_NOTHING, blank=True, null=True)
objects = DataFrameManager()
class Meta:
managed = False
db_table = 'palworldplayermetrics'
app_label = 'databot'
class Palwordservers(
models
.
Model
):
name = models.TextField(blank=True, null=True)
/r/django
https://redd.it/1kcjenx
I'm relatively new to Django, and I will admit, I've been struggling on how to get this to work a while. Currently, I have left this feature out of the dashboard out till a future version, but it still bugs me.
class Palworldplayermetrics(
models
.
Model
):
id = models.BigAutoField(primary_key=True)
player = models.ForeignKey('Palworldplayers', models.DO_NOTHING, related_name='playerinfo', blank=True, null=True)
palplayermetrictype = models.TextField(blank=True, null=True) # ALWAYS PING
data = models.FloatField(blank=True, null=True)
insert_time = models.DateTimeField(blank=True, null=True)
server = models.ForeignKey(Palwordservers, models.DO_NOTHING, blank=True, null=True)
objects = DataFrameManager()
class Meta:
managed = False
db_table = 'palworldplayermetrics'
app_label = 'databot'
class Palwordservers(
models
.
Model
):
name = models.TextField(blank=True, null=True)
/r/django
https://redd.it/1kcjenx
Reddit
From the django community on Reddit
Explore this post and more from the django community
Friday Daily Thread: r/Python Meta and Free-Talk Fridays
# Weekly Thread: Meta Discussions and Free Talk Friday ποΈ
Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!
## How it Works:
1. Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
2. Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
3. News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.
## Guidelines:
All topics should be related to Python or the /r/python community.
Be respectful and follow Reddit's Code of Conduct.
## Example Topics:
1. New Python Release: What do you think about the new features in Python 3.11?
2. Community Events: Any Python meetups or webinars coming up?
3. Learning Resources: Found a great Python tutorial? Share it here!
4. Job Market: How has Python impacted your career?
5. Hot Takes: Got a controversial Python opinion? Let's hear it!
6. Community Ideas: Something you'd like to see us do? tell us.
Let's keep the conversation going. Happy discussing! π
/r/Python
https://redd.it/1kcnhrk
# Weekly Thread: Meta Discussions and Free Talk Friday ποΈ
Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!
## How it Works:
1. Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
2. Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
3. News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.
## Guidelines:
All topics should be related to Python or the /r/python community.
Be respectful and follow Reddit's Code of Conduct.
## Example Topics:
1. New Python Release: What do you think about the new features in Python 3.11?
2. Community Events: Any Python meetups or webinars coming up?
3. Learning Resources: Found a great Python tutorial? Share it here!
4. Job Market: How has Python impacted your career?
5. Hot Takes: Got a controversial Python opinion? Let's hear it!
6. Community Ideas: Something you'd like to see us do? tell us.
Let's keep the conversation going. Happy discussing! π
/r/Python
https://redd.it/1kcnhrk
Redditinc
Reddit Rules
Reddit Rules - Reddit
How to deploy your flask application.
Hi guys, it's me again:) after a few days of work with the flask wiki community, we've come up with a little tutorial on application deployment!
I hope it can help you, any feedback, error reporting etc is welcome as usual!
https://flaskwiki.wiki/rs/deployment-guide
We also have a github now where you can participate in the wiki yourself! We are and always will be free, I really hope that all together we can make flask more popular, give it the light it deserves.!!
https://github.com/Ciela2002/flaskwiki/tree/main
/r/flask
https://redd.it/1kazu6h
Hi guys, it's me again:) after a few days of work with the flask wiki community, we've come up with a little tutorial on application deployment!
I hope it can help you, any feedback, error reporting etc is welcome as usual!
https://flaskwiki.wiki/rs/deployment-guide
We also have a github now where you can participate in the wiki yourself! We are and always will be free, I really hope that all together we can make flask more popular, give it the light it deserves.!!
https://github.com/Ciela2002/flaskwiki/tree/main
/r/flask
https://redd.it/1kazu6h
Template strings in Python 3.14: an useful new feature or just an extra syntax?
Python foundation just accepted PEP 750 for template strings, or called t-strings. It will come with Python 3.14.
There are already so many methods for string formatting in Python, why another one??
Here is an article to dicsuss its usefulness and motivation. What's your view?
/r/Python
https://redd.it/1kch7hf
Python foundation just accepted PEP 750 for template strings, or called t-strings. It will come with Python 3.14.
There are already so many methods for string formatting in Python, why another one??
Here is an article to dicsuss its usefulness and motivation. What's your view?
/r/Python
https://redd.it/1kch7hf
Python Enhancement Proposals (PEPs)
PEP 750 β Template Strings | peps.python.org
This PEP introduces template strings for custom string processing.
I just published an update for my articles on Python packaging (PEP 751) and some remaining issues
Hi everyone!
My last two articles on Python packaging received a lot of, interactions. So when PEP 751 was accepted I thought of updating my articles, but it felt, dishonest. I mean, one could just read the PEP and get the gist of it. Like, it doesn't require a whole article for it. But then at work I had to help a lot across projects on the packaging part and through the questions I got asked here and there, I could see a structure for a somewhat interesting article.
So the structure goes like this, why not just use the good old requirements.txt (yes we still do, or, did, that here and there at work), what were the issues with it, how some can be solved, how the lock file solves some of them, why the current `pylock.toml` is not perfect yet, the differences with `uv.lock`.
And since CUDA is the bane of my existence, I decided to also include a section talking about different issues with the current Python packaging state. This was the hardest part I think. Because it has to be simple enough to onboard everyone and not too simple that it's simply wrong from an expert's point of
/r/Python
https://redd.it/1kcumjf
Hi everyone!
My last two articles on Python packaging received a lot of, interactions. So when PEP 751 was accepted I thought of updating my articles, but it felt, dishonest. I mean, one could just read the PEP and get the gist of it. Like, it doesn't require a whole article for it. But then at work I had to help a lot across projects on the packaging part and through the questions I got asked here and there, I could see a structure for a somewhat interesting article.
So the structure goes like this, why not just use the good old requirements.txt (yes we still do, or, did, that here and there at work), what were the issues with it, how some can be solved, how the lock file solves some of them, why the current `pylock.toml` is not perfect yet, the differences with `uv.lock`.
And since CUDA is the bane of my existence, I decided to also include a section talking about different issues with the current Python packaging state. This was the hardest part I think. Because it has to be simple enough to onboard everyone and not too simple that it's simply wrong from an expert's point of
/r/Python
https://redd.it/1kcumjf
Reddit
From the Python community on Reddit
Explore this post and more from the Python community
I Made AI Powered Bulk Background Remover
What My Project Does
A desktop tool that removes backgrounds from multiple images in bulk using the
Target Audience
Ideal for individuals or small businesses needing fast, unlimited, and offline background removal.
Comparison
Unlike most online tools, itβs completely free, offline, and has no usage limits. (This is exactly why I did this project)
Github
/r/Python
https://redd.it/1kclan0
What My Project Does
A desktop tool that removes backgrounds from multiple images in bulk using the
rembg library.Target Audience
Ideal for individuals or small businesses needing fast, unlimited, and offline background removal.
Comparison
Unlike most online tools, itβs completely free, offline, and has no usage limits. (This is exactly why I did this project)
Github
/r/Python
https://redd.it/1kclan0
GitHub
GitHub - verlorengest/BGTANK: Bulk Background Remover
Bulk Background Remover. Contribute to verlorengest/BGTANK development by creating an account on GitHub.
Looking for intermediate/advanced level python courses for data analytics
I have foundational knowledge on pandas, NumPy, Matplotlib, Sci-kit learn, plotly SQL, SQLite, and PostgreSQL. Are there any courses out that that skip the basics and go straight into more complex projects? Or, do you have any other suggestions on how I can gain strengthen my skills? My goal is to become a data analyst. I am still undecided on what field/topic I am most interested in but I have good faith that I will figure it out on the way. I appreciate any wisdom you all have to share!
/r/Python
https://redd.it/1kco45y
I have foundational knowledge on pandas, NumPy, Matplotlib, Sci-kit learn, plotly SQL, SQLite, and PostgreSQL. Are there any courses out that that skip the basics and go straight into more complex projects? Or, do you have any other suggestions on how I can gain strengthen my skills? My goal is to become a data analyst. I am still undecided on what field/topic I am most interested in but I have good faith that I will figure it out on the way. I appreciate any wisdom you all have to share!
/r/Python
https://redd.it/1kco45y
Reddit
From the Python community on Reddit
Explore this post and more from the Python community
What do you prefer Bootstrap or Tailwind?
I am from the "older" generation. We started with Bootstrap, and it worked for years without fail. The classes are easy to remember and clean.
Tailwind, on the other hand, looks really professional, modern, and sleek. I like the fonts and colours that come with the library by default, but I don't like having 3000 classes in my markup, and I am okay with writing custom CSS.
With that said, I am using Tailwind more and more now just because it looks so good without me having to add extra CSS. How about you? Django developers tend to still stick with Bootstrap or are we moving along into Tailwind?
/r/django
https://redd.it/1kcxe2x
I am from the "older" generation. We started with Bootstrap, and it worked for years without fail. The classes are easy to remember and clean.
Tailwind, on the other hand, looks really professional, modern, and sleek. I like the fonts and colours that come with the library by default, but I don't like having 3000 classes in my markup, and I am okay with writing custom CSS.
With that said, I am using Tailwind more and more now just because it looks so good without me having to add extra CSS. How about you? Django developers tend to still stick with Bootstrap or are we moving along into Tailwind?
/r/django
https://redd.it/1kcxe2x
Reddit
From the django community on Reddit
Explore this post and more from the django community
Steps to learning deployment
Currently im using DO's App Platform to run my client's app. However I want to learn to deploy an app from scratch by myself. What are the steps I need to learn? Do I use docker on a vps or go some other route?
/r/django
https://redd.it/1kcy6ny
Currently im using DO's App Platform to run my client's app. However I want to learn to deploy an app from scratch by myself. What are the steps I need to learn? Do I use docker on a vps or go some other route?
/r/django
https://redd.it/1kcy6ny
Reddit
From the django community on Reddit
Explore this post and more from the django community
Storing lists in Database (django-react-docker)
Hello,
I'm working on a react-django project, the website is for courses showcasing, each course has it's own information to display, at first I hard coded each course, and stored the data in react in a json file, and since I'm working on a multilingual website this came in handy (I've used i18n for this). Anyway but I was recommended to store the courses in a database instead, and that's what I'm trying to do.
in Django I created a model for the courses, and I connected it to react and it worked just fine, but for some of the details of the course they're written as a list, I tried to store them in the database with /n/ but it didn't work. also some paragraphs I needed to separate them or style them, it's difficult now that's it's all stored as one paragraph in DB. Any advice on how should I store them? or any advice on this matter would be much appreciated.
Now for the database at first I sticked with default django's sql, but chat gpt recommended that I use PostgreSQL (I've never used it) and use Docker for it too, I'm having trouble with Docker as well,
/r/django
https://redd.it/1kd0bjt
Hello,
I'm working on a react-django project, the website is for courses showcasing, each course has it's own information to display, at first I hard coded each course, and stored the data in react in a json file, and since I'm working on a multilingual website this came in handy (I've used i18n for this). Anyway but I was recommended to store the courses in a database instead, and that's what I'm trying to do.
in Django I created a model for the courses, and I connected it to react and it worked just fine, but for some of the details of the course they're written as a list, I tried to store them in the database with /n/ but it didn't work. also some paragraphs I needed to separate them or style them, it's difficult now that's it's all stored as one paragraph in DB. Any advice on how should I store them? or any advice on this matter would be much appreciated.
Now for the database at first I sticked with default django's sql, but chat gpt recommended that I use PostgreSQL (I've never used it) and use Docker for it too, I'm having trouble with Docker as well,
/r/django
https://redd.it/1kd0bjt
Reddit
From the django community on Reddit
Explore this post and more from the django community
ETL template with clean architecture
Hey folks π
Iβve put together a simple yet production-ready ETL (Extract - Transform - Load) template project that aims to go beyond the typical examples.
Link: [https://github.com/mglowinski93/EtlTemplate](https://github.com/mglowinski93/EtlTemplate)
What it offers:
* Isolated business logic
* CQRS (separate read/write models)
* Django-based API with Swagger docs
* Admin panel for exporting results
* Framework-agnostic core β you can swap Django for something else if needed
**What my it does?**
It's simple good quality showcase of ETL process.
**Target audience:**
Anyone building or experimenting with ETL pipelines in a structured, maintainable way β especially if you're tired of seeing everything shoved into one etl.py.
**Comparison:**
Most ETL templates out there skip over Domain-Driven Design (DDD) and Clean Architecture concepts. This project is a minimal example to showcase how those ideas can be applied in a real ETL setup.
Happy to hear feedback or ideas!
/r/Python
https://redd.it/1kd4aib
Hey folks π
Iβve put together a simple yet production-ready ETL (Extract - Transform - Load) template project that aims to go beyond the typical examples.
Link: [https://github.com/mglowinski93/EtlTemplate](https://github.com/mglowinski93/EtlTemplate)
What it offers:
* Isolated business logic
* CQRS (separate read/write models)
* Django-based API with Swagger docs
* Admin panel for exporting results
* Framework-agnostic core β you can swap Django for something else if needed
**What my it does?**
It's simple good quality showcase of ETL process.
**Target audience:**
Anyone building or experimenting with ETL pipelines in a structured, maintainable way β especially if you're tired of seeing everything shoved into one etl.py.
**Comparison:**
Most ETL templates out there skip over Domain-Driven Design (DDD) and Clean Architecture concepts. This project is a minimal example to showcase how those ideas can be applied in a real ETL setup.
Happy to hear feedback or ideas!
/r/Python
https://redd.it/1kd4aib
GitHub
GitHub - mglowinski93/EtlTemplate: Template for Extract-Transform-Load (ETL).
Template for Extract-Transform-Load (ETL). Contribute to mglowinski93/EtlTemplate development by creating an account on GitHub.
Programmatically navigate between directories in the file browser pane (JupyterLab)
Hello.
I would like to know if there is a way to programmatically change the folder shown in the file browser pane in JupyterLab?
For context.
I have a directory A and B both of which share a parent 5 folders up.
My notebook that I run is in directory A (and so the file browser pane will be showing the contents of directory A) and as part of the code it produces files that are saved in directory B. I would like there to be a quick way for the users of the notebook to quickly jump to the results folder (directory B) without having to navigate folder by folder to the results. Ideally with a click of a button.
Thank you in advance.
/r/IPython
https://redd.it/1kd6j86
Hello.
I would like to know if there is a way to programmatically change the folder shown in the file browser pane in JupyterLab?
For context.
I have a directory A and B both of which share a parent 5 folders up.
My notebook that I run is in directory A (and so the file browser pane will be showing the contents of directory A) and as part of the code it produces files that are saved in directory B. I would like there to be a quick way for the users of the notebook to quickly jump to the results folder (directory B) without having to navigate folder by folder to the results. Ideally with a click of a button.
Thank you in advance.
/r/IPython
https://redd.it/1kd6j86
Reddit
From the IPython community on Reddit
Explore this post and more from the IPython community
How to Deploy Django Project with tailwind css styling on Render
So , when I locally want to test, first i build Tailwind CSS using the command
The issue I am facing is that I have successfully deployed it on render but the styling is not being applied . What I tried was to use gunicorn to run it on port locally, and tried this
But the error is that tailwind is an unknown command. Can you guys help me? I know there
/r/django
https://redd.it/1kctslf
So , when I locally want to test, first i build Tailwind CSS using the command
python `manage.py` tailwind start When Tailwind is built, then on parallel I run python `manage.py` runserver . And that's how I get all the styling of Tailwind classes The issue I am facing is that I have successfully deployed it on render but the styling is not being applied . What I tried was to use gunicorn to run it on port locally, and tried this
: import os from django.core.wsgi import get_wsgi_application from `django.core.management` import call_command try: call_command('tailwind', 'start') except Exception as e: print(f"Tailwind build failed: {e}") os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'wcp.settings') application = get_wsgi_application() import os from django.core.wsgi import get_wsgi_application from `django.core.management` import call_command try: call_command('tailwind', 'start') except Exception as e: print(f"Tailwind build failed: {e}") os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings') application = get_wsgi_application()But the error is that tailwind is an unknown command. Can you guys help me? I know there
/r/django
https://redd.it/1kctslf
Is there a way to do this
Hello guys hope you are all doing well, i am working on an app that automate the process of cv creation because i am tired on updating my cv by hand each time to match a specific job description , espicially that for a lot of jobs i need to change the template i am using completely , and not only this but probably some freinds gonna use it too. Anyways here how it work , the user chose the templates he want , a form is then submited to the user where he fills his data , a prview of the template is generated then the user can download it if he want , my question is do i need to create a form and a view for each template manually or does anyone have an idea how to make this process dynamic . I hope i explained this well english isn t my first language and thank you in advance :)
/r/django
https://redd.it/1kcv3kn
Hello guys hope you are all doing well, i am working on an app that automate the process of cv creation because i am tired on updating my cv by hand each time to match a specific job description , espicially that for a lot of jobs i need to change the template i am using completely , and not only this but probably some freinds gonna use it too. Anyways here how it work , the user chose the templates he want , a form is then submited to the user where he fills his data , a prview of the template is generated then the user can download it if he want , my question is do i need to create a form and a view for each template manually or does anyone have an idea how to make this process dynamic . I hope i explained this well english isn t my first language and thank you in advance :)
/r/django
https://redd.it/1kcv3kn
Reddit
From the django community on Reddit
Explore this post and more from the django community
D Self-Promotion Thread
Please post your personal projects, startups, product placements, collaboration needs, blogs etc.
Please mention the payment and pricing requirements for products and services.
Please do not post link shorteners, link aggregator websites , or auto-subscribe links.
\--
Any abuse of trust will lead to bans.
Encourage others who create new posts for questions to post here instead!
Thread will stay alive until next one so keep posting after the date in the title.
\--
Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.
/r/MachineLearning
https://redd.it/1kcq3du
Please post your personal projects, startups, product placements, collaboration needs, blogs etc.
Please mention the payment and pricing requirements for products and services.
Please do not post link shorteners, link aggregator websites , or auto-subscribe links.
\--
Any abuse of trust will lead to bans.
Encourage others who create new posts for questions to post here instead!
Thread will stay alive until next one so keep posting after the date in the title.
\--
Meta: This is an experiment. If the community doesnt like this, we will cancel it. This is to encourage those in the community to promote their work by not spamming the main threads.
/r/MachineLearning
https://redd.it/1kcq3du
Reddit
From the MachineLearning community on Reddit
Explore this post and more from the MachineLearning community
Saturday Daily Thread: Resource Request and Sharing! Daily Thread
# Weekly Thread: Resource Request and Sharing π
Stumbled upon a useful Python resource? Or are you looking for a guide on a specific topic? Welcome to the Resource Request and Sharing thread!
## How it Works:
1. Request: Can't find a resource on a particular topic? Ask here!
2. Share: Found something useful? Share it with the community.
3. Review: Give or get opinions on Python resources you've used.
## Guidelines:
Please include the type of resource (e.g., book, video, article) and the topic.
Always be respectful when reviewing someone else's shared resource.
## Example Shares:
1. Book: "Fluent Python" \- Great for understanding Pythonic idioms.
2. Video: Python Data Structures \- Excellent overview of Python's built-in data structures.
3. Article: Understanding Python Decorators \- A deep dive into decorators.
## Example Requests:
1. Looking for: Video tutorials on web scraping with Python.
2. Need: Book recommendations for Python machine learning.
Share the knowledge, enrich the community. Happy learning! π
/r/Python
https://redd.it/1kdfo8x
# Weekly Thread: Resource Request and Sharing π
Stumbled upon a useful Python resource? Or are you looking for a guide on a specific topic? Welcome to the Resource Request and Sharing thread!
## How it Works:
1. Request: Can't find a resource on a particular topic? Ask here!
2. Share: Found something useful? Share it with the community.
3. Review: Give or get opinions on Python resources you've used.
## Guidelines:
Please include the type of resource (e.g., book, video, article) and the topic.
Always be respectful when reviewing someone else's shared resource.
## Example Shares:
1. Book: "Fluent Python" \- Great for understanding Pythonic idioms.
2. Video: Python Data Structures \- Excellent overview of Python's built-in data structures.
3. Article: Understanding Python Decorators \- A deep dive into decorators.
## Example Requests:
1. Looking for: Video tutorials on web scraping with Python.
2. Need: Book recommendations for Python machine learning.
Share the knowledge, enrich the community. Happy learning! π
/r/Python
https://redd.it/1kdfo8x
Amazon
Fluent Python: Clear, Concise, and Effective Programming
Fluent Python: Clear, Concise, and Effective Programming [Ramalho, Luciano] on Amazon.com. *FREE* shipping on qualifying offers. Fluent Python: Clear, Concise, and Effective Programming
I built a PySpark data validation framework to replace PyDeequ β feedback welcome
Hey everyone,
Iβd like to share a project Iβve been working on: [**SparkDQ**](https://github.com/sparkdq-community/sparkdq) β an open-source framework for validating data in PySpark.
**What it does:**
SparkDQ helps you validate your data β both at the row level and aggregate level β directly inside your Spark pipelines.
It supports Python-native and declarative configs (e.g. YAML, JSON, or external sources like DynamoDB), with built-in support for fail-fast and quarantine-based validation strategies.
**Target audience:**
This is built for data engineers and analysts working with Spark in production. Whether you're building ETL pipelines or preparing data for ML, SparkDQ is designed to give you full control over your data quality logic β without relying on heavy wrappers.
**Comparison:**
* Fully written in Python
* Row-level visibility with structured error metadata
* Plugin architecture for custom checks
* Zero heavy dependencies (just PySpark + Pydantic)
* Clean separation of valid and invalid data β with built-in handling for quarantining bad records
If youβve used PyDeequ or struggled with validating Spark data in a Pythonic way, Iβd love your feedback β on naming, structure, design, anything.
* β [GitHub Repo β SparkDQ](https://github.com/sparkdq-community/sparkdq)
* βοΈ [Medium article β Why I moved beyond PyDeequ](https://medium.com/aws-tip/goodbye-pydeequ-time-to-upgrade-your-data-quality-stack-d86fe9cdc5be)
Thanks for reading!
/r/Python
https://redd.it/1kdgumc
Hey everyone,
Iβd like to share a project Iβve been working on: [**SparkDQ**](https://github.com/sparkdq-community/sparkdq) β an open-source framework for validating data in PySpark.
**What it does:**
SparkDQ helps you validate your data β both at the row level and aggregate level β directly inside your Spark pipelines.
It supports Python-native and declarative configs (e.g. YAML, JSON, or external sources like DynamoDB), with built-in support for fail-fast and quarantine-based validation strategies.
**Target audience:**
This is built for data engineers and analysts working with Spark in production. Whether you're building ETL pipelines or preparing data for ML, SparkDQ is designed to give you full control over your data quality logic β without relying on heavy wrappers.
**Comparison:**
* Fully written in Python
* Row-level visibility with structured error metadata
* Plugin architecture for custom checks
* Zero heavy dependencies (just PySpark + Pydantic)
* Clean separation of valid and invalid data β with built-in handling for quarantining bad records
If youβve used PyDeequ or struggled with validating Spark data in a Pythonic way, Iβd love your feedback β on naming, structure, design, anything.
* β [GitHub Repo β SparkDQ](https://github.com/sparkdq-community/sparkdq)
* βοΈ [Medium article β Why I moved beyond PyDeequ](https://medium.com/aws-tip/goodbye-pydeequ-time-to-upgrade-your-data-quality-stack-d86fe9cdc5be)
Thanks for reading!
/r/Python
https://redd.it/1kdgumc
GitHub
GitHub - sparkdq-community/sparkdq: A declarative PySpark framework for row- and aggregate-level data quality validation.
A declarative PySpark framework for row- and aggregate-level data quality validation. - sparkdq-community/sparkdq