Speed improvements in Polars over Pandas

I'm giving a talk on polars in July. It's been pretty fast for us, but I'm curious to hear some examples of improvements other people have seen. I got one process down from over three minutes to around 10 seconds.
Also curious whether people have switched over to using polars instead of pandas or they reserve it for specific use cases.

/r/Python
https://redd.it/1cy9vpt

Thank You PyConUS 2024 !!!

First timer this year, currently at the airport leaving Pittsburgh after 6 days of PyCon...

I've never seen such an intelligent, inclusive, humble, diverse, and inspiring group of human beings. The Python community serves as a beautiful model of what tech culture should strive towards. I could go on and on about how much fun I had, but in short, thanks to all the volunteers, staff, and FOSS developers that have cultivated such an amazing culture.

/r/Python
https://redd.it/1cyceoq

I'm stuck, please help.

Hi guys,

I'm learning Python as a completely new in programming and I'm stuck in VS code. Running python3 on macOS Sonoma, last version VS code.

Look what it does to me:

a = ("Hi ")
b = ("guys")
c = a + b
print(c)

//now if I run it it returns>

>>> print(c)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'c' is not defined
>>> // all runs in macOS terminal seamlessly.

//VS doesnt see all code, it runs just one line. When I sellect all and run, it returns this>

>>> a = ("Hi ")


/r/IPython
https://redd.it/1cynv2h

TPC-H Cloud Benchmarks: Spark, Dask, DuckDB, Polars

I hit publish on [a blogpost](https://docs.coiled.io/blog/tpch.html) last week on running Spark, Dask, DuckDB, and Polars on the TPC-H benchmark across a variety of scales (10 GiB, 100 GiB, 1 TiB, 10 TiB), both locally on a Macbook Pro and on the cloud.  It’s a broad set of configurations.  The results are interesting.

No project wins uniformly.  They all perform differently at different scales: 

* DuckDB and Polars are crazy fast on local machines
* Dask and DuckDB seem to win on cloud and at scale
* Dask ends up being most robust, especially at scale
* DuckDB does shockingly well on large datasets on a single large machine
* Spark performs oddly poorly, despite being the standard choice 😢

Tons of charts in this post to try to make sense of the data.  If folks are curious, here’s the post:

[https://docs.coiled.io/blog/tpch.html](https://docs.coiled.io/blog/tpch.html)

And here's [the code.](https://github.com/coiled/benchmarks/tree/main/tests/tpch) Performance isn’t everything of course.  Each project has its die-hard fans/critics for loads of different reasons. I'd be curious to hear if people want to defend/critique their project of choice.

/r/Python
https://redd.it/1cyqj6c

SSO and Multiple Django Apps

Hi there,

I'm working on distinct interconnected Django projects and I would like to use a single SSO provider for all of them.

Similar to Stack Exchange, where each portal is separate but there is a federated identity across all , I want to implement a similar solution.

The only self-hosted option that seems to fit my use cases is Zitadel. However, before committing to try it, I would like to know if anyone else has already tinkered with this idea and if it's worth investigating further.

Thanks !

/r/django
https://redd.it/1cyu5yv

Need to understand System Design related to Django Rest Framework.

Hi there,

I’ve been working on Django Rest Framework for a couple of months now.
The thing is I’m working on building APIs Models and view etc.
My go to approach is to have model classes (PosgreSQL tables) then use serializer, and make API inside a view set class.
So is this the approach used every where else to make scalable systems? I’m I doing the right thing?

I’m working for a small company and internal application so not many users but I want to know whether the approach I’m using is correct or not.

Are there any blogs and channels specific to Django System design because when I see system design youtube channels they teach from Java perspective and I can’t relate to things like interface (nothing like that in python afaik)?

Thanks!

/r/django
https://redd.it/1cytl2c

I built a pipeline sending my wife and I SMSs twice a week with budgeting advice generated by AI

**What My Project Does**:
I built a pipeline of Dagger modules to send my wife and I SMSs twice a week with actionable financial advice generated by AI based on data from bank accounts regarding our daily spending.

**Details:**

Dagger is an open source programmable CI/CD engine. I built each step in the pipeline as a Dagger method. Dagger spins up ephemeral containers, running everything within its own container. I use GitHub Actions to trigger dagger methods that;

* retrieve data from a source
* filter for new transactions
* Categorizes transactions using a zero shot model, facebook/bart-large-mnli through the HuggingFace API. This process is optimized by sending data in dynamically sized batches asynchronously. 
* Writes the data to a MongoDB database
* Retrieves the data, using Atlas search to aggregate the data by week and categories
* Sends the data to openAI to generate financial advice. In this module, I implement a memory using LangChain. I store this memory in MongoDB to persist the memory between build runs. I designed the database to rewrite the data whenever I receive new data. The memory keeps track of feedback given, enabling the advice to improve based on feedback
* This response is sent via SMS through the TextBelt API

Full Blog: [https://emmanuelsibanda.hashnode.dev/a-dagger-pipeline-sending-weekly-smss-with-financial-advice-generated-by-ai](https://emmanuelsibanda.hashnode.dev/a-dagger-pipeline-sending-weekly-smss-with-financial-advice-generated-by-ai)

Video Demo: [https://youtu.be/S45n89gzH4Y](https://youtu.be/S45n89gzH4Y)

GitHub

/r/Python
https://redd.it/1cyy4hs

My VS went crazy with python. Any ideas?

https://preview.redd.it/gvgbmgv9g62d1.png?width=864&format=png&auto=webp&s=0c04ab2053f11afff0301fa80741927e2241c0a2

MacOS Sonoma, python 3.12.3, VS 1.89.1. I am completely new to programming. Feel free to tell me what I'm doing stupidly if you can give me some advice on how to fix it. Thank you.

/r/IPython
https://redd.it/1cysrjf

D Phi-3 models compared side-by-side.

https://preview.redd.it/8l04pnfhq62d1.png?width=661&format=png&auto=webp&s=7fe616ca8cd7da974070c86b6b47ffab3ab545e5

---------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------------------------------------------------

https://preview.redd.it/hr7fr1uiq62d1.png?width=688&format=png&auto=webp&s=bd3de359bfe4c1ed82d092be92ae38c246bdfda2

---------------------------------------------------------------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------------------------------------------------

https://preview.redd.it/v6k3v39kq62d1.png?width=450&format=png&auto=webp&s=c0abb0e397a498ef7ccfb35b1b1cb598198f66ad



For anyone looking to compare the Phi-3 benchmarks in one place.

Interesting comparisons for: ANLI, Hellaswag, MedQA, TriviaQA, Language understanding, Factual Knowledge and Robustness.

Note: Phi-3 mini model table have labels in different order.

/r/MachineLearning
https://redd.it/1cytxb5

Django Getting Started with Django 2024:Mastering the Django Admin Interface [Part 6/16] Follow my blog on Meduim.com to learn Django.
https://medium.com/@mathur.danduprolu/django-getting-started-with-django-2024-mastering-the-django-admin-interface-part-6-16-e522bfef5a82

/r/djangolearning
https://redd.it/1cz03qn

I made a small Python script that uses NASA'S APOD API to set cool backgrounds on a Windows machine

https://github.com/william7491681/APOD\_Wallpaper\_Script

# What my project does

NASA has a ton of accessible API's, one of which being the APOD (Astronomy Picture Of the Day) API. I made a script to get the last 9 pictures of the day and set them as my Windows 10 background, and then used task scheduler to have the script re-run every day at noon and whenever the computer boots up.

It's fairly hard coded for my setup (specific file paths, 1920x1080 monitor, etc), but it shouldn't be too hard to change if one wanted to.

# Target audience

Anyone who likes space backgrounds

# Comparison

Idk, automod made me put this section

/r/Python
https://redd.it/1cz43o6

Python Anywhere vs Heroku

Hey folks, I’m thinking about deploying a side project that I have been working on. It will store basic user data (name, email, password, etc.). I am debating between Python Anywhere and Heroku. Python Anywhere seems to be giving me more for the basic 5 buck plan vs Heroku, but I’ve heard Heroku is more reliable and easier to scale. Has anyone tried either or both? Any input would be greatly appreciated. Thanks

/r/django
https://redd.it/1cz8yl1

Friday Daily Thread: r/Python Meta and Free-Talk Fridays

# Weekly Thread: Meta Discussions and Free Talk Friday 🎙️

Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!

## How it Works:

1. Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
2. Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
3. News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.

## Guidelines:

All topics should be related to Python or the /r/python community.
Be respectful and follow Reddit's Code of Conduct.

## Example Topics:

1. New Python Release: What do you think about the new features in Python 3.11?
2. Community Events: Any Python meetups or webinars coming up?
3. Learning Resources: Found a great Python tutorial? Share it here!
4. Job Market: How has Python impacted your career?
5. Hot Takes: Got a controversial Python opinion? Let's hear it!
6. Community Ideas: Something you'd like to see us do? tell us.

Let's keep the conversation going. Happy discussing! 🌟

/r/Python
https://redd.it/1cz7jo9

JSX Syntax inside Python files. (Packed)

There was a JSX-style syntax preprocessor for Python called "Packed," which allowed us to write JSX inside Python (*.pyx and *.py) files. It's unclear why they chose *.pyx for the file extension, as it conflicts with the naming of Cythonic file extensions (I have checked their issues). This project might have thrived with sufficient contributions and could have changed the way apps are built. However, the project is now archived on GitHub. The last commit was 5 years ago (LICENSE), and the last development commit was 9 years ago. This repository needs someone to revive it, but I don't have enough experience to take on that task. Even though I don't have enough information, we should start with Rust + Python to build a compiler (aka. template replacer) (this doesn't compile Python but replaces all JSX with a dictionary) and cleaner syntax. Integration with Django (Packed has an example too), Flask, FastAPI, Robyn etc.

We may also need plugins for the language server, I recommend supporting with *.pyh or *.psx (a fork renamed name) the extension file name (Derived from Python + HTML). VSCODE and NVIM Extensions are required to build support for this. The existing modern syntax of native Python

/r/Python
https://redd.it/1czbu57

Session variable is empty when accessed by other Flask routes

I can't figure out why my session variables aren't transferring to other flask routes. I believe I have it set up properly, but I'm having a problem with this code:

from flask import Flask, request, jsonify, redirect, url_for, render_template, session
from flask_session import Session
import requests
import os
import json

app = Flask(__name__)
app.secret_key = os.urandom(24)

# Session setup
SESSION_TYPE = 'filesystem'
app.config.from_object(__name__)
Session(app)

@app.route('/')
def index():
shop_origin = session.get('shop_origin', 'Shop Origin In Session Not Set')
print("index shop_origin is equal to: %s" % shop_origin, flush=True)
if not shop_origin:
session['have_shop_info_in_session'] = False

if 'have_shop_info_in_session' in session:


/r/flask
https://redd.it/1cz77fi

R Introducing SSAMBA: The Self-Supervised Audio Mamba!

Hey Reddit,

Tired of transformers? Is attention really all you need? Meet SSAMBA (Self-Supervised Audio Mamba)! 🐍✨

This attention-free, purely state-space model (SSM)-based, self-supervised marvel doesn’t just hiss—it roars! SSAMBA achieves better or similar performance to its transformer-based counterparts (SSAST) on tasks like speaker identification, keyword spotting, and audio classification. But here's the kicker: it’s much more GPU memory efficient and quicker at inference, especially with longer audio lengths.

Curious? Check out the full paper here: SSAMBA on arXiv

Thanks for tuning in!

/r/MachineLearning
https://redd.it/1cz1yoa

D Detecting Objects of Same Shape but Different Colors

I'm struggling with detecting objects that have the same shape but different colors, with no other distinguishing features. When there are distinguishable patterns, CNN-based architectures like YOLO work wonders and achieve high accuracy. However, I need a method that can accurately classify objects based purely on color.

My current challenge is that these objects are not separable when I try to segment them by color in RGB space. Does anyone have suggestions or methods that achieve good accuracy in determining object classes by color?

I've included an image below for reference. Any help would be greatly appreciated!

https://preview.redd.it/83c6e7dbbb2d1.png?width=793&format=png&auto=webp&s=532c7cffcbaea96eb48d374e073bd49d5f029212



/r/MachineLearning
https://redd.it/1czdmty

D Paperswithcode relevant?

I feel like paperswithcode became less relevant for tracking progress in ML in general for me.

But it’s hard to say, in my field (tabular ML/DL) there are not many established academic benchmarks (no need for something like papers with code yet)

In NLP and foundation model space leaderboards in hf spaces became a thing (mostly in NLP).

Overall, paperswithcode just feels less maintained and less useful.

Do you use paperswithcode often? What do you use it for? What’s your field where it is useful?

/r/MachineLearning
https://redd.it/1cz1t4x

Tutorial: Sure Fire Streamlit - A Sidebar And Multi-visual Layout You'll Love

Python Streamlit is a fast-growing framework for creating interactive maps using Python code.

Interactive maps are better than static maps as they provide granular configuration opportunities for your users.

In this tutorial:

1. A sidebar with two dropdown menus allowing the user to select both a year and a colour scheme
2. A global choropleth map showing results by country (for a chosen year)
3. A horizontal bar chart showing the Top 10 countries by value (for a chosen year)
4. A set of range values showing high, average, and low (for a chosen year)

Link to tutorial HERE

/r/Python
https://redd.it/1czkfo7

PythonAnywhere service degradation

Hello all,

When I started learning Django 10 years ago, PythonAnywhere helped me take off with easy set-up and operations. I have never had any issue with them over almost all of that time frame.

For over a year now, they have become unreliable, with regular non-scheduled outages in the middle of the day, lasting anywhere from 5 to 30 minutes. I work with several businesses for which a downtime in a critical moment can have a huge business impact.

They have communicated ZERO TIMES on these issues. They don't have a status page. I can understand having technical issues, but I cannot forgive the fact that they seem to be trying to hide these issues from their users.

When asked, they reply saying they are having hardware issues. But again, there is no proactive communications, only replies when asked.

They were acquired by Anaconda in June of 2022. One has to wonder if this triggered a hardware migration that is causing these issues.

I am writing here to you all to know if you have had the same experience. I have multiple accounts and multiple sites running, all on the EU servers of PythonAnywhere, so I know it's not my account, but I'd still like

/r/django
https://redd.it/1czkkpn