D Efficient way to store large datasets

I’m collecting trajectories for imitation learning (RL) and each trajectory is about 1500 time steps long, consists of 4 image streams of about 600x600 pixels.
Obviously, the dataset size grows extremely quickly with the number of trajectories.

What are some good libraries for efficiently (in terms of disk space) storing such data? I tried h5py with level 9 gzip compression but the files are still way too large. Is there a better alternative?

Saving and loading times do not really matter.

Most resources online are aimed at efficiently loading large datasets or handling them in memory which is not relevant for my question.

I already use uint8 as datatype for the rgb streams.

/r/MachineLearning
https://redd.it/1f951p8

Yet another 12 factor configuration helper : python-direnv

Hi Python! I've developed another app to manage environment variables.

# What My Project Does

This allows you to `load_direnv()` your environment variables like `load_dotenv()`, once you allow them through `direnv`, but unlocks the full power of the bash shell instead of being limited to basic key-value pairs. Contributions welcome (the package is not yet on Pypi.org, working on it) !

# Target Audience

If you are developing using the 12 factor app principles and are a user or direnv, Nix, or Guix you might want to check out this package I just wrote : https://github.com/nicolas-graves/python-direnv

# Comparison

This package is actually quite different to other similar projects. It doesn't use a parser but loads environment variables based on shell environment changes. This allows for instance full environment configuration using Nix or Guix for instance, thus providing a way to load a complete environment from a simple `__file__` variable, in any Python project. I have another project on the way that needed this for real-life application ;)



/r/Python
https://redd.it/1f9jfgv

simple project suggestion

i don't know if this is the place to ask this but can you suggest a github repo of a simple (not necessarily) django project that inside it users can enter their information inside a form and then the website has a section that displays all these users somewhere, anything remotely close to this is good enough i'll change it up a little bit, its a project for my teacher but i don't have a lot of time to do it all myself, just need a simple local project, Thank you in advance

/r/django
https://redd.it/1f9j943

I've been tracking Python, Django, NumPy and several other frameworks in job listings this year

Hi all, I built a website to track programing languages/skills/frameworks in jobs.
Perhaps unsurprsingly Python is by far the biggest category in software engineering:

I'm tracking many other Python frameworks and libraries as well:

[Django](https://job.zip/trend/django)
Scikit-learn
[NumPy](https://job.zip/trend/numpy)
Flask
[FastAPI](https://job.zip/trend/fastapi)
Keras
[MicroPython](https://job.zip/trend/micropython)
Ruff

I hope this is of some use of you, if there's another framework you'd like me track, please let me know!

Also there's a Python component as well, I use Python to identify trends in my dataset. Every month I load up 10 million new jobs and compare them with the months before to identify new types of jobs to add to the site.

/r/Python
https://redd.it/1f9l9zt

Python Libraries to Extract Table from PDF

Here's a blog with a tutorial using multiple Python libraries to extract tables: https://unstract.com/blog/extract-tables-from-pdf-python/

Video tutorial: https://www.youtube.com/live/YfW5vVwgbyo?t=2799s

/r/Python
https://redd.it/1f9fv7y

nb2dash: Convert Jupyter notebooks to interactive dashboards with WASM

Hey r/ipython! Today, I'm lauching nb2dash, a tool to convert Jupyter notebooks into dashboards, and I'd love to get your feedback.

https://nb2dash.ploomberapp.io

You can see a sample dashboard here: https://nb2dash.ploomberapp.io/notebook/bb8086c0

https://reddit.com/link/1f9rajw/video/hr40j3dht0nd1/player



# What it does

Upload your Jupyter notebook (.ipynb file)
Get back a link to an interactive dashboard version of your notebook

# Target Audience

Data practitioners who want to easily share an interactive analysis, machine learning model or any other interactive app.

# Key features

Free to use: No sign-up required
Easy: Just drag and drop your notebook file
Quick: Processing usually takes 1-2 minutes
Shareable: Get a link you can send to anyone

# Tech stack

Flask for the web app (receive uploads and process notebook)
NGINX for rendering processed notebooks
Voici for converting notebooks into standalone WASM pages
Redis and rq for task queueing

# Comparison

When people want to share a notebook, they often convert it into HTML or PDF. However, this is hinders interactivity. Alternatively, you might use Voila to deploy it as a web app but that requires paying to a hosting provider. Voici uses WASM, meaning your notebook is a static site and all compute happens locally, reducing cost by a huge margin. Note that WASM and Voici are still early technologies

/r/IPython
https://redd.it/1f9rajw

One more Question

in jupyter notebook when i try to save a photo for object detection it shows this error:

<string>:1: SyntaxWarning: invalid escape sequence '\w'

even when i try to set variables

Does anyone know the reason?

/r/IPython
https://redd.it/1f9lwqd

PEP695 Migration Tool

Now that PEP 695 is a little over 2 years old and it has been almost a year since it was officially introduced in python 3.12, the new type parameter syntax will slowly become the standard way of defining TypeVars, ParamSpecs and TypeVarTuples as well as creating Type aliases with the type keyword. After I spent some time using python the last couple of weeks, I realized that adapting "old" projects of mine to the new syntax really takes more time than I'm willing to invest, especially as some have a pretty large codebase.

# What My Project Does

This is why the past few days I spent time building pep695, a tool that automatically rewrites code to conform to PEP 695 syntax for you.

Install pep695 using pip:
pip install pep695

Format code using pep695 format <path1> <path2> <pathN> ... [-p | --parallel] or just check for errors using pep695 check <path1> <path2> <pathN>. I just published this project today which is why I'm still looking for people to test this project on their own codebase, doesn't matter if it's small or large, any feedback is appreciated!

More info, including examples of code that pep695 can rewrite for you on the github repo:
https://github.com/yowoda/pep695

# Target

/r/Python
https://redd.it/1f9uai8

Any tips for courses with a full application that aren't on video?

I recently discovered William Vincent's book which is a step by step application, starting with Docker, I'm looking for things like that because it's a way to learn DevOps as well. However, most of them are Udemy courses, and video is more complex!

Note: The only problem with the book is that it's from 2019, but I'll adapt to the time.

/r/django
https://redd.it/1f9r2h6

Interactive Mapping with ipyopenlayers
https://blog.jupyter.org/interactive-mapping-with-ipyopenlayers-2b8bc93bb6c4

/r/IPython
https://redd.it/1f9r0hs

Hey Django community, I heard a lot of you use PostgreSQL! If you do, please take a moment and fill out the 2024 State of PostgreSQL Survey. It's created for the community, by the community; the more responses, the more accurate and helpful the results. Any questions or comments? Let's talk!
https://form.typeform.com/to/wtUFIZwA

/r/django
https://redd.it/1fa298d

[P] This week, I implemented the paper, "Pay Attention to MLPs", in Tinygrad! :D

To experiment with more interesting model architectures, I implemented gMLP in Tinygrad!

If anyone wants to give some feedback, it will be welcomed.

* \[Repository\]: [https://github.com/EthanBnntt/tinygrad-gmlp](https://github.com/EthanBnntt/tinygrad-gmlp)
* \[Installation\]: `pip install gmlp_tinygrad`
* \[Original Paper\]: [https://doi.org/10.48550/ARXIV.2105.08050](https://doi.org/10.48550/ARXIV.2105.08050)

[A diagram showing the gMLP architecture](https://preview.redd.it/3s58nla804nd1.png?width=330&format=png&auto=webp&s=3f00f8364e9e0ac00da13ce28199d59330ebaef3)

/r/MachineLearning
https://redd.it/1fa5kop

Friday Daily Thread: r/Python Meta and Free-Talk Fridays

# Weekly Thread: Meta Discussions and Free Talk Friday 🎙️

Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!

## How it Works:

1. Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
2. Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
3. News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.

## Guidelines:

All topics should be related to Python or the /r/python community.
Be respectful and follow Reddit's Code of Conduct.

## Example Topics:

1. New Python Release: What do you think about the new features in Python 3.11?
2. Community Events: Any Python meetups or webinars coming up?
3. Learning Resources: Found a great Python tutorial? Share it here!
4. Job Market: How has Python impacted your career?
5. Hot Takes: Got a controversial Python opinion? Let's hear it!
6. Community Ideas: Something you'd like to see us do? tell us.

Let's keep the conversation going. Happy discussing! 🌟

/r/Python
https://redd.it/1fa1gk2

Best platform to host production django apps

Hey everyone,


We are currently using Heroku to host our application and looking for alternatives.


We are using railway for our staging environment but don't have much confidence yet. We are also considering Elastic Beanstalk too.


Any comments are much appreciated.


Thanks!

/r/django
https://redd.it/1fa6p1q

What is the purpose or a use-case of a Django admin?

I always ever worked with FastAPI, Flask and ExpressJS for creating APIs for web projects, and now I'm trying out Django.

I followed a tutorial to try setting up and re-doing API's I've built with the other frameworks, and I found myself with a dozen tables related to Django, popping up in my database.

I went to the /admin route and saw that I could login.

What is the purpose of having this kind of user management for accessing the database? I never had to use anything like that with the other frameworks.

Is that suited for some kind of work environment where you want to give employees certain rights, like some can't add new tables and others can? Is that the scope of this admin feature?

If so, I guess I can skip it for my personal projects?

/r/django
https://redd.it/1f9wqog

Django News - Django security releases issued: 5.1.1, 5.0.9, and 4.2.16
https://django-news.com/issues/249

/r/django
https://redd.it/1fac6s4

How To Perform Unit Testing in Flask
https://www.digitalocean.com/community/tutorials/unit-test-in-flask

/r/flask
https://redd.it/1fa9mno

Need Help with Offline License System and Periodic Server Checks in Flask App

Hi everyone,

I’ve developed a local Flask app that uses SQLite and a Tkinter interface to run locally. I recently added a license system but think it could be improved, especially when it comes to managing license expiration and periodic checks.
Here’s how my current system works:

I have an admin dashboard (hosted on my server) where I generate and manage licenses.
Each license has a start date, end date, and activation_status (default = 0), and I generate an activation code for the user.
The user enters a username, password, and the activation code in the app, and the app sends a request to my server to verify the license and get the relevant dates and status.
If the license is valid, I store a hashed combination of the username, password, and hardware ID (HWID) in a file called license on the user's device. The start and end dates are also saved in a local SQLite database on the user’s device.
The app is mostly designed to work offline, but the activation process requires internet access.
I have a login_required decorator that checks the license status before allowing access to any protected route. If the license has

/r/flask
https://redd.it/1fa9g27

Desperate and Willing to pay for ALL-AUTH solution.

I have a problem. When a user signs up to my site using their email, then later logs in with email provider (eg. Google), all auth treats this like a new user and objects to them having the same email. How can I make it understand that it's the same user and give them access?

I've looked around for far too long and found nothing that solves this. I'm willing to pay for a solution, and I will make the answer public for all to benefit.

P.s apologies if I'm not allowed to solicit but after all, I am desperate.

/r/django
https://redd.it/1fac0m6

Pass JSON to Templates using Include

How do I pass Json to Template from main Template where the variable gets defined not by view.py but main_template.html

template

main template

In the above example, I want to define the icon here itself in the main_template.html rather than the view.

/r/django
https://redd.it/1faee90

Best approach to allowing only the staff users to have the access

I have two snippets here and which one is the best approach/practice for only allowing staff users have the access to certain data. In my case accessing user profile. Any suggestion will be greatly appreciated. Thank you very much.

example 1:

@api_view(['GET'])
@authentication_classes[TokenAutentication]
def get_profile_view(request):
if request.user.is_staff:
profiles = Profile.objects.all()
serializer = ProfileSerializer(profiles, many=True)
return Response(serializer.data, status=status.HTTP_200_OK)
return Response({'error': 'Not allowed'}, status=status.HTTP_400_BAD_REQUEST)


example 2:

@api_view(['GET'])
@permission_classes([IsAdminUser])
@authentication_classes[TokenAutentication]
def get_profile_view(request):
profiles = Profile.objects.all()
serializer = ProfileSerializer(profiles, many=True)
return Response(serializer.data, status=status.HTTP_200_OK)

/r/django
https://redd.it/1fajw75