P New tokenization method improves LLM performance & context-length by 25%+

I've been working on this new tokenization method to optimally represent text with fewer tokens than current methods. It's MIT licensed.

Code at Github.

Test it out.

The general-english-65535 vocabulary, and the code versions are already complete. The general-english-32000 should be finished within a few hours. Then I'm going test a non-greedy version which should do even better.

Intro from README:

tokenmonster is a novel approach to tokenization with broad-ranging use potential, but its primary motivation is to increase the inference speed and context-length of large language models by choosing better tokens. By selecting more optimal tokens, text can be represented with 20-30% less tokens compared to other modern tokenizing methods, increasing the speed of inference, training and the length of text by 20-30%. The code-optimized tokenizers do even better, see it for yourself.

I also believe that tokenmonster vocabularies will improve the comprehension of Large Language Models. For more details see How and Why.

## Features

Longer text generation at faster speed
Determines the optimal token combination for a greedy tokenizer (non-greedy support coming)
Successfully identifies common phrases and figures of speech
Works with all languages and formats, even binary
Quickly skims over HTML tags, sequential spaces, tabs, etc. without wasting context
Does not require normalization

/r/MachineLearning
https://redd.it/13gdfw0

Best SAAS stack and template for building quick MVPs?

I want to learn a new stack. All I need is a template with authentication (firebase ideally), payment processor (like stripe), and database. Any God-tier tutorial, repo, or stack you know of? I really like Flask but I am wondering if other web frameworks allow you to do things quicker and plug things in quicker. I am hearing a lot about Next.js but yeah… not sure what to go for. All I want is a template that I can use multiple times to spam MVP after MVP. A one page presenting the product, login / logout system, some tokens given to the user and an API call to midjourney or ChatGPT that provides the thing the customer is paying for.

/r/flask
https://redd.it/13gm2ak

Google Sign in AllAuth

Hey yall So i configured google and django allauth and I have successfully gotten to the sign in page for google but once I click on a gmail account I want to log in as I get a

This site can’t be reached localhost refused to connect.

Idk if its because the site was connected to the social account in the admin panel or what? I would connect the site and google but it doesn't pop up only groups, users, social accounts. any ideas?

/r/django
https://redd.it/13go7a4

[R] Large Language Models trained on code reason better, even on benchmarks that have nothing to do with code
https://arxiv.org/abs/2210.07128

/r/MachineLearning
https://redd.it/13gk5da

Architecting Django allauth with Multi-Tenant Schemas

Hello everyone,

I am currently working on a Django project that implements a multi-tenancy architecture using the Django Tenants library. The purpose is to provide data isolation between different tenants in the system, each having its own set of schemas.

However, I am facing a critical architectural decision concerning the integration of Django allauth within this multi-tenant system. The issue arises from the fact that Django allauth inherently depends on the User model.

The challenge here is how to implement Django allauth in such a system. If I make Django allauth tenant-specific, each tenant would have its own registration and login pages, and users would need to register separately for each tenant they want to access. This also means that the same email could be used to create different accounts on different tenants, and there would be no option to share authentication or user information between tenants.

On the other hand, if I make Django allauth shared (not tenant-specific), then it becomes a challenge to manage user-tenant relationships and ensure data isolation since the User model would need to be accessible across all schemas.

I am seeking advice on the best way to implement Django allauth in a multi-tenant Django application. Has anyone faced a

/r/django
https://redd.it/13gp2g6

R Enhancing Language Model Performance through Context Preservation: A Novel Approach Utilizing Internal State Symbols

Abstract

In the domain of conversational AI, the quality of output generated by large language models (LLMs) is of significant importance. This paper explores a novel approach to provide context and improve the quality of LLM responses in conversational settings. The proposed technique involves instructing the LLM to output a series of symbols representing its internal state at the end of its last response, which encapsulates the context and process that led to that answer. When provided with symbols from the user's previous conversation, the LLM can restore its internal state before reviewing the newly-received message, thus enabling it to understand the context of the entire conversation better. Although a quantitative analysis has not been conducted, subjective evaluations reveal evident improvements in the quality of responses, drawing parallels with human conversation dynamics.

1. Introduction

As artificial intelligence (AI) technologies continue to advance, LLMs have emerged as essential tools in the development of conversational AI systems. While these models are capable of producing impressive results, their performance can be further enhanced by providing additional context in conversational settings. This paper presents a novel approach to augment LLMs with context by leveraging their internal state representations, and discusses the potential benefits of this method in

/r/MachineLearning
https://redd.it/13gsqkl

OC Python Cheat Sheet in PCB design

I have created a Python Cheat Sheet and would like to collect feedback from you. If you want to have a cheat sheet as a real PCB, just take a look. I would be very happy if you support my project. I'm happy to answer any questions :)

https://www.westartfactory.com/#python

Python Cheat Sheet in PCB design

​

Python cheat sheet from a real circuit board - Front

Python cheat sheet from a real circuit board - Back

/r/Python
https://redd.it/13gpfiu

Sqlalchemy help

I’m currently teaching myself sqlalchemy and I’ve made a website where users can log on and create a video game library using the igdb api.
I’ve created a database that stores all the “user” data, and I have a separate database to store the “game” data. At this point all the users share the same “game” database. I’d like each user to only be able to access their own game library but I can’t figure out how to do it or what to search to learn how to do it. Can anyone point me in the right direction?

/r/flask
https://redd.it/13geo76

Nonetype error in filepath = os.path.join(app.config'UPLOAD_FOLDER', filename)

I am making an app with Flask and this is my app.py code

from flask import Flask, rendertemplate, request, redirect, urlfor
from werkzeug.utils import securefilename
from backgroundsubtractor import BackgroundSubtractor
import cv2
import webbrowser
import numpy as np
import base64
import os

app = Flask(name)

# Set up the upload folder
UPLOADFOLDER = 'uploads'
app.config['UPLOADFOLDER'] = os.path.join(os.path.dirname(os.path.abspath(file)), UPLOADFOLDER)

# Set up the static folder
STATICFOLDER = 'static'
app.config'STATIC_FOLDER' = os.path.join(os.path.dirname(os.path.abspath(file)), STATICFOLDER)



@app.route('/')
def index():
return rendertemplate('index.html')

@app.route('/upload', methods='GET', 'POST')

def upload():
if request.method == 'POST':
if

/r/flask
https://redd.it/13h26r6

Flask Error message

Hi everyone , I have a project to create a website and link it to my trained classification model (Vgg16)

so I started to use Flask and I created two files:

the first one

from flask import Flask, render_template, request

from tensorflow.keras.models import load_model

from tensorflow.keras.preprocessing import image

import numpy as np

app = Flask(__name__)

model = load_model('modelV2_VGG16_8to2_ep20.h5') # load your trained model here

u/app.route('/', methods=['GET', 'POST'\])

def index():

result = None # initialize the result variable

if request.method == 'POST':

file = request.files['image'\] # get the uploaded file

img = image.load_img(file, target_size=(224, 224)) # load the image

img = image.img_to_array(img) # convert to array

img = np.expand_dims(img, axis=0) # add batch dimension

img = img/255.0 # normalize pixel values

​

prediction = model.predict(img) # make the prediction

result = np.argmax(prediction) # get the index of the predicted class

return render_template('mytemplates/index.html', result=result)

if __name__ == '__main__':

print(app.template_folder)

app.run(debug=True)

​

and the second one is
<!DOCTYPE html>

<html>

<head>

<title>Image Classification</title>

</head>

<body>

<h1>Image Classification</h1>

{% if result %}

<p>The predicted class is {{ result }}</p>

{% else %}

<form method="POST" enctype="multipart/form-data">

<input type="file" name="image">

<input type="submit" value="Upload">

</form>

{% endif %}

</body>

</html>

&#x200B;

However, after I run the flask run on command,I got this message "Internal Server Error The server encountered an internal error and was unable to complete your request.

/r/flask
https://redd.it/13gl5t5

{if request.user.is_authenticated} not work on html

I have a weird issue. For some reason `{if request.user.is_authenticated}` not work on navbar on index.html, but it work on other html pages. It always just shows Home and Login. Any help will be greatly appreciated. Thank you.

<nav>
<div class="brand-name">
<a href="/">BrandName</a>
</div>
<ul>
<li>
<a href="{% url 'articles:home' %}">Home</a>
</li>
{% if request.user.is_authenticated %}
<li>
<a href="{% url 'articles:article-create' %}">Create article</a>
</li>


/r/djangolearning
https://redd.it/13gmsgb

Sunday Daily Thread: What's everyone working on this week?

Tell /r/python what you're working on this week! You can be bragging, grousing, sharing your passion, or explaining your pain. Talk about your current project or your pet project; whatever you want to share.

/r/Python
https://redd.it/13gwu0m

Which Docker tutorial to use?

I've been trying to deploy my Django app for a while, and because I don't have any experience with docker and trying to deploy ASAP, I tried to follow tutorials and here's what I found:

reddit post \- This reddit post linked an article, which isn't that old, plus is using exactly what I need: Django (DRF in my case), Redis, Celery and Postgres. But it's saying that

>server and worker share the same Dockerfile

So that means that I'm hosting both Celery and Django on the same server. It's not a big problem, but I've already hosted celery on Railway and Django on Render, they work fine, the problem is that the article is not showing the deployment process and is using NGINX. Do I deploy the same way I do usually on Render or do I have to follow the article and eventually encounter problems with deployment?

youtube video \- Many people say good stuff about this one, but I think it's outdated. It was released in 2020 (correct me if I'm wrong).

and this youtube video \- He's also explaining the same things simply, but I'd say the same thing about it. Outdated???

/r/django
https://redd.it/13hb9i4

Color Django shell by development environment
https://django.wtf/blog/django-shell-colors-by-development-environment/

/r/django
https://redd.it/13hczrz

Guide for a complete beginner - Tips, tricks and tutorials

Hi everyone, I’m fresh out of CS50 and looking to expand in BE and DevOps. Is there any good course or tutorial that can help me to build from ground up, already know some Python, C and JS.

Would love the help!

/r/django
https://redd.it/13hi1rc

Real Multithreading is Coming to Python - Learn How You Can Use It Now
https://betterprogramming.pub/real-multithreading-is-coming-to-python-learn-how-you-can-use-it-now-90dd7fb81bdf

/r/Python
https://redd.it/13heek6

Trouble Deploying Django to Railway.app

I am trying to get Railway to work as I like their features. I had the Railway account for a few days before I realized that I could get to learn it way quicker if I upgraded my account to the Developer one. My intention is to deploy the Django project's development environment (app plus database) onto Railway. My Django project is API only and I need the frontend dev environment (which I will host in a separate environment on Railway) to have access to the backend's API endpoints online.

When I deploy the development environment now, after referencing https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/Deployment#getting\_your\_website\_ready\_to\_publish, I get the following deploy log "Watching for file changes with StatReloader

"Performing system checks... System check identified no issues (0 silenced). May 14, 2023 - 16:19:46 Django version 4.2.1, using settings 'project.settings.local' Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C."

However, I get "Application failed to respond" when I try to access the project using the generated domain.

Here are the rest of my configurations and logs: build log: https://pastebin.com/pKw65eGe (I hid some data by replacing it with "xxx") base.py (base settings) https://pastebin.com/5fTkXTGB local.py (dev environment) https://pastebin.com/cfLPiDzm requirements.txt https://pastebin.com/T35yEriz (I'll clean it up once I figure out which libraries I do

/r/django
https://redd.it/13hj7bl

Monday Daily Thread: Project ideas!

Comment any project ideas beginner or advanced in this thread for others to give a try! If you complete one make sure to reply to the comment with how you found it and attach some source code! If you're looking for project ideas, you might be interested in checking out Al Sweigart's, "The Big Book of Small Python Projects" which provides a list of projects and the code to make them work.

/r/Python
https://redd.it/13hrkfo

Using NextJS for templates a sensible choice?

Hey! I have a personal project I’ve been working on for over a year as a solo dev. I am experienced with Python, and have been using Django for my project. It has an API for mobile, and I am building a dashboard.

I gave templates/htmx/alpine a try, and found limitations pretty fast. These could be due to my inexperience with these, but I find making reusable components (how one would in say, react) to be difficult. And trying to have an edit form modal that shows a toast afterwards seemed like it was a lot to figure out with htmx/alpine (I was able to get them individually, but not both in order).

I work professionally with Vue, but all the component libraries for React seem great and make developing beautiful UIs and reusable components quite easy. I could be wrong, but I think doing a UI with something like Mantine could be a similar amount of work as Django templates - but far more flexible and easy to get nice results.

The benefits that NextJS provides seem really interesting, but the one thing I wonder about is if it is a bad approach to have Django being a service that next

/r/django
https://redd.it/13hu98j

[R] Bark: Real-time Open-Source Text-to-Audio Rivaling ElevenLabs
https://neocadia.com/updates/bark-open-source-tts-rivals-eleven-labs/

/r/MachineLearning
https://redd.it/13hi7es

Understanding Python Decorators and How to Use Them Effectively
https://soshace.com/understanding-python-decorators-and-how-to-use-them-effectively/

/r/Python
https://redd.it/13hchou