Lessons Learned While Trying to Scrape Google Search Results With Python

I’ve been experimenting with Python web scraping recently, and one of the toughest challenges so far has been scraping Google search results reliably. I expected it to be as simple as requests + BeautifulSoup, but it wasn’t.

Here’s what I’ve learned from trial and error:

🔹 Google is quick to detect scraping attempts.
Even with random headers, delays, and proxies, it doesn’t take long before CAPTCHAs or temporary blocks pop up. At one point, I could barely scrape 2–3 pages before getting cut off.

🔹 Pagination isn’t consistent.
It’s not just &start=10 for every page—sometimes the results shift or display fewer items than what’s in the browser. You need to account for unexpected page behavior.

🔹 JavaScript rendering is almost a requirement now.
Some SERPs don’t fully load without JS enabled. Static requests often return stripped-down or incomplete results, which makes BeautifulSoup parsing unreliable unless you use something like Playwright or an API that supports rendering.

🔹 Data cleaning matters.
Google adds a ton of extra formatting, tracking parameters, and “People Also Ask” blocks. I ended up writing extra functions just to extract clean titles, links, and snippets.

I know scraping Google is a grey area, but it’s a common data engineering

/r/Python
https://redd.it/1md4zmu

large django project experiencing 502

My project has been experiencing 502 recently. I am running on gunicon with nginx. I don't really want to increase the timeout unless I have too. I have several models with object counts into 400k and another in 2 million objects. The 502 only occurs on PATCH requests. I suspect that the number of objects is causing the issue. What are some possible solutions I should look into?

/r/django
https://redd.it/1mddk6d

`tokenize`: a tip and a trap

[`tokenize`](https://docs.python.org/3/library/tokenize.html) from the standard library is not often useful, but I had the pleasure of using it in a recent project.

Try `python -m tokenize <some-short-program>`, or `python -m tokenize` to experiment at the command line.

-----

The tip is this: `tokenize.generate_tokens` expects [a readline function that spits out lines as strings when called repeatedly](https://docs.python.org/3/library/tokenize.html#tokenize.generate_tokens), so if you want to mock calls to it, you need something like this:

lines = s.splitlines()
return tokenize.generate_tokens(iter(lines).__next__)

(Use `tokenize.tokenize` if you always have strings.)

----

The trap: there was a breaking change in the tokenizer between Python 3.11 and Python 3.12 because of the formalization of the grammar for f-strings from [PEP 701](https://docs.python.org/3/whatsnew/3.12.html#pep-701-syntactic-formalization-of-f-strings).

$ echo 'a = f" {h:{w}} "' | python3.11 -m tokenize
1,0-1,1: NAME 'a'
1,2-1,3: OP '='

/r/Python
https://redd.it/1mdag10

CLI Tool For Quickly Navigating Your File System (Arch Linux)

So i just made and uploaded my first package to the aur, the source code is availble at https://github.com/BravestCheetah/DirLink .

The Idea

So as i am an arch user and is obsessed with clean folder structure, so my coding projects are quite deep in my file system, i looked for some type of macro or tool to store paths to quickly access them later so i dont have to type out " cd /mnt/nvme0/programming/python/DirLinkAUR/dirlink" all the time when coding (thats an example path). Sadly i found nothing and decided to develop it myself.

Problems I Encountered

I encountered one big problem, my first idea was to save paths and then with a single command it would automatically cd into that directory, but i realised quite quickly i couldnt run a cd command in the users active command prompt, so i kinda went around it, by utilizing pyperclip i managed to copy the command to the users clipboard instead of automatically running the command, even though the user now has to do one more step it turned out great and it is still a REALLY useful tool, at least for me.

What My Project Does

I resulted in a cli tool which has the "dirlink" command with

/r/Python
https://redd.it/1mdfeh3

Thursday Daily Thread: Python Careers, Courses, and Furthering Education!

# Weekly Thread: Professional Use, Jobs, and Education 🏢

Welcome to this week's discussion on Python in the professional world! This is your spot to talk about job hunting, career growth, and educational resources in Python. Please note, this thread is not for recruitment.

---

## How it Works:

1. Career Talk: Discuss using Python in your job, or the job market for Python roles.
2. Education Q&A: Ask or answer questions about Python courses, certifications, and educational resources.
3. Workplace Chat: Share your experiences, challenges, or success stories about using Python professionally.

---

## Guidelines:

- This thread is not for recruitment. For job postings, please see r/PythonJobs or the recruitment thread in the sidebar.
- Keep discussions relevant to Python in the professional and educational context.

---

## Example Topics:

1. Career Paths: What kinds of roles are out there for Python developers?
2. Certifications: Are Python certifications worth it?
3. Course Recommendations: Any good advanced Python courses to recommend?
4. Workplace Tools: What Python libraries are indispensable in your professional work?
5. Interview Tips: What types of Python questions are commonly asked in interviews?

---

Let's help each other grow in our careers and education. Happy discussing! 🌟

/r/Python
https://redd.it/1mdmo51

Granian 2.5 is out

Granian – the Rust HTTP server for Python applications – 2.5 was just released.

Main highlights from this release are:

support for listening on Unix Domain Sockets
memory limiter for workers

Full release details: https://github.com/emmett-framework/granian/releases/tag/v2.5.0
Project repo: https://github.com/emmett-framework/granian
PyPi: https://pypi.org/p/granian

/r/Python
https://redd.it/1mdi14s

Step-by-step guide to deploy your FastAPI app using Railway, Dokku on a VPS, or AWS EC2 — with real

[https://fastlaunchapi.dev/blog/how-to-deploy-fastapi-app/](https://fastlaunchapi.dev/blog/how-to-deploy-fastapi-app/)

# How to Deploy a FastAPI App (Railway, Dokku, AWS EC2)

Once you’ve finished building your FastAPI app and tested it locally, the next big step is getting it online so others can use it. Deployment can seem a little overwhelming at first, especially if you're deciding between different hosting options, but it doesn’t have to be.

In this guide, I’ll walk you through how to deploy a FastAPI application using three different platforms. Each option suits a slightly different use case, whether you're experimenting with a personal project or deploying something more production-ready.

We’ll cover:

* **Railway**, for quick and easy deployments with minimal setup
* **Dokku**, a self-hosted solution that gives you more control while keeping things simple
* **AWS EC2**, for when you need full control over your server environment

/r/Python
https://redd.it/1mdkmvd

How to properly render select2 widget in a modal window?

Hi, i have a Book model that has a authors field which is a ManyToMany field to Author model.

I'm using django-autocomplete-light to render a select2 widget in my templates that will allow me to select more than one authors when creating new books. (Using ModelSelect2Multiple)

So the field renders OK in a regular html page. But when i try to render the same exact form in a DaisyUI modal window, the dropdown menu that it should open will be opened in the back of the modal window (like i can see it is being displayed behind the modal window).

Here is my form:

class BookForm(forms.ModelForm):    
class Meta:
        model = Book
        fields = (
            'title',
            'authors',
        )

        widgets = {
            'authors': autocomplete.ModelSelect2Multiple(


/r/djangolearning
https://redd.it/1md535t

Is it a sin to serve just the password reset from Django directly?

Right. Ive been avoiding asking thinking I will eventually fix this but no dice. It’s nearly midnight(wrecked), I’m now in bed(fiance will kill me if I wake her), spiritually defeated(it's temporary), and here we are.

I’m building a personal project with a decoupled setup: DRF as the backend and a super minimal HTML and JS(it's as vanilla as you can get) frontend... essentially a glorified test harness. Nothing fancy, just enough to click buttons and cry.

Here’s the problem: I can't get the password reset form to show up properly after clicking the reset link that gets emailed. The link to send the reset email works fine when I use Django’s built-in templates on port 8000. But when I try to handle it through my frontend setup? Nada. Just silence and broken dreams(empty index file).

So now I’m wondering would it really be that bad if I just let Django serve this one thing directly? Let it have its moment in the spotlight with the password reset form while the rest of the app sticks to the decoupled API and JS plan?

Is this a common workaround? A sign of weakness? A pact with the devil? Just looking for some wisdom (or permission) from

/r/django
https://redd.it/1mdla52

Proxy for using LSP in a Docker container

I just solved a specific problem: handling the LSP inside a Docker container without requiring the libraries to be installed on the host. This was focused in Python using Pyright and Ruff, but can be extensible to another language.

https://github.com/richardhapb/lsproxy

/r/Python
https://redd.it/1mdq489

Looking for advice on a crash course

Hey everyone, I am a hobbyist that hasn't tinkered with django much for about 5+ years. I have previously only made very simple apps like a building directory in the past.

I am looking to create a new application and would like to re-familiarize myself with django and am hoping someone may be able to recommend a course covering the current version. I'm willing to pay for a course (I previously used Jose Portillo's course on udemy).

/r/django
https://redd.it/1mdr8qe

Real‑world ML course with personalized gamified challenges—feedback wanted on structure & format! 🎓

Hi everyone — I've been lurking these subreddits for years and finally wrapped up a course that’s very much inspired by what I’ve learned from this community.

I previously created a Udemy course—but in retrospect it felt too one‑size‑fits‑all and lacked engagement. Feedback showed that it wasn’t personalized enough, and students tends to drop off without reaching applied concepts.

So this iteration (on [Uphop.ai](https://www.uphop.ai/app/c/02d00637-0d71-40b3-af0b-ace55c2b6378?code=e12cd)) has been designed from scratch to tackle those issues:

* **Practice games at the end of every unit**, not just quiz questions—scenario-based immersive tasks. It’s true gamification applied to learning design, which literatures show can really boost engagement and performance when tailored to individual user preferences.
* **Hyper‑personalized experience**: learners get to pick challenges or paths that suit their goals, pacing, and interests, instead of being forced into a rigid progression.
* **Core modules**: Supervised/Unsupervised Learning, NLP, Deep Learning, AI ethics, Cloud deployments.

I’d love your honest feedback on:

1. Does the idea of challenge-based “games” at the end of modules sound motivating to you?
2. Would a hyper-personalized track (choose‑your‑own‑challenge or order) make a difference in how you'd stick with a course?
3. How balanced does the path from foundations → advanced topics sound? Any parts you’d reorder or expand?

The first unit is completely

/r/Python
https://redd.it/1mdvynt

datatrees & xdatatrees Release: Improved Forward Reference Handling and New XML Field Types

Just released a new version of the `datatrees` and `xdatatrees` libraries with several key updates.

* `datatrees 0.3.6`: An extension for Python `dataclasses`.
* `xdatatrees 0.1.2`: A declarative XML serialization library for `datatrees`.

# Key Changes:

**1. Improved Forward Reference Diagnostics (**`datatrees`**)** Using an undefined forward reference (e.g., `'MyClass'`) no longer results in a generic `NameError`. The library now raises a specific `TypeError` that clearly identifies the unresolved type hint and the class it belongs to, simplifying debugging.

**2. New Field Type:** `TextElement` **(**`xdatatrees`**)** This new field type directly maps a class attribute to a simple XML text element.

* **Example Class:**

&#8203;

@xdatatree
class Product:
name: str = xfield(ftype=TextElement)

* **Resulting XML:**
```xml
<product><name>My Product</name></product>

**3. New Field Type:** `TextContent` **(**`xdatatrees`**)** This new field type maps a class attribute to the text content of its parent XML element, which is essential for handling mixed-content XML.

* **Example Class:**

&#8203;

@xdatatree
class Address:
label: str = xfield(ftype=Attribute)


/r/Python
https://redd.it/1mdzcyf

How to encrypt the database?

I've seen many apps say their data is encrypted. I've personally never heard of encryption in django.
How to encrypt the data, (when) is that actually necessary?

/r/django
https://redd.it/1mdysok

Flask - AI-powered Image Search App using OpenAI’s CLIP model - Step by Step!!

https://youtu.be/38LsOFesigg?si=RgTFuHGytW6vEs3t

Learn how to build an AI-powered Image Search App using OpenAI’s CLIP model and Flask — step by step!
This project shows you how to:

Generate embeddings for images using CLIP.
Perform text-to-image search.
Build a Flask web app to search and display similar images.
Run everything on CPU — no GPU required!

GitHub Repo: https://github.com/datageekrj/Flask-Image-Search-YouTube-Tutorial
AI, image search, CLIP model, Python tutorial, Flask tutorial, OpenAI CLIP, image search engine, AI image search, computer vision, machine learning, search engine with AI, Python AI project, beginner AI project, flask AI project, CLIP image search

/r/flask
https://redd.it/1me0i1t

OAuth/API Authorization Redirects to Wrong App - Flask/Strava API

Hey all,

I'm building a small web app with a Flask backend and Vue frontend. I'm trying to use the Strava API for user authentication, but I'm running into a very strange problem.

When a user tries to log in, my Flask backend correctly uses my application's Client ID to build the authorization URL. However, the resulting page is for a completely different app called "Simon's Journey Viz" (with its own name, description, and scopes).

I've double-checked my Client ID/Secret, cleared my browser's cache, and even verified my app.py is loading the correct credentials. I've also found that I can't manage my own Strava API app (I can't delete it or create a new one).

Has anyone seen a similar OAuth/API redirect issue where the wrong application is triggered on the authorization page? Could this be related to a specific Flask configuration or something on the API's server-side?

Any insights or potential solutions would be much appreciated!

Thanks

/r/flask
https://redd.it/1me12p8

Using TailwindCss in django

Hey guys,


Recently I built a project and wanted to use tailwindcss for the frontend because you know it saves time and energy and I thought oh I will just use it in Django templates it can't be that bad right.... oh boy I was wrong. So without further ado here is everything I learned trying to put tailwindcss in Django templates using the https://github.com/MrBin99/django-vite package :)

1. Follow this tutorial for basic setup: https://www.youtube.com/watch?v=wgN04Byqi9c
This tutorial explains everything really well and it came out after the tailwindcss 4 update so it is up to date as well because I had a whole heap of pain trying to understand why nothing was working only to realize I was trying to install an outdated tailwindcss package.


2. Trying to remember to include the three tags(

{% load djangovite %}
<head>
{% vitehmrclient %}
{% viteasset '<path to your assets>' %}
</head>

in every single page is an absolute nightmare so Django templating becomes 10x more useful trust me.


3. Django-vite in production is an absolute nightmare because firstly

/r/django
https://redd.it/1me3j8l

Friday Daily Thread: r/Python Meta and Free-Talk Fridays

# Weekly Thread: Meta Discussions and Free Talk Friday 🎙️

Welcome to Free Talk Friday on /r/Python! This is the place to discuss the r/Python community (meta discussions), Python news, projects, or anything else Python-related!

## How it Works:

1. Open Mic: Share your thoughts, questions, or anything you'd like related to Python or the community.
2. Community Pulse: Discuss what you feel is working well or what could be improved in the /r/python community.
3. News & Updates: Keep up-to-date with the latest in Python and share any news you find interesting.

## Guidelines:

All topics should be related to Python or the /r/python community.
Be respectful and follow Reddit's Code of Conduct.

## Example Topics:

1. New Python Release: What do you think about the new features in Python 3.11?
2. Community Events: Any Python meetups or webinars coming up?
3. Learning Resources: Found a great Python tutorial? Share it here!
4. Job Market: How has Python impacted your career?
5. Hot Takes: Got a controversial Python opinion? Let's hear it!
6. Community Ideas: Something you'd like to see us do? tell us.

Let's keep the conversation going. Happy discussing! 🌟

/r/Python
https://redd.it/1mehndi

Can't deploy Flask application in Render

I'm having trouble trying to deploy my Flask backend in Render. I keep getting the same error:

gunicorn.errors.AppImportError: Failed to find attribute 'app' in 'app'. I had to hide some other information

This is my app.py and it's not inside any other file:

# app.py

from flask import Flask

def createapp():
    app = Flask(name)
    CORS(app)

if name == 'main':
    createapp().run(debug=True, port=5000)

/r/flask
https://redd.it/1menxd3

What are some good projects for resume

I am currently learning django and wanted to know some projects that would be good to be put on my resume
Pls Help Me

/r/django
https://redd.it/1mens9t