Dynamic analysis for static typing in Python!

Недавно инженеры Instagram открыли исходный код типизатора MonkeyType — инструмента для динамического run-time анализа программ на Python и автоматизации внедрения статических аннотаций типов в исходниках на Python, с использованием gradual typing расширения mypy (PEP-484) для интерпретатора CPython.
Статические аннотации типов и постепенная типизация (gradual typing) в динамических языках повышают надёжность и предсказуемость поведения программы во время исполнения, уменьшают количество run-time ошибок и исключительных ситуаций, необходимых тестов для их покрытия и выявления, а также повышают быстродействие приложений, т.к. во время исполнения (in run-time) программы уже не производится динамический вывод (type inference), проверка (type matching), диспетчеризация (dynamic dispatching) и связывание (type linkng) типов, что не вызывает задержек проверки типов по времени исполнения кода, а только сокращает их и таким образом ощутимо повышает производительность и скорость исполнения кода приложения.

https://engineering.instagram.com/let-your-code-type-hint-itself-introducing-open-source-monkeytype-a855c7284881

https://github.com/Instagram/MonkeyType

#Python

Ссылки на материалы по теме:
О CPython+Mypy, постепенной типизации (gradual typing) и source maping отладке run-time исключений с применением Sentry:
https://xn--r1a.website/technologique/1124
https://xn--r1a.website/technologique/1125
https://xn--r1a.website/technologique/155

https://www.python.org/dev/peps/pep-0484/
https://github.com/python/mypy

Оптимизация критических участков проектов на Python с помощью Rust:
https://xn--r1a.website/technologique/1123
https://xn--r1a.website/technologique/1153

Multi-threading & Multi-processing for Python

Одни из лучших (если не самые лучшие) библиотечные решения для создания и управления thread-pools и process-pools для Python, обхода GIL ограничений интерпретатора CPython в плане многопоточной обработки данных и работы с реальной системной многогопоточностью (системными потоками), вытесняющей (pre-emptive) многозадачностью, для создания thread-mapper'ов (M:N dispatcher/scheduler/monitor) программных потоков сопрограмм/генераторов/asyncIO на системные треды.
Естественно у данных решений есть своя цена, overhead и накладные расходы.

https://github.com/eXascaleInfolab/PyExPool

https://github.com/noxdafox/pebble
https://pypi.org/project/Pebble/
https://pebble.readthedocs.io/en/latest/

#Python
#multithreading
#multiprocessing

Python 3.14

https://blog.miguelgrinberg.com/post/python-3-14-is-here-how-fast-is-it

In short - new Python 3.14 it's awesome! Worth to update immediately!

3.14 is way much better in performance than any previous versions, has optionally enabled JIT (doesn't give too much performance boost, due to the too much dynamic nature of Python and vibrant run-time objects lifetimes) and optionally disabled GIL for multi-threading (installed as separately compiled binary in a system).
But PyPy JIT still outperform CPython.

Much love for Python anyways! 🙌 Python is a cross-system glue now!

Comparison with Rust is just for fun here - Python always will be much more slower, due to the dynamic types dispatch through vtables. And due to the dynamic nature Python always will allow run-time unexpected behavior and run-time crashes (thus should be covered thoroughly with tests for everything), while Rust is fully static (even Dyn trait impls checked by compiler in compile time) and fully type safe (in compile time, before running).

There are also more consistent benchmarking test suite across languages:
https://benchmarksgame-team.pages.debian.net/benchmarksgame/box-plot-summary-charts.html
(They should update Python environment soon and we'll see 3.14 results - now 3,13 used.)

#Python
#Rust

NoGIL is definitely a huge leap forward!

From 3.13 GIL can be disabled... but for this we need customly build interpreter from sources. That's the point should be refined.

Cause not every main Linux distro now provide prebuilt packages, only Fedora (python3.14-freethreading package), OpenSUSE (python314-nogil package), Ubuntu (python3.14-nogil package through external PPA) and Nix (python314FreeThreading package), in Gentoo via own ebuild, or in Arch via own pkgbuild script.

This will provide python3.14t with NoGIL enabled by default, and we can enable GIL with PYTHON_GIL environment variable or the command-line option -X gil for CPython.

But... free-threaded CPython build is not thread safe!
Thread safety, i.e. managing shared mutable state for simultaneous threads, using locks, mutexes and other synchronization primitives - are fully on developer. Python code is thread safe. But CLang code (via FFI) and Python interpreter code itself, that written in CLang, can allow access to the same memory, for pointers in several threads, lead to data race and deadlocks. Also can lead to dead/hanging objects in memory and thus memory leaks in long uptimes.
And this will affect run-time and revealed only in run-time.
(While in Rust for example pointers/references are typed and type-safe, thus allocations/deallocations, objects lifetimes tracking, pointers/references to same data and memory regions, are tracked in compile time, via move semantics, which completely prevents dangling pointers.)

Thus memory sanitizers and threads sanitizers should be used for free-threaded CPython. And not all main/core libraries in PyPI now support free-threading.

https://docs.python.org/3/howto/free-threading-python.html

https://py-free-threading.github.io/installing-cpython/

https://py-free-threading.github.io/running-gil-disabled/

https://py-free-threading.github.io/debugging/

https://py-free-threading.github.io/thread_sanitizer/

#Python
#notes