​​Keep an eye on your #docker image size and prevent it from growing too big!

Limit your docker image size with a simple CLI command. Perfect to be used inside your CI process.

Also available as a Github Action.

https://github.com/wemake-services/docker-image-size-limit

#python

​​Clearly is a real time monitor for your celery tasks and workers. Clearly see and debug your celery cluster in real time!

> While I do like flower, to me it's not been totally up to the task (pun intended :).
Why is that? flower needs page refreshes, filters only one task type at a time, displays results as plain strings without any formatting or syntax highlighting, and on top of that also truncates them!

> And clearly is actually real time, has multiple simultaneous filters, a beautiful syntax highlighting system, an advanced formating system that shows parameters, results and tracebacks just as an ipython would, has complete un-truncated results and is very easy to use! Also you can install it very easily with a docker image!

> It's great to actually see in a totally real time way what's going on in your celery workers! So it's very nice for inspecting, debugging, and even demonstrating your company async-superpowers (put clearly on a big screen TV showing all tasks of your production environment)!

> Clearly is composed of a server, which collects real time events from the celery cluster, generates missing states, and streams filtered data to connected clients; and a client, which you use to send filter commands and display both real time and stored data. They communicate with each other via gRPC and ProtocolBuffers.

https://github.com/rsalmei/clearly

#python

​​Beartype is an open-source pure #python PEP-compliant O(1) constant-time runtime type checker emphasizing efficiency, portability, and thrilling puns.

Beartype brings Rust inspired zero-cost abstractions into the lawless world of dynamically-typed pure Python.

Beartype is portably implemented in Python 3, continuously stress-tested via GitHub Actions + tox + pytest, and permissively distributed under the MIT license. Beartype has no runtime dependencies, only one test-time dependency, and supports all Python 3.x releases still in active development.

https://github.com/beartype/beartype

wemake-python-styleguide@0.15 is just released!

It is the strictest and most opinionated #python linter ever.
Now with even more rules!

https://github.com/wemake-services/wemake-python-styleguide/releases/tag/0.15.0

Features that I really like in this new release:
- python3.9 official support
- Python gets adding new stuff - we continue to forbid using it, now with relaxed decorator syntax. Only old style is allowed, yeay!
- Unpythonic getters and setters detection: no more get_x and set_x methods!
- Consistent comprehension structure: now all your comprehensions will look the same!
- Configuration option to forbid ignoring some violations inline with noqa, might be really useful for complexity rules!

Check it out!

Even more goodness will land in the future vesions. We have a lot of ideas! Do you want to help with the project development? Drop me a line!

So, after recent cryptography incident (https://github.com/pyca/cryptography/issues/5771) our build-times inside the Alpine-based docker-in-docker CI went from several seconds to ~6 minutes just to install #python dependencies.

This forced me to create our own dind image (https://github.com/wemake-services/wemake-dind/) with everything properly set up: python, rust (required to build `cryptography`), docker-compose, and several #docker quality tools like:
- https://github.com/wemake-services/dump-env which enforces strict .env configuration policies
- https://github.com/wemake-services/docker-image-size-limit which enforces strict image size control

Feel free to use it in your own CI pipelines! Not just for python, but for any docker-in-docker setups that use docker-compose.

My new article is out!

This time I want to discuss a pretty useful idea for library authors and their users: there are better ways to test your code!

I give three examples (in #python, but the idea itself applies to almost any language) of how user projects can be self-tested without actually writing any real test cases by the end-user. One is hypothetical about django and two examples are real and working: featuring deal and dry-python/returns.

Short example with deal:

import deal

@deal.pre(lambda a, b: a >= 0 and b >= 0)
@deal.raises(ZeroDivisionError)  # this function can raise if `b=0`, it is ok
def div(a: int, b: int) -> float:
    if a > 50:  # Custom, in real life this would be a bug in our logic:
        raise Exception('Oh no! Bug happened!')
    return a / b

This bug can be automatically found by writing a single line of test code: test_div = deal.cases(div). As easy as it gets!

From this article you will learn:
-How to use property-based testing on the next level
- How a simple decorator @deal.pre(lambda a, b: a >= 0 and b >= 0) can help you to generate hundreds of test cases with almost no effort
- What "Monad laws as values" is all about and how dry-python/returns helps its users to build their own monads

I really like this idea! And I would appreciate your feedback on it.
Link: https://sobolevn.me/2021/02/make-tests-a-part-of-your-app

My new article is out! It is all about typeclasses in Python and dry-python/classes.

Today I am explaining what typeclasses are and how to use them. I give examples in 4 very different languages: #rust, #elixir, #haskell, and #python to show that this concept is universal.

I am also showing that this idea is very pythonic by comparing our classes implementation with functools.singledispatch. There are lots of different details!

Check how easy it is to define a typeclass with classes:

from classes import AssociatedType, Supports, typeclass

class Greet(AssociatedType):
    """Special type to represent that some instance can `greet`."""

@typeclass(Greet)
def greet(instance) -> str:
    """No implementation needed."""

@greet.instance(str)
def _greet_str(instance: str) -> str:
    return 'Hello, {0}!'.format(instance)

def greet_and_print(instance: Supports[Greet]) -> None:
    print(greet(instance))

greet_and_print('world')

Check it out!

Friendly reminder: you can join our dry-python telegram chat: https://xn--r1a.website/drypython

We discuss functional programming, different new languages and ideas, typing, and #python! Stop by and chat!

Textual - a Text User Interface with Rich as the renderer.

Why is it cool?
1. It has an amazing grid layout
2. It is from the same team as Rich, which proved to be amazing
3. It has nice user-facing API (it even has async parts), way better than curses

But, it is still WIP. We need to wait for a little bit.

https://github.com/willmcgugan/textual

#python

​​pdb++, a drop-in replacement for pdb (the #python debugger)

This module is an extension of the pdb module of the standard library. It is meant to be fully compatible with its predecessor, yet it introduces a number of new features to make your debugging experience as nice as possible.

pdb++ features include:
- colorful TAB completion of Python expressions (through fancycompleter)
- optional syntax highlighting of code listings (through Pygments)
- sticky mode
- several new commands to be used from the interactive (Pdb++) prompt
- smart command parsing (hint: have you ever typed r or c at the prompt to print the value of some variable?)
- additional convenience functions in the pdb module, to be used from your program

Personal opinion: I don't like pdb and debuggers in general. Whenever I open a debugger - it means that I have a really big problem. But, this one seems like the best solution (except PyCharm, of course).

https://github.com/pdbpp/pdbpp

​​Type4Py: Deep Similarity Learning-Based Type Inference for #python

Over the past decade, machine learning (ML) has been applied successfully to a variety of tasks such as computer vision and natural language processing. Motivated by this, in recent years, researchers have employed ML techniques to solve code-related problems, including but not limited to, code completion, code generation, program repair, and type inference.

Dynamic programming languages like Python and TypeScript allows developers to optionally define type annotations and benefit from the advantages of static typing such as better code completion, early bug detection, and etc. However, retrofitting types is a cumbersome and error-prone process. To address this, we propose Type4Py, an ML-based type auto-completion for Python. It assists developers to gradually add type annotations to their codebases.

https://github.com/saltudelft/type4py
Announcing post: https://mirblog.net/index.php/2021/07/31/development-and-release-of-type4py-machine-learning-based-type-auto-completion-for-python/

Paranoid Scientist is a #python module for verifying scientific software.

It provides:
- Runtime verification of entry and exit conditions written in pure Python, including hyperproperties.
- Conditions specified using pure Python refinement types, i.e. types are defined by predicates.
- Automated unit testing of individual functions.

It is inspired by contract-oriented programming, type classes, static type checking, and fuzz testing.

Paranoid Scientist is a tool to make sure scientific code is correct. Verification is extremely important for scientific software because, unlike most software, we don’t know what the output will be until we run the program. In fact, the program is written in order to examine the output. However, we have no robust way of knowing whether the output is due to a software bug. For example, code performing a complex statistical test could normalize the wrong column, an error which would likely go undetected.

Paranoid Scientist attempts to remedy this situation by providing some key tools from the software verification community to the scientific community. Traditional program verification asks the question, “If I run my code, will it run correctly?” In practice, this is time consuming and requires highly specialized training. For scientific programming, it is acceptable to instead ask, “If I already ran my code, did it run correctly?” In other words, it is not as important to know before executing the program whether it will run correctly. Paranoid Scientist is already in use in scientific software.

https://github.com/mwshinn/paranoidscientist

​​KDL is a document language with xml-like semantics that looks like you're invoking a bunch of CLI commands! It's meant to be used both as a serialization format and a configuration language, much like JSON, YAML, or XML.

Features:
- Node-based, you won't have to mess with identation to make it work
- Type annotations
- Implementations in different languages: #python, #elixir, #rust, #java, #js, #ruby, #php, etc

https://kdl.dev/

CLI tool and python library that converts the output of popular command-line tools and file-types to JSON or Dictionaries. This allows piping of output to tools like jq and simplifying automation scripts.

https://github.com/kellyjonbrazil/jc

#python

​​Interactive Redis: A Terminal Client for Redis with AutoCompletion and Syntax Highlighting.

IRedis is a terminal client for redis with auto-completion and syntax highlighting. IRedis lets you type Redis commands smoothly, and displays results in a user-friendly format.

IRedis is an alternative for redis-cli. In most cases, IRedis behaves exactly the same as redis-cli. Besides, it is safer to use IRedis on production servers than redis-cli: IRedis will prevent accidentally running dangerous commands, like KEYS *

Has many other awesome features!

https://github.com/laixintao/iredis

#python

⚡️ Breaking news!

Python3.10 was released several hours ago (not so breaking news, right?). One of the main new features is pattern matching.

Today, I am releasing dry-python/returns@0.17.0 with full pattern matching support. Here's an example of how you can use it together with Result type:

from returns.result import Failure, Success, safe

@safe
def div(first_number: int, second_number: int) -> int:
    return first_number // second_number

match div(1, 0):
    # Matches if the result stored inside `Success` is `10`
    case Success(10):
        print('Result is "10"')

    # Matches any `Success` instance and binds its value to the `value` variable
    case Success(value):
        print('Result is "{0}"'.format(value))

    # Matches if the result stored inside `Failure` is `ZeroDivisionError`
    case Failure(ZeroDivisionError):
        print('"ZeroDivisionError" was raised')

    # Matches any `Failure` instance
    case Failure(_):
        print('The division was a failure')

Cool, isn't it? Today Python made one more giant step for better functional programming support. And no doubt, that dry-python is the first one to officially support it.

Check out our:
- Docs: https://returns.readthedocs.io/en/latest/pages/result.html#pattern-matching
- Release notes: https://github.com/dry-python/returns/releases/tag/0.17.0
- Python3.10 release notes: https://docs.python.org/3.10/whatsnew/3.10.html

Soon we will be adding pattern matching support for mypy as well. So, it would be type checked correctly.

Big day for #python!

​​SQLFluff is a dialect-flexible and configurable SQL linter. Designed with ELT applications in mind, SQLFluff also works with Jinja templating and dbt. SQLFluff will auto-fix most linting errors, allowing you to focus your time on what matters.

Although SQL is reasonably consistent in its implementations, there are several different dialects available with variations of syntax and grammar. SQLFluff currently supports the following SQL dialects (though perhaps not in full):
- ANSI SQL - this is the base version and on occasion may not strictly follow the ANSI/ISO SQL definition
- BigQuery
- Exasol
- Hive
- MySQL
- PostgreSQL (aka Postgres)
- Snowflake
- SQLite
- Teradata
- Transact-SQL (aka T-SQL)

https://github.com/sqlfluff/sqlfluff

#python

Сегодня говорим про bytes!

Вышел восьмой урок "Лучшего курса по Питону": https://www.youtube.com/watch?v=RbznhbK3vC0

Что вообще такое "Лучший курс по Питону"?
- Я решил разобрать все исходники CPython и показать, как на самом деле работают все его части
- В каждом видео я рассказываю про одну узкую тему
- Каждое видео я делю на три уровня сложности: для джунов, мидлов и сениоров
- Переодически беру интервью у других core-разработчиков CPython про разные части интерпретатора в их зоне интересов
- Получается очень хардкорно!

Например, в bytes я показываю, как устроен PyBytesObject (он простой):

typedef struct {
    PyObject_VAR_HEAD
    Py_DEPRECATED(3.11) Py_hash_t ob_shash;
    char ob_sval[1];

    /* Invariants:
     *     ob_sval contains space for 'ob_size+1' elements.
     *     ob_sval[ob_size] == 0.
     *     ob_shash is the hash of the byte string or -1 if not computed yet.
     */
} PyBytesObject;

Как устроен Buffer протокол для bytes с его __buffer__ и __release_buffer__:

static int
bytes_buffer_getbuffer(PyBytesObject *self, Py_buffer *view, int flags)
{
    return PyBuffer_FillInfo(view, (PyObject*)self, (void *)self->ob_sval, Py_SIZE(self), 1, flags);
}

static PyBufferProcs bytes_as_buffer = {
    (getbufferproc)bytes_buffer_getbuffer,
    NULL,
};

Дополнительные материалы (не вошли в выпуск):
- mypy: bytes и bytearray c disable_bytearray_promotion и disable_memoryview_promotion https://github.com/python/mypy/commit/2d70ac0b33b448d5ef51c0856571068dd0754af6
- Мутабельность bytes https://docs.python.org/3.13/c-api/bytes.html#c._PyBytes_Resize
- PyBytes_Writer API: https://github.com/capi-workgroup/decisions/issues/39
- ob_shash deprecation: https://github.com/python/cpython/issues/91020

Поддержать проект можно тут: https://boosty.to/sobolevn

#лкпп #python #c

PythoNN: видео с апрельского митапа

4 апреля прошел очередной #python митап в Нижнем Новгороде.

Было очень душевно и интересно.
Случился аншлаг! Пришло много нижегородцев и приехало очень много гостей: из Москвы, Питера, Кирова и других городов. Спасибо всем!

Было 4 крутых доклада:
- "Are you NATS?" – Гурбанов Михаил https://youtube.com/watch?v=atD3JVWurno
- "Почему исправление опечаток сложнее, чем кажется, и как мы с этим српавляемся" – Дмитрий Бровкин https://youtube.com/watch?v=9HRBwwaMIfA
- "Современный web с современными темплейтами" – Алексей Гончарук https://youtube.com/watch?v=lN3Pz_hUCio
- "Демистификация PostgreSQL-индексов" – Алексей Голобурдин https://youtube.com/watch?v=6kVGSLdj28k

А потом мы сидели в баре до 5 утра.

Что улучшить?
- Первый раз записывал на StreamYard, сделал плохую композицию слайдов и видео докладчика, исправим в следующий раз. Прикрепил все слайды в описании докладов – чтобы была возможность все прочитать и скопировать код
- Поработаем над звуком, сейчас он немного прыгал

Хотите присоединиться?
- Если хотите сделать доклад, пишите мне в личку – лично учу новичков выступать и делать слайды, полная свобода в выборе темы
- Если хотите просто послушать – следите за анонсами в чате и подписывайтесь на мой канал с записями

У нас в Нижнем – просто офигенно, всех ждем в гости! 🌆

| Поддержать | YouTube | GitHub | Чат |

Находки в опенсорсе: EasyP – тулбокс для ProtoBuf файлов

https://www.youtube.com/watch?v=XI-dNpM77iM

easyp – пакетный менеджер, билд-система и линтер для .proto файлов.
Хоть easyp и написан на #go 😱, одна из его фишек в том – что вы можете использовать любые плагины для генерации финального кода: он может быть хоть на #python, хоть на #rust.

Если много используете ProtoBuf – обязательно для ознакомления!

Как оно работает?

# Секция для правил линтера:
lint:
  use:
    - DEFAULT

# Секция с зависимостями:
deps:
  - github.com/googleapis/googleapis
  - github.com/grpc-ecosystem/grpc-gateway@v2.20.0

# Секция для правил сборки и генерации итоговых файлов:
generate:
  plugins:
    - name: go
      out: .
      opts:
        paths: source_relative
    - name: go-grpc
      out: .
      opts:
        paths: source_relative
        require_unimplemented_servers: false

Прощайте огромные Makefile с кучей скриптов для сборки.

Что будет в видео?

В видео:
- спросим зачем ребята его сделали
- узнаем как оно работает
- покажем, откуда можно устанавливать зависимости
- потыкаем разные юзкейсы
- поговорим про безопасность решения
- обсудим планы на следующие релизы
- расскажем как находить ломающие изменения в вашей ProtoBuf спецификации

А еще в видео кот на гитаре играет!

Репозиторий: https://github.com/easyp-tech/easyp
Документация: https://easyp.tech
Чатик сообщества: @easyptech

Поставьте пацанам звездочек, если проект понравился!

Обсуждение: Какие инструменты для ProtoBuf вы используете? Какие есть пролемы? Что можно было бы добавить в качестве новой фичи в easyp?

| Поддержать | YouTube | GitHub | Чат |