​​Dictionaries in CPython are everywhere, classes, global variables, kwargs parameters are based on them, the interpreter creates thousands of dictionaries, even if you did not add any curly brackets in your script. And it is not surprising that their implementation continues to improve and increasingly acquire various tricks.

The internal structure of dictionaries in Python is not limited only to buckets and closed hashing. If you don’t know the number of elements in the dictionary you just created, how much memory is spent for each element, why now (CPython 3.6>) the dictionary is implemented in two arrays and how it relates to maintaining the insertion order, or you just didn’t watch the presentation by Raymond Hettinger "Modern Python Dictionaries A confluence of a dozen great ideas. Then the time has come.

Recommended 👌

#python

Singleton pattern in Python

Do you like Singletons? I don't too — they are a bit complicated.

But you know what? I've never seen in any code(except some famous libraries) and on any interview good implementation of Singleton pattern. We need to fix it!

Please check the following implementation:

 weakref import WeakValueDictionary

class Singleton(type):
    _instances = WeakValueDictionary()

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            instance = super(Singleton, cls).__call__(*args, **kwargs)
            cls._instances[cls] = instance
        return cls._instances[cls]


class Config(metaclass=Singleton):
    pass

Do you know a better implementation? Send me yours and we will discuss

#python

Wednesday

Service which turns photos into comics of the style of the russian comic book published Bubble, has gained popularity.

It handles faces poorly — they becoming scary and ugly especially if the photo is not contrast or blurred. But photos of the cats are interesting.

https://face.bubble.ru/en/

#stuff

​​Guido van Rossum retires from Dropbox. Thank you, Guido.

Jupyter notebook will have to make room for a new solution from netflix. Polynote — a new, polyglot notebook with first-class Scala support, Apache Spark integration, multi-language interoperability including Scala, Python, and SQL, as-you-type autocomplete, and more.

https://medium.com/netflix-techblog/open-sourcing-polynote-an-ide-inspired-polyglot-notebook-7f929d3f447

#stuff

What is the definition of a good software engineer? This question's aim is to be personal, it focuses on the thoughts of the people you're asking it. I will show you my thoughts in this post.

​​https://streamlit.io/
Web prototyping / reporting framework. It looks like a great alternative to bokeh/plotly - pure python without callbacks and with advanced data caching

Those parts of the system that you can hit with a hammer (not advised) are called "hardware"; those that you can only curse at are called "software"
— Anonymous

​​I have bad news for devops citizens — you have nothing to strive for. My deepest condolences.

https://stackoverflow.blog/2019/10/16/coding-salaries-in-2019-updating-the-stack-overflow-salary-calculator/

Rammstein soloist Till Lindemann released a clip that generated by GAN. He showed us again what psychedelics really are.

I like the meaning that he tries to put into this.

Ich weiß nicht, wer ich bin
...
Ich brauche keinen Spiegel
Weiß nicht um mein Gesicht

It doesn't know who it is and doesn't remember anything, but all of these faces and things are things that snail used to be in one past life or another a.k.a reincarnation. GAN implementation is also symbolic — those mixture of reality and fixture is cool (my interpretation).

https://youtu.be/obY4c9aqUqs

#stuff

Wednesday

Python developers decided to switch to a new release cycle. Significant new issues of the language will now be issued once a year, not once in 1.5 years as it was before. Good? No.

#python

Review the popular tools for analyzing Python code and its details.

https://luminousmen.com/post/python-static-analysis-tools

Nice tips for those who use Airflow, worth reading

https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f

#big_data

Type checking in runtime

By default, function annotations do not influence how your code is working, but merely help you to point code intentions and raise linting errors.

However, you can enforce type checking in runtime with tools like enforce, this can help you in debugging (there are many cases when type hinting is not working).

@enforce.runtime_validation
def foo(text: str) -> None:
    print(text)

foo('Hi') # ok
foo(5)    # fails

@enforce.runtime_validation
def any2(x: List[bool]) -> bool:
    return any(x)

any([False, False, True, False]) # True
any([False, False, True, False]) # True

any(['False']) # True
any(['False']) # fails

any([False, None, "", 0]) # False
any([False, None, "", 0]) # fails

#python

Use pathlib

pathlib is a default module in python3, that helps you to avoid tons of confusing os.path.joins:

from pathlib import Path

dataset_dir = 'data'
dirpath = Path('/path/to/dir/')

full_path = dirpath / dataset_dir

for filepath in dataset_dir.iterdir():
    with filepath.open() as f:
        # do stuff

Previously it was always tempting to use string concatenation (concise, but obviously bad), now with pathlib the code is safe, concise, and readable.

Also pathlib.Path has a bunch of methods and properties, that I previously had to google:

p.exists()
p.is_dir()
p.parts
p.with_name('sibling.png') # only change the name, but keep the folder
p.with_suffix('.jpg') # only change the extension, but keep the folder and the name
p.chmod(mode)
p.rmdir()

See how easy it to get all images recursively(without glob module):

found_images = pathlib.Path('/path/').glob('**/*.jpg')

#python

Guide into bucketing - an optimization technique that uses buckets to determine data partitioning and avoid data shuffle.

https://luminousmen.com/post/the-5-minute-guide-to-using-bucketing-in-pyspark