​​I have bad news for devops citizens — you have nothing to strive for. My deepest condolences.

https://stackoverflow.blog/2019/10/16/coding-salaries-in-2019-updating-the-stack-overflow-salary-calculator/

Rammstein soloist Till Lindemann released a clip that generated by GAN. He showed us again what psychedelics really are.

I like the meaning that he tries to put into this.

Ich weiß nicht, wer ich bin
...
Ich brauche keinen Spiegel
Weiß nicht um mein Gesicht

It doesn't know who it is and doesn't remember anything, but all of these faces and things are things that snail used to be in one past life or another a.k.a reincarnation. GAN implementation is also symbolic — those mixture of reality and fixture is cool (my interpretation).

https://youtu.be/obY4c9aqUqs

#stuff

Wednesday

Python developers decided to switch to a new release cycle. Significant new issues of the language will now be issued once a year, not once in 1.5 years as it was before. Good? No.

#python

Review the popular tools for analyzing Python code and its details.

https://luminousmen.com/post/python-static-analysis-tools

Nice tips for those who use Airflow, worth reading

https://medium.com/datareply/airflow-lesser-known-tips-tricks-and-best-practises-cf4d4a90f8f

#big_data

Type checking in runtime

By default, function annotations do not influence how your code is working, but merely help you to point code intentions and raise linting errors.

However, you can enforce type checking in runtime with tools like enforce, this can help you in debugging (there are many cases when type hinting is not working).

@enforce.runtime_validation
def foo(text: str) -> None:
    print(text)

foo('Hi') # ok
foo(5)    # fails

@enforce.runtime_validation
def any2(x: List[bool]) -> bool:
    return any(x)

any([False, False, True, False]) # True
any([False, False, True, False]) # True

any(['False']) # True
any(['False']) # fails

any([False, None, "", 0]) # False
any([False, None, "", 0]) # fails

#python

Use pathlib

pathlib is a default module in python3, that helps you to avoid tons of confusing os.path.joins:

from pathlib import Path

dataset_dir = 'data'
dirpath = Path('/path/to/dir/')

full_path = dirpath / dataset_dir

for filepath in dataset_dir.iterdir():
    with filepath.open() as f:
        # do stuff

Previously it was always tempting to use string concatenation (concise, but obviously bad), now with pathlib the code is safe, concise, and readable.

Also pathlib.Path has a bunch of methods and properties, that I previously had to google:

p.exists()
p.is_dir()
p.parts
p.with_name('sibling.png') # only change the name, but keep the folder
p.with_suffix('.jpg') # only change the extension, but keep the folder and the name
p.chmod(mode)
p.rmdir()

See how easy it to get all images recursively(without glob module):

found_images = pathlib.Path('/path/').glob('**/*.jpg')

#python

Guide into bucketing - an optimization technique that uses buckets to determine data partitioning and avoid data shuffle.

https://luminousmen.com/post/the-5-minute-guide-to-using-bucketing-in-pyspark

Pip constraints files

In python, it is common practice to write all the application dependencies that are installed via pip into a separate text file called requirements.txt. It's good practice to fully specify package versions in your requirements file. And in our case, everything will be there — both direct dependencies of our application and dependency dependencies, etc.

But sometimes, especially on a long-lived project, it's hard to understand what dependencies were original. It is necessary to update them on time, not depend on packages that are outdated or no longer needed for some reason.

For example, which of the following dependencies is the original?

# requirements.txt
numpy==1.17.4
pandas==0.24.2
python-dateutil==2.8.1
pytz==2019.3
six==1.13.0

Yes, it's pandas.

One of the mechanisms for separating dependencies is implemented using another text file called constants.txt.

It looks exactly like requirements.txt:

# constants.txt
numpy==1.17.4
python-dateutil==2.8.1
pytz==2019.3
six==1.13.0

Constraints files differ from requirements files in one key way: putting a package in the constraints file does not cause the package to be installed, whereas a requirements file will install all packages listed. Constraints files are simply requirements files that control which version of a package will be installed but provide no control over the actual installation.

To use this file, you can do it via the requirements.txt file:

-c constraints.txt
pandas==0.24.2

or

pip install -c constraints.txt

will install all packages from requirements.txt and using constraints.txt files for version constraint.

#python

Go test the most advanced neural network model for generating text - GPT-2. You write a phrase, the neural network complements it to a small text.

Try it yourself: https://talktotransformer.com/

#ml

Saturday?

It's time to send an email!

https://youtu.be/szdbKz5CyhA

Drum machine in Excel

Disgustingly interesting...

http://youtube.com/watch?v=To2JIXGoYzA

"In all likelihood, sorting is one of the most researched classes of algorithms. It is a fundamental task in Computer Science, both on its own and as a step in other algorithms. Efficient algorithms for sorting and searching are now taught in core undergraduate classes. Are they at their best, or is there more blood to squeeze from that stone? This talk will explore a few less known – but more allegro! – variants of classic sorting algorithms. And as they say, the road matters more than the destination. Along the way, we’ll encounter many wondrous surprises and we’ll learn how to cope with the puzzling behavior of modern complex architectures."

If u know who is Andrei Alexandrescu than it's recommended 👌

https://youtu.be/FJJTYQYB1JQ

#dev

DataFrames are the wave of the future in the Spark world so let's push your Pyspark SQL knowledge in using various join types

https://luminousmen.com/post/introduction-to-pyspark-join-types