If you think there's a lot of good things in a particular field, you're an amateur in this field. If you're growing up, don't be surprised that everything around you will seem dull over time.

What is the definition of a good software engineer? This question's aim is to be personal, it focuses on the thoughts of the people you're asking it. I will show you my thoughts in this post.

https://luminousmen.com/post/what-is-the-definition-of-good-engineer

CPU vs GPU

https://youtu.be/-P28LKWTzrI

#ml

Tuning partitions in Spark

https://luminousmen.com/post/spark-tips-partition-tuning

The story of how Facebook, along with the FBI, hunted a man who regularly insulted and attacked little girls. Facebook even paid for a zero-day vulnerability in Tails(!) that allowed to identify the real address of the criminal.

https://www.vice.com/en_us/article/v7gd9b/facebook-helped-fbi-hack-child-predator-buster-hernandez?utm_content=buffer15b0e&utm_medium=social&utm_source=twitter&utm_campaign=buffer

Recommended for reading 👌

#privacy

Congrats! Web scraping is legal! (US precedent)

https://www.reddit.com/r/datascience/comments/excxlv/congrats_web_scraping_is_legal_us_precedent/

Found this little diamond on wiki - you can quote it like:
ninety percent of everything is crap
or
everybody is wrong on the internet

https://en.m.wikipedia.org/wiki/Sturgeon%27s_law

#stuff

Lex has released a rather large but interesting interview with Andrew Ng, a man whose name you've met many times if you've ever been interested in artificial intelligence, machine learning, or technology in general.

https://youtu.be/0jspaMLxBig

#ds #ml

​​Check out background removal from videos. At the end you're getting gifs and it's really quick

http://unscreen.com

#usefullinks

Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.0 is here. Try it out!

https://spark.apache.org/releases/spark-release-3-0-0.html

#spark

Russia lifts its ban on the Telegram messenger app

https://www.theverge.com/2020/6/18/21295535/russia-telegram-ban-lifted-security

#privacy

Have you heard of microservices? Of course you have - any housewife already knows how to deploy them on a k8s cluster. Here's some thinking about them.

https://luminousmen.com/post/thoughts-on-microservices

Wow! Here comes Nextgen - Unreal Engine 5 with high quality textures. It creates as much geometric detail as the eye can see, and dynamic lighting that responds to changes in the scene and lighting with tools and libraries. That means that it can be done without much effort for artists, and we will see it soon enough in many games.

https://youtu.be/qC5KtatMcUw

#stuff

SpaceX use Chromium and JavaScript for the Dragon 2 flight interface. Now I can officially say that frontend nowadays is a rocket science.

Check out the exact replica of the interface by the link below

https://iss-sim.spacex.com/

#stuff

Spark 3.0 now on Databricks

https://databricks.com/blog/2020/06/18/introducing-apache-spark-3-0-now-available-in-databricks-runtime-7-0.html

#spark #big_data

I think it's clear now who's the winner
With that said it can be obvious that standard data scientist tech stack will change in the nearest future. As an example pandas can be somewhat easily replaced by koalas. By the way, new koalas is here and it covers 80% of pandas api - https://github.com/databricks/koalas/releases/tag/v1.0.0

#spark #big_data

Technological degradation

Technology degrades from generation to generation, because no project starts with nothing, you always choose tools, libraries, etc. to solve a problem. Of course you can do everything from scratch using the assembler, but nobody will do it because it doesn't make sense. And when you take the X library which depends on the Y and Z libraries, it doesn't mean that you know what Y and Z do, you just know what X does and how to use it. And so while any problem in IT can be solved by adding an abstraction level, this abstraction stack is extremely hard to know for one person, and without proper communication between old developers and new developers, those connections are lost. And each generation grows at its abstraction level without understanding what's going on below.

https://youtu.be/pW-SOdj4Kkk.

#dev

Understanding Explain Formatted in Spark 3.0

https://medium.com/analytics-vidhya/spark-3-understanding-explain-formatted-d4f33c1dee86

#spark #big_data