This guy created a testing tool for distributed systems called Jepsen. So he tested Riak, MongoDB, Redis, Cassandra, NuoDB, Kafka, Zookeeper, etcd, Consul, Elasticsearch, RabbitMQ, Aerospike, etc. So, Zookeeper is passed his tests, and that's it, just Zookeeper.

https://youtu.be/tRc0O9VgzB0

#big_data

Old post about non-practicality of CAP-theorem by Martin Kleppmann

https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-cp-or-ap.html

#dev

M-Motivation, M-money, M-Mastery, M-Mystery

https://luminousmen.com/post/m-motivation

In depth explanation of a new features in Apache Spark 3.0

- The new Adaptive Query Execution (AQE) framework within Spark 3.0 can yield query performance gains. Based on a 3TB TPC-DS benchmark, two queries had more than a 1.5x speedup, and another 37 queries had more than 1.1x speedup.
- With Dynamic Partition Pruning (DPP), we can significantly speed up performance by pruning partitions based on the joins between the fact and dimension tables common in star schema design.
- Accelerator-aware Scheduling helps Spark take advantage of GPU and hardware accelerators for certain workloads (e.g deep learning). This release enhances the scheduler and makes the cluster manager accelerator-aware.
- Spark 3.0 also introduces new Pandas UDF types and new Pandas function APIs for improved performance and usability.
- Enhanced monitoring capabilities including the new UI for Structured Streaming, enhanced EXPLAIN command, and observable metrics.


https://youtu.be/g-qZslQsOuE

#spark #big_data

A large database of datasets for computer vision projects. Can be sorted by popularity, addition and publication times. Please use whose interested: https://www.visualdata.io/discovery.

#cv #ds

Procedural content creation pipeline importing OpenStreetMap data into Unity using Houdini to generate real time environments based on real cities. Everything in the video placed or generated procedurally.

https://twitter.com/stinastinzen/status/1287692890253271041

#ml

Data & AI Landscape 2019, Source: https://mattturck.com/data2019/

Hard to see anything, right?

#big_data

Thoughts on management

The manager is like a system administrator. The better he works, the more free time he has (which can be used for larger tasks rather than firefighting and micromanaging). If you get fired because of it, then this is a stupid company, and there is no point in catching something there. Or you just outgrew the company, and it's time to look for new tasks on the side.

If you set up the processes and organize the work so that it doesn't take 100% of your time, you can take more employees, run more projects. Lead a department or a team.

You probably know that you can do a lot more with other people's hands. More people are bigger tasks and a bigger contribution to the company's success.

#management

The AI bot defeated a human pilot(5:0) in a series of virtual air battles that unfolded in the sky, albeit in an air simulator, during a competition conducted by the American military research unit DARPA. This is a detailed 5-hour video of AI actions. All the fun starts at 4:40.

Well, now we've got a machine that kills people better than any human...

https://youtu.be/NzdhIA2S35w

#ml

Lex Fridman explains the number of GPT-3 parameters and explains how much will it cost to train a language model the size of the human brain.

https://youtu.be/kpiY_LemaTc

#ml

Define "production-ready"

I came across this interesting question on one of the stackoverflow-like sites. I've never thought about it in terms of definition. Interesting read.

https://softwareengineering.stackexchange.com/questions/61726/define-production-ready.

#dev

Check it out - upscaled and colorized footage about the earliest born person ever (1905) to be caught on film

https://www.reddit.com/r/interestingasfuck/comments/idbtrg/i_upscaled_and_colorized_the_footage_about_the/

At the moment Kubernetes is one of the most exciting technologies in the world of DevOps. Recently a lot of hype has formed around it for one simple reason, and this reason is the mighty containers.

https://luminousmen.com/post/kubernetes-101

It[AI] wears a cloak of black velvet; and a cowl, covering its face. It sits, on a throne made of dry bones of the dead, at the center of a large hall...
— GPT-3

A small fantastic story in which part of the text and the dialogue at the end are written by neuron GPT-3. At the edges there is communication between the author and the network, for me it seems the most interesting part.

https://jamesyu.org/singular/

#ds

Researchers from Google Research offered a whole family of scalable and efficient classifiers called EfficientDet.

Tests have shown that the new technology is able to show accuracy commensurate with its predecessors, while being 9 times smaller and using less computing power.

https://ai.googleblog.com/2020/04/efficientdet-towards-scalable-and.html (https://ai.googleblog.com/2020/04/efficientdet-towards-scalable-and.html)

#ml #ds

Facebook developed an online solution called TransCoder, whose main task is to translate code from one language to another using deep learning. Now the solution can successfully translate functions between C++, Python 3 and Java.

Now it's easy to move to Python from Java ;)


https://ai.facebook.com/blog/deep-learning-to-translate-between-programming-languages/

#python #ml

The researchers fed raw data from the Kepler telescope (retired) to a ML model that had previously been trained to recognize exoplanets - and received potentially 50 new exoplanets that were previously unknown.

https://www.cnn.com/2020/08/26/tech/ai-new-planets-confirmed-intl-hnk-scli-scn/index.html?tg.

#ml

Why TPU so fast?

https://youtu.be/JC84GCU7zqA

#ml