n8n: Free and Open Workflow Automation Tool.

n8n (pronounced nodemation) helps you to interconnect every app with an API in the world with each other to share and manipulate its data without a single line of code. It is an easy to use, user-friendly and highly customizable service, which uses an intuitive user interface for you to design your unique workflows very fast. Hosted on your server and not based in the cloud, it keeps your sensible data very secure in your own trusted database. n8n has 100+ different nodes to automate workflows.

https://github.com/n8n-io/n8n

#ts #nocode

​​Mypy stubs, i.e., type information, for numpy, pandas and matplotlib for your #ds #python projects.

Lots of functions are already typed, but a lot is still missing (numpy and pandas are huge libraries).

https://github.com/predictive-analytics-lab/data-science-types

​​Great Expectations: Always know what to expect from your data.

Great Expectations helps data teams eliminate pipeline debt, through data testing, documentation, and profiling.

Software developers have long known that testing and documentation are essential for managing complex codebases. Great Expectations brings the same confidence, integrity, and acceleration to data science and data engineering teams.

See Down with Pipeline Debt! for an introduction to the philosophy of pipeline testing: https://medium.com/@expectgreatdata/down-with-pipeline-debt-introducing-great-expectations-862ddc46782a

Key features:
- Expectations or assertions for data. They are the workhorse abstraction in Great Expectations, covering all kinds of common data issues
- Batteries-included data validation
- Tests are docs and docs are tests: many data teams struggle to maintain up-to-date data documentation. Great Expectations solves this problem by rendering Expectations directly into clean, human-readable documentation
- Automated data profiling: wouldn't it be great if your tests could write themselves? Run your data through one of Great Expectations' data profilers and it will automatically generate Expectations and data documentation
- Pluggable and extensible

https://github.com/great-expectations/great_expectations

#python #ds #docops

​​Materialize is a streaming database for real-time applications. Written in #rust

Materialize lets you ask questions of your live data, which it answers and then maintains for you as your data continue to change. The moment you need a refreshed answer, you can get it in milliseconds. Materialize is designed to help you interactively explore your streaming data, perform data warehousing analytics against live relational data, or just increase the freshness and reduce the load of your dashboard and monitoring tasks.

Materialize focuses on providing correct and consistent answers with minimal latency. It does not ask you to accept either approximate answers or eventual consistency. Whenever Materialize answers a query, that answer is the correct result on some specific (and recent) version of your data. Materialize does all of this by recasting your SQL92 queries as dataflows, which can react efficiently to changes in your data as they happen. Materialize is powered by timely dataflow, which connects the times at which your inputs change with the times of answers reported back to you.

We support a large fraction of PostgreSQL, and are actively working on supporting more builtin PostgreSQL functions.

Example:

 Views define commonly reused subqueries.
CREATE VIEW revenue (supplier_no, total_revenue) AS
    SELECT
        l_suppkey,
        SUM(l_extendedprice * (1 - l_discount))
    FROM
        lineitem
    WHERE
        l_shipdate >= DATE '1996-01-01'
        AND l_shipdate < DATE '1996-01-01' + INTERVAL '3' month
    GROUP BY
        l_suppkey;

-- Materialized views are maintained automatically, and can depend on non-materialized views.
CREATE MATERIALIZED VIEW tpch_q15 AS
  SELECT
    s_suppkey,
    s_name,
    s_address,
    s_phone,
    total_revenue
FROM
    supplier,
    revenue
WHERE
    s_suppkey = supplier_no
    AND total_revenue = (
        SELECT
            max(total_revenue)
        FROM
            revenue
    )
ORDER BY
    s_suppkey

https://github.com/MaterializeInc/materialize

​​A modified browser built using Electron that helps in responsive web development.

Features
- Mirrored User-interactions across all devices.
- Customizable preview layout to suit all your needs.
- One handy elements inspector for all devices in preview.
- 30+ built-in device profiles with option to add custom devices.
- One-click screenshot all your devices.
- Hot reloading supported for developers.

https://github.com/manojVivek/responsively-app

#js

Ormolu is a formatter for #haskell source code. The project was created with the following goals in mind:

- Using GHC's own parser to avoid parsing problems caused by haskell-src-exts.
- Let some whitespace be programmable. The layout of the input influences the layout choices in the output. This means that the choices between single-line/multi-line layouts in each particular situation are made by the user, not by an algorithm. This makes the implementation simpler and leaves some control to the user while still guaranteeing that the formatted code is stylistically consistent.
- Writing code in such a way so it's easy to modify and maintain.
- Implementing one “true” formatting style which admits no configuration.
- That formatting style aims to result in minimal diffs while still remaining very close to “conventional” Haskell formatting people use.
- Choose a style compatible with modern dialects of Haskell. As new Haskell extensions enter broad use, we may change the style to accomodate them.
- Idempotence: formatting already formatted code doesn't change it.
- Be well-tested and robust to the point that it can be used in large projects without exposing unfortunate, disappointing bugs here and there.

https://github.com/tweag/ormolu

​​Introducing @nuxt/content

The content/ directory for your #nuxt app, acting as a git-based headless CMS.

- #vue components in Markdown
- Powerful QueryBuilder API
- Handles MD, CSV, YAML, JSON
- Blazing fast hot reload in dev
- Syntax highlighting

https://content.nuxtjs.org/

#js #ts

​​sql-lint is a linter for SQL, it brings back any error from the supported servers as well as custom errors written for sql-lint.

sql-lint will show errors about the following things (and more)

- DELETE statements missing WHERE clauses
- DROP/TRUNCATE/ALTER/CREATE statements with invalid options specified
- Odd code points in queries
- Invalid arguments to the LIMIT clause
- Any SQL server errors

sql-lint brings back errors from the supported servers too. It will catch any error from the server. these include but are not limited to:

- Unknown columns on a table
- A non existent database
- A non existent table
- Syntax errors

https://github.com/joereynolds/sql-lint

#ts #sql

Modern Database Access for TypeScript & Node.js
Prisma is an open-source database toolkit. It replaces traditional ORMs and makes database access easy with an auto-generated query builder for TypeScript & Node.js.

Prisma makes database access easy with an auto-generated and type-safe query builder that's tailored to your database schema. It provides an alternative to traditional ORMs and SQL query builders. Start with a new database or introspect an existing one to get started.

Also supports schema migrations.

https://www.prisma.io/

Personal opinion: it looks amazing! We need something like this in Python!

#ts #js

​​A crawler for automated functional testing of #python web applications.

Crawling a server-side-rendered web application is a low cost way to get low quality test coverage of your JavaScript-light web application.

If you have only partial test coverage of your routes, but still want to protect against silly mistakes, then this is for you.

Why should I use this? Here's an example: Flaskr, the Flask tutorial application has 166 lines of test code to achieve 100% test coverage.

Using Python Testing Crawler in a similar way to the Usage example below, we can hit 73% with very little effort. Disclaimer: Of course! It's not the same quality or utility of testing! But it is better than no tests, a complement to hand-written unit or functional tests and a useful stopgap.

https://github.com/python-testing-crawler/python-testing-crawler

Your desktop is a mess? You cannot find anything in your downloads and documents? Sorting and renaming all these files by hand is too tedious? Time to automate it once and benefit from it forever with organize!

Config example:

  - folders: ~/Downloads
    subfolders: true
    filters:
      - extension: pdf
    actions:
      - echo: "Found PDF!"
      - move: "~/Documents/pdfs"

https://github.com/tfeldmann/organize

#python

​​A next-generation curated knowledge sharing platform for data scientists and other technical professions.

The Knowledge Repo project is focused on facilitating the sharing of knowledge between data scientists and other technical roles using data formats and tools that make sense in these professions. It provides various data stores (and utilities to manage them) for "knowledge posts", with a particular focus on notebooks (R Markdown and Jupyter / IPython Notebook) to better promote reproducible research.

For more information about the motivation and inspiration behind this project, we encourage you to read our Medium Post: https://medium.com/airbnb-engineering/scaling-knowledge-at-airbnb-875d73eff091

https://github.com/airbnb/knowledge-repo

#python #ds #docops

A process for automating #docker container base image updates.

With watchtower you can update the running version of your containerized app simply by pushing a new image to the Docker Hub or your own image registry. Watchtower will pull down your new image, gracefully shut down your existing container and restart it with the same options that were used when it was deployed initially. Run the watchtower container with the following command:

 docker run -d \
    --name watchtower \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower

https://github.com/containrrr/watchtower

#go #devops

Pyanalyze is a tool for programmatically detecting common mistakes in #python code, such as references to undefined variables and some categories of type mismatches. It can be extended to add additional rules and perform checks specific to particular functions.

It was opensourced just yesterday, so its still WIP.

https://github.com/quora/pyanalyze

​​Listens to changes in a PostgreSQL Database and broadcasts them over websockets.

It works like this:

1. the Phoenix server listens to PostgreSQL's replication functionality (using Postgres' logical decoding)
2. it converts the byte stream into JSON
3. it then broadcasts over websockets.

https://github.com/supabase/realtime

#elixir

​​#django gRPC framework is a toolkit for building gRPC services, inspired by djangorestframework.

https://github.com/fengsp/django-grpc-framework

#python

​​Super-fast #ts / #js compiler written in #rust

It consumes a javascript or typescript file which uses recently added features like async-await and emits javascript code which can be executed on old browsers. It's 20x faster than babel.

https://github.com/swc-project/swc

​​Lightweight, fault-tolerant message streams.

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. The goal of Liftbridge is to provide a message-streaming solution with a focus on simplicity and usability. Use it as a simpler and lighter alternative to systems like Kafka and Pulsar or to add streaming semantics to an existing NATS deployment.

https://github.com/liftbridge-io/liftbridge

#go

​​The webworkers driven UI #js framework.

This is a new paradigm. If you want to enter a new era of making better Web Based User Interfaces, the following concepts will be addictive.

What if a framework & all the apps you build are running inside a separate thread (web worker)? You get extreme performance!

https://github.com/neomjs/neo

WebPipe allows you to pipe from your servers to the browser!

Just a single command to run:

docker run --rm --publish 8000:8000 hyperngn/webpipe

https://github.com/hyperngn/webpipe

#elixir

​​snakeware is a free Linux distro with a Python userspace inspired by the Commodore 64. You are booted directly into a Python interpreter, which you can use to do whatever you want with your computer.

The idea is that a #python OS would be fun to use and very easy to contribute to. Even relative beginners might be able to find ways to meaningfully contribute apps and other code to this distro.

We also are not going to be using any other huge and opaque software such as systemd, etc. The goal is to eventually have a usable set of userspace apps and utilities written entirely in Python, because Python is fun and it Just Werks.

https://github.com/joshiemoore/snakeware