Tips for releasing research code in ML
with official NeurIPS 2020 recommendations

In repo you can find template that you can use for releasing ML research repositories. The sections in the template were derived by looking at existing repositories, seeing which had the best reception in the community, and then looking at common components that correlate with popularity.

The ML Code Completness Checklist consists of five items:
1 Specification of dependencies
2 Training code
3 Evaluation code
4 Pre-trained models
5 README file including table of results accompanied by precise commands to run/produce those results

Also, you can find additional awesome resources for releasing research code like: where to hosting pretrained models files, standardized model interfaces, results leaderboards, and etc.

github: https://github.com/paperswithcode/releasing-research-code

​​#StyleGan2 applied to maps

Ever imagined what happens in Inception on bigger scale?

#mapdreamer #GAN

Dear audience, if you want to share any links don’t hesitate to paste them into @opendatasciencebot

​​ResNeSt: Split-Attention Networks

A novel variation of ResNet architecture that outperforms other networks with similar model complexities.
Usually, downstream applications use the ResNet or one of its variants as the backbone CNN. Its simple and modular design can be easily adapted to various tasks. However, since ResNet models are originally designed for image classification, they may not be suitable for various downstream applications because of the limited receptive-field size and lack of cross-channel interaction.

Main contributions of the paper:
- Split-Attention block. Each block divides the feature-map into several groups (along the channel dimension) and finer-grained subgroups or splits, where the feature representation of each group is determined via a weighted combination of the representations of its splits. By stacking several Split-Attention blocks, they get a ResNet-like network called ResNeSt (S stands for “split”). This architecture requires no more computation than existing ResNet-variants, and is easy to be adopted as a backbone for other vision tasks
- a lot of large scale benchmarks on image classification and transfer learning.

Models utilizing a ResNeSt backbone are able to achieve SOTA performance on several tasks, namely: image classification, object detection, instance segmentation, and semantic segmentation.
ResNeSt-50 achieves 81.13% top-1 accuracy on ImageNet using a single crop-size of 224 × 224, outperforming previous best ResNet variant by more than 1% accuracy


Paper: https://arxiv.org/abs/2004.08955
Github: https://github.com/zhanghang1989/ResNeSt

#computervision #deeplearning #resnet #image #backbone #downstream #sota

COVID-19 Challenge final stage begins! Participants have to build an algorithm that most accurately predicts the dynamics of the number of reported cases of COVID-19 over the next 7 days. Last stage prizes are 1 000 000 RUB. 5 best forecasts + 5 best public solutions will be rewarded.

The objective of the competition is to draw attention to the forecasts of the coronavirus pandemic. Perhaps while solving this problem, you could find problems in the data sources or make a suitable forecast based on the most reliable data. Remember, we are developing an open science in ODS.ai by creating new and testing the existing forecasting methods. Only solving the tasks based on the open and public benchmark we can test and compare different approaches, as well as come to the best practices, and make them accessible to the entire research community.

✅ Click the link
🤓 Act
👏 May the best win!

Nvidia AI Noise Reduction

#Nvidia launches #KrispAI competitor Noise Reduction by AI on RTX Videocards.

Seems it works significantly better then other that kind of tools. But it needs to have Nvidia RTX officially.

But it possible to run it on older cards. The instruction is below. Or you can just download already hacked executable (also, below)

Setup Guide: https://www.nvidia.com/en-us/geforce/guides/nvidia-rtx-voice-setup-guide/
The instruction: https://forums.guru3d.com/threads/nvidia-rtx-voice-works-without-rtx-gpu-heres-how.431781/
Executable (use it on your own risk): https://mega.nz/file/CJ0xDYTB#LPorY_aPVqVKfHqWVV7zxK8fNfRmxt6iw6KdkHodz1M

#noisereduction #soundlearning #dl #noise #sound #speech #nvidia

Training with quantization noise for extreme model compression

It is a new technique to enable extreme compression of models that still deliver high performance when deployed in practical applications mimics the effect of quantization during training time.
This method delivers performance that nearly matches that of the original uncompressed models while reducing the memory footprint by 10x to 20x. This significantly exceeds the 4x compression with int8 currently available in both PyTorch and Tensorflow. Quant-Noise can be used to shrink models even further – by more than 50x – in use cases where greater performance trade-offs are acceptable. Quant-Noise changes model training only by adding a regularization noise similar to dropout, with no impact on either the convergence rate or training speed.

At training time during the forward pass, it takes a subset of the weights and then randomly applies simulated quantization noise. This makes the model resilient to quantization and enables large compression ratios without much loss in accuracy.

Quant-Noise is applied to only a subset of the weights. This method has the advantage that the unbiased gradients still flow from the weights that are unaffected by the noise.

The authors demonstrated that their framework compresses the SOTA EfficientNet-B3 model from ~50 MB to 3.3 MB while achieving 80% top-1 accuracy on ImageNet, compared with 81.7% for the uncompressed model. Compress RoBERTa Base model from 480 MB to 14 MB while achieving 82.5% on MNLI, compared with 84.8% for the original model.


blogpost: https://ai.facebook.com/blog/training-with-quantization-noise-for-extreme-model-compression/
paper: https://arxiv.org/abs/2004.07320
github: https://github.com/pytorch/fairseq/tree/master/examples/quant_noise

#quantization #compression #shrinking

​​Stealing pics from the best😎

Caption text: Python at work

​​StarGAN v2 code release on GitHub

The better news is if you put a human into the animal model you do in fact get out a feline version of the human, and it's even wearing a suit.

GitHub: https://github.com/clovaai/stargan-v2
ArXiV: https://arxiv.org/abs/1912.01865
YouTube: https://www.youtube.com/watch?v=0EVh5Ki4dIY&feature=youtu.be

#GAN #StarGAN #PyTorch

​​The Ingredients of Real World Robotic Reinforcement Learning

Blog post describing experiments on applying #RL in real world.

Blog post: https://bair.berkeley.edu/blog/2020/04/27/ingredients/
Paper: https://openreview.net/forum?id=rJe2syrtvS

#DL #robotics

​​ICASSP 2020 – FREE
45th International Conference on Acoustics, Speech, and Signal Processing

Registration includes full access to the virtual conference and all sessions, virtual patron and exhibitor experiences, as well as the conference app and any live and asynchronous discussion forums, and an electronic download of the conference proceedings.

link: https://cmsworkshops.com/ICASSP2020/Registration.asp


#icassp #conference

​​Scheduled DropHead: A Regularization Method for Transformer Models

In this paper introduced DropHead, a structured dropout method specifically designed for regularizing the multi-head attention mechanism, which is a key component of the transformer, a SOTA model for various NLP tasks.

In contrast to the conventional dropout mechanisms which randomly drop units or connections, the proposed DropHead is a structured dropout method. It drops entire attention heads during training and It prevents the multi-head attention model from being dominated by a small portion of attention heads while also reduces the risk of overfitting the training data, thus making use of the multi-head attention mechanism more efficiently.


paper: https://arxiv.org/abs/2004.13342

#nlp #regularization #transformer

​​🎙🎶Improved audio generative model from OpenAI

Wow! OpenAI just released Jukebox – neural net and service that generates music from genre, artist name, and some lyrics that you can supply. It is can generate even some singing like from corrupted magnet compact cassette.

Some of the sounds seem it is from hell. Agonizing Michel Jakson for example or Creepy Eminiem or Celien Dion

#OpenAI 's approach is to use 3 levels of quantized variational autoencoders VQVAE-2 to learn discrete representations of audio and compress audio by 8x, 32x, and 128x and use the spectral loss to reconstruct spectrograms. And after that, they use sparse transformers conditioned on lyrics to generate new patterns and upsample it to higher discrete samples and decode it to the song.

The net can even learn and generates some solo parts during the track.

explore some creepy songs: https://jukebox.openai.com/
code: https://github.com/openai/jukebox/
paper: https://cdn.openai.com/papers/jukebox.pdf
blog: https://openai.com/blog/jukebox/

#openAI #music #sound #cool #fan #creepy #vae #audiolearning #soundlearning

​​🔥Consistent Video Depth Estimation

New algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video.

Obviously, later there will be various VR/AR effects based on this research. Looking forward to it.

Paper: https://arxiv.org/abs/2004.15021
Project site: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/
Video: https://www.youtube.com/watch?v=5Tia2oblJAg

Video lecture on geodesics

Geodesics generalize the idea of "straight lines" to curved spaces—like the circular arc an airplane takes across the globe. This video gives a crash course on geodesics, using geometric algorithms to help tell the story.

Video: https://youtu.be/uojNGbVtlsQ

#closeenough #othermath

ICLR 2020 Recordings

All recordings for papers and workshops are now available to everyone!

​​MLSUM: The Multilingual Summarization Corpus

The first large-scale MultiLingual SUMmarization dataset, comprising over 1.5M article/summary pairs in French, German, Russian, Spanish, and Turkish. Its complementary nature to the CNN/DM summarization dataset for English.

For each language, they selected an online newspaper from 2010 to 2019 which met the following requirements:
0 being a generalist newspaper: ensuring that a broad range of topics is represented for each language allows minimizing the risk of training topic-specific models, a fact which would hinder comparative cross-lingual analyses of the models.
1 having a large number of articles in their public online archive.
2 Providing human written highlights/summaries for the articles that can be extracted from the HTML code of the web page.

Also, in this paper, you can remember about similar other datasets

paper: https://arxiv.org/abs/2004.14900
github: https://github.com/recitalAI/MLSUM
Instructions and code will soon.

#nlp #corpus #dataset #multilingual