​​Self-training Improves Pre-training for Natural Language Understanding
Facebook AI & Stanford

Most semi-supervised NLP approaches require specifically in-domain unlabeled data. It means that for the best results, the unlabeled portion of the data which we want to use for semi-supervised training must be from the same domain as the annotated dataset.

This paper proposes SenAugment - a method that constructs task-specific in-domain unannotated datasets on the fly from the large external bank of sentences. So for any new NLP task where we have only a small dataset, we don't need to bother anymore to collect a very similar unannotated dataset if we want to use semi-supervised training.
Now we can sort of cheat to improve the performance of an NLP model on almost any downstream task using Self-training (which is also called Teacher-Student training):
1. We retrieve the most relevant sentences (few millions of them) for the current downstream task from the external bank. For retrieval we use the embedding space of a sentence encoder - Transformer, pre-trained with masked language modeling and finetuned to maximize cosine similarity between similar sentences.
2. We train the teacher model - a RoBERTa-Large model finetuned on the downstream task.
3. Then we use a teacher model to annotate the retrieved unlabeled in-domain sentences. We perform additional filtering by keeping the ones that have the high-confident predictions.
4. As our student model, we then finetune a new RoBERTa-Large using KL-divergence on the synthetic data by considering the post-softmax class probabilities as labels (i.e., not only the most confident class but the entire class distribution is used as a label for every sentence).

Such a self-training procedure significantly boosts the performance compared to the baseline. And the positive effect is higher when fewer GT annotated sentences are available.

As a large-scale external bank of unannotated sentences, authors use CommonCrowl. In particular, they use a corpus with 5 billion sentences (100B words). Because of its scale and diversity, the sentence bank contains data from various domains and with different styles, allowing to retrieve relevant data for many downstream tasks. To retrieve the most relevant sentences for a specific downstream task, we need to obtain an embedding for the task. Several options exist: (1) average embeddings of all sentences in the training set; (2) average embeddings for every class; (3) keep original sentences embeddings.

📝 Paper
🛠 Code

#paper_explained #nlp

​​What happens if you augment your training dataset with a load of stylized images as well?

Someone trained a StyleGAN2-ada on the images augmented with style transfer and synced the output with audio 🎶.

Designing, Visualizing and Understanding Deep Neural Networks, CS182

Sergey Levine released his new lectures for deep learning class, CS182! This is an introductory deep learning course (advanced undergraduate + graduate) covering a broad range of deep learning topics. Prof. Levine is an Assistant Professor at UC Berkeley and is the head of the Robotic Artificial Intelligence and Learning Lab, I have posted about him a few months ago.

🔖 Course website
▶️ Lectures playlist

Neural Corgi 🤓

StyleGAN2-ADA trained on cute Corgi images. Looks amazing!

1. Scrape 350k Corgi images from Instagram.
2. Detect dogs using YOLOv3.
3. Remove small detections and dogs not facing in the camera.
4. Remove duplicates and crop the images. Around 130k 1024x1024 were obtained at this step.
5. Upsample crops to 1024 x 1024.
6. Train StyleGAN2-ADA for 5 million iterations for 18 days on Tesla V100.
7. Profit ?!

🌀 Colab
🛠Code and dataset

MacaquePose: A Novel “In the Wild” Macaque Monkey Pose Dataset

Recently, Computer vision for animals is getting more traction. Several works on this topic have already been discussed in this channel: post [1], post [2] , post [3].

❓ Why?
Pose estimation is fundamental for analyzing the relationship between the animal’s behaviors and its brain functions and malfunctions. And Macaque monkeys are excellent non-human primate models, especially for studying neuroscience.
Another possible application is Instagram / Snapchat masks and effects for your cute quadruple friends.

🏛 Dataset
This dataset provides keypoints for macaques in naturalistic scenes, it consists of 13k images and 16k monkey instances.
- 17 keypoints and instance segmentation for each monkey in COCO format.
- Annotations are of high quality because crowd-sourced annotations were curated and refined by 8 researchers working specifically with macaques.

I totally need glasses that move with my eyebrows. (c) Yann LeCun

The quality is wicked because of the pesky twitter compression.

Cute RoboCat 🐈 learned how to track objects. Fun application of Computer Vision. Is anyone among my subscribers working in robotics?

Source: IG @bio.makers