Contextual embeddings have revolutionized NLP, but are highly computationally expensive. In this work we focus on the question of when contextual embeddings are worth their cost, versus when it is possible to use more efficient word representations without…