Hendrik Strobelt, Benjamin Hoover, Arvind Satyanarayan, Sebastian Gehrmann
This work has been developed in part during BigScience, the paper was accepted at EMNLP demo track 2021 and won NeurIPS best demo award 2020.
Paper: https://arxiv.org/abs/2111.01582
Intro Video: https://vimeo.com/640945461
GitHub: https://github.com/HendrikStrobelt/LMdiff
While different language models are ubiquitous in NLP, it is hard to contrast their outputs and identify which contexts one can handle better than the other. To address this question, we introduce LMdiff, a tool that visually compares probability distributions of two models that differ, e.g., through finetuning, distillation, or simply training with different parameter sizes. LMdiff allows the generation of hypotheses about model behavior by investigating text instances token by token and further assists in choosing these interesting text instances by identifying the most interesting phrases from large corpora. We showcase the applicability of LMdiff for hypothesis generation across multiple case studies. A demo is available at