[Preview] Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models
Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, Michael Terry
CHI'22: ACM Conference on Human Factors in Computing Systems
Session: Natural Language
Abstract
In this paper, we present a natural language code synthesis tool, GenLine, backed by 1) a large generative language model and 2) a set of task-specific prompts that create or change code. To understand the user experience of natural language code synthesis with these new types of models, we conducted a user study in which participants applied GenLine to two programming tasks. Our results indicate that while natural language code synthesis can sometimes provide a magical experience, participants still faced challenges. In particular, participants felt that they needed to learn the model’s ``syntax,'' despite their input being natural language. Participants also struggled to form an accurate mental model of the types of requests the model can reliably translate and developed a set of strategies to debug model input. From these findings, we discuss design implications for future natural language code synthesis tools built using large generative language models.
WEB:: http://programs.sigchi.org/chi/2022/program/content/68967
Presentation Video:: https://www.youtube.com/watch?v=dDqO8-Zb_pg
DOI:: https://doi.org/10.1145/3491102.3501870
Video previews for CHI 2022 papers