Gaudí: Conversational Interactions with Deep Representations to Generate Image Collections.
Published in NeurIPS Workshop on Machine Learning for Creativity and Design, 2021
Recommended citation: Victor S. Bursztyn, Jennifer Healey, and Vishwa Vinay. 2021. Gaudí: Conversational Interactions with Deep Representations to Generate Image Collections. In the 2021 NeurIPS Workshop on Machine Learning for Creativity and Design. https://neuripscreativityworkshop.github.io/2021/accepted/ncw_104.pdf
Based on recent advances in realistic language modeling (GPT-3) and cross-modal representations (CLIP), Gaudí was developed to help designers search for inspirational images using natural language. In the early stages of the design process, with the goal of eliciting a client’s preferred creative direction, designers will typically create thematic collections of inspirational images called “mood-boards”. Creating a mood-board involves sequential image searches which are currently performed using keywords or images. Gaudí transforms this process into a conversation where the user is gradually detailing the mood-board’s theme. This representation allows our AI to generate new search queries from scratch, straight from a project briefing, following a theme hypothesized by GPT-3. Compared to previous computational approaches to mood-board creation, to the best of our knowledge, ours is the first attempt to represent mood-boards as the stories that designers tell when presenting a creative direction to a client.