StylerDALLE: Language-Guided Style Transfer Using a Vector-Quantized Tokenizer of a Large-Scale Generative Model

Zipeng XuEnver SanginetoNicu Sebe

Mar 2023

摘要

Despite the progress made in the style transfer task, most previous workfocus on transferring only relatively simple features like color or texture,while missing more abstract concepts such as overall art expression orpainter-specific traits. However, these abstract semantics can be captured bymodels like DALL-E or CLIP, which have been trained using huge datasets ofimages and textual documents. In this paper, we propose StylerDALLE, a styletransfer method that exploits both of these models and uses natural language todescribe abstract art styles. Specifically, we formulate the language-guidedstyle transfer task as a non-autoregressive token sequence translation, i.e.,from input content image to output stylized image, in the discrete latent spaceof a large-scale pretrained vector-quantized tokenizer. To incorporate styleinformation, we propose a Reinforcement Learning strategy with CLIP-basedlanguage supervision that ensures stylization and content preservationsimultaneously. Experimental results demonstrate the superiority of our method,which can effectively transfer art styles using language instructions atdifferent granularities. Code is available athttps://github.com/zipengxuc/StylerDALLE.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域

· Computer Science · CV · CLIP · Translation · Style Transfer