StyleBabel: Artistic Style Tagging and Captioning
Authors:
Dan Ruta,
Andrew Gilbert,
Pranav Aggarwal,
Naveen Marri,
Ajinkya Kale,
Jo Briggs,
Chris Speed,
Hailin Jin,
Baldo Faieta,
Alex Filipkowski,
Zhe Lin,
John Collomosse
Abstract:
We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by `Grounded Theory': a qualitative approach that enables annotation while co…
▽ More
We present StyleBabel, a unique open access dataset of natural language captions and free-form tags describing the artistic style of over 135K digital artworks, collected via a novel participatory method from experts studying at specialist art and design schools. StyleBabel was collected via an iterative method, inspired by `Grounded Theory': a qualitative approach that enables annotation while co-evolving a shared language for fine-grained artistic style attribute description. We demonstrate several downstream tasks for StyleBabel, adapting the recent ALADIN architecture for fine-grained style similarity, to train cross-modal embeddings for: 1) free-form tag generation; 2) natural language description of artistic style; 3) fine-grained text search of style. To do so, we extend ALADIN with recent advances in Visual Transformer (ViT) and cross-modal representation learning, achieving a state of the art accuracy in fine-grained style retrieval.
△ Less
Submitted 11 March, 2022; v1 submitted 10 March, 2022;
originally announced March 2022.
ALADIN: All Layer Adaptive Instance Normalization for Fine-grained Style Similarity
Authors:
Dan Ruta,
Saeid Motiian,
Baldo Faieta,
Zhe Lin,
Hailin Jin,
Alex Filipkowski,
Andrew Gilbert,
John Collomosse
Abstract:
We present ALADIN (All Layer AdaIN); a novel architecture for searching images based on the similarity of their artistic style. Representation learning is critical to visual search, where distance in the learned search embedding reflects image similarity. Learning an embedding that discriminates fine-grained variations in style is hard, due to the difficulty of defining and labelling style. ALADIN…
▽ More
We present ALADIN (All Layer AdaIN); a novel architecture for searching images based on the similarity of their artistic style. Representation learning is critical to visual search, where distance in the learned search embedding reflects image similarity. Learning an embedding that discriminates fine-grained variations in style is hard, due to the difficulty of defining and labelling style. ALADIN takes a weakly supervised approach to learning a representation for fine-grained style similarity of digital artworks, leveraging BAM-FG, a novel large-scale dataset of user generated content groupings gathered from the web. ALADIN sets a new state of the art accuracy for style-based visual search over both coarse labelled style data (BAM) and BAM-FG; a new 2.62 million image dataset of 310,000 fine-grained style groupings also contributed by this work.
△ Less
Submitted 17 March, 2021;
originally announced March 2021.
