Sketches are distinctly different to photos. They are highly abstract and exhibit a severe lack of visual cues. Prior works have therefore explored additional traits unique to sketches to help recognition such as stroke ordering. In this paper, we pioneer in studying the role of structure in sketches, for the task of sketch recognition. In particular, we propose a novel graph representation specifically designed for sketches, which follows the inherent hierarchical relationship ("segment-stroke-sketch") of sketching elements. By conforming to this hierarchy, we also introduce a joint network that encapsulates both the structural and temporal traits of sketches for sketch recognition, termed S(3)Net. S(3)Net employs a recurrent neural network (RNN) to extract segment-level features, followed by a graph convolutional network (GCN) to aggregate them into sketch-level features. The RNN first encodes temporal cues in sketches while its outputs are used as node embedding to construct a hierarchical sketch-graph. The GCN module then takes in this sketch-graph to produce a structure-aware embedding for sketches. Extensive experiments on the QuickDraw dataset, exhibit superior performance over state-of-the-arts, surpassing them by over 4%. Ablative studies further demonstrate the effectiveness of the proposed structural graph for both inter-class, and intra-class feature discrimination. Code is available at: https://github.com/yanglan0225/s3net.
- (SNET)-N-3: GRAPH REPRESENTATIONAL NETWORK FOR SKETCH RECOGNITION
- Lan Yang - Beijing Univ Posts & Telecommun, Beijing, Peoples R ChinaAneeshan Sain - Univ Surrey, SketchX, CVSSP, Guildford, Surrey, EnglandLinpeng Li - Beijing Univ Posts & Telecommun, Beijing, Peoples R ChinaYonggang Qi - Beijing Univ Posts & Telecommun, Beijing, Peoples R ChinaHonggang Zhang - Beijing Univ Posts & Telecommun, Beijing, Peoples R ChinaYi-Zhe Song - Univ Surrey, SketchX, CVSSP, Guildford, Surrey, England
- 2020 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO (ICME)
- IEEE International Conference on Multimedia and Expo
- IEEE
- 6
- 01/01/2020
- 61806184; 61601042 / National Natural Science Foundation of China; National Natural Science Foundation of China (NSFC) BUPT Excellent PhD Student Foundation
- 99777767302346
- School of Computer Science and Electronic Engineering; Centre for Vision, Speech & Signal Processing (CVSSP)
- English
- Conference proceeding