Abstract
Sketching is an intuitive process which has been used throughout human history as a communication tool. Due to the recent proliferation of touch-screen devices, free-hand sketching has now become more pervasive than ever before: sketches can be drawn at anytime and anywhere on a smartphone using one’s finger. With this mass usage and an imminent commercial potential, amateur human free-hand sketching has attracted increasing attention from the research community. Various sketch related problems have been studied, including sketch recognition, sketch based image retrieval, and sketch synthesis. A fundamental challenge in analyzing free-hand sketches is that sketches drawn by different people for the same object category/instance often differ significantly, especially in their levels of abstraction. It does not seem to matter if a particular sketch is drawn out of memory, given the object name or it is drawn using a reference photo. However, we notice that detailed sketches with low levels of abstraction are easily recognizable, but as the abstraction level increases and the details in the depicted sketches become scarce, it becomes harder to recognize the object in the sketches. Furthermore, the variations of abstraction level can also be influenced by what an individual person perceives as salient in the context of a particular goal or objective. The peculiarity of sketches in fact consists of their sparse and abstract representation which highlights specific visual features while downplaying others.
Depending on what the goal or the objective is, different parts of the sketch may be drawn with prominence while others are reduced or completely removed.
Another peculiar characteristic of free-hand sketches is that they are distinctively different to photos. Photos are static and pixel-perfect two-dimensional visual representations of the world (e.g. scenes, objects). Sketches on the other hand are the result of a dynamic drawing process consisting of a sequence of strokes. While it is possible to represent a sketch as a 2D pixel image, the direct adaptation of models initially proposed for 2D photos often fail or provide poor results due to the sparse nature of sketches with no colour or texture information to exploit. This underlines the need for sketch-specific solutions that can exploit sketch-specific characteristics to solve a sketch-specific problem such as abstraction.
In this work we tackle the problem of sketch abstraction by first defining it as a process of trade-off between recognizability and brevity/compactness (number of strokes). Based on this insight, we propose two distinct approaches to solve the abstraction problem: elimination-based abstraction (EA) and selection-based abstraction (SA). In the first approach the EA model learns to find the most compact subset of input strokes for which the sketch is still recognizable. It does so by observing parts of the sketch (strokes) one by one following the original human drawing order, and deciding for each part whether it should be kept or removed from the abstracted output sketch. The impact of any given stroke removal on recognizability is interdependent on which other strokes are kept/removed. We thus model this dependency as a sequential decision making problem described by the Markov Decision Process (MDP). In the second approach the objective is again to find a compact subset of strokes for which the sketch is still recognizable, but this is achieved by applying an inverse strategy to the EA approach. Here we aim to learn which strokes should be selected, rather than removed, from the input sketch to form the most compact abstracted sketch representation that doesn’t lose its recognizability. The input is processed holistically, rather than being constrained by the original order, and the output is built sequentially by picking one stroke among all the available strokes at each time step. In both approaches the sequential decision process is formalized as the Markov Decision Process (MDP), which is trained with the reinforcement learning (RL) framework due to the lack of relevant stroke-level annotation.
We introduce a novel goal-driven abstraction task, that aims to extract/preserve particular parts of the input sketch according to a specific goal. This implies that the same input can output a different abstracted sketch depending on the abstraction goal. Due to the particular RL based architecture of the aforementioned approaches for sketch abstraction (EA and SA), the goal-driven sketch abstraction task can easily be amalgamated with these proposed methods by solely adjusting the reward function.
We also explore the potential of applying the abstraction problem and the proposed solutions to other sequential domains, such as videos and text. We do this by following the same problem formulation and model architecture of the EA/SA approaches proposed for sketches, with the substitution of stroke (coordinates) data with video-shots (frames) and sentences (words).
Finally, we explore the potential of combining sketch abstraction and synthesis tasks, so that a model can not only autonomously generate a sketch of given visual concept but can also achieve recognizable rendition with very few strokes. In order to effectively evaluate the performance of such a drawing agent, we set up a Pictionary-like game that we name Pixelary, where the competitors are pixel-synchronized instead of time. We imagine a common game scenario where two parties (our drawing agent and a human competitor) simultaneously sketch a common visual concept – stroke-by-stroke – with a judge in the background constantly guessing, and the winner is the one whose sketch the judge recognizes first. In achieving this goal, we importantly advance the line of sketch research by contributing an agent that not only knows how to sketch – but more importantly masters the dynamics of sketching, so as to surpass humans at conveying meaning quickly.
For all the aforementioned tasks, we perform thorough experiments on various publicly available datasets. The quantitative and qualitative results are reported to demonstrate the effectiveness of all the proposed solutions.