Sketch Retrieval based on Qualitative Shape Similarity Matching: Towards a Tool for Teaching Geometry to Children

An approach for a query-by-sketch system on qualitative shape information for image retrieval in databases is proposed and evaluated. The use of qualitative methods for shape description allows the gathering of semantic information from the sketches. The qualitative description and recognition of sketches are evaluated in order to verify that it is possible to use the proposed qualitative method for the de-velopment of a learning application for children.


Introduction
A sketch is a freehand drawing which is commonly employed to represent the essentials of an idea. Sketches are used every day in design, architecture, arts and software engineering, and also in nontechnical situations such as providing orientation instructions in a city, etc.
Previous work was successful in using sketches as spatial abstractions to represent maps in geographic information [11] and in robot navigation [14]. In the liter-* Corresponding author: Lledó Museros, Universitat Jaume I, Avg. Vicent Sos Baynat s/n, E-12071 Castelló de la Plana, Spain. ature, query-by-sketch approaches based on qualitative representations are popular for retrieval in geographic databases. Using a sketch containing spatial relations as a means of querying a geographic database was first proposed by Egenhofer [4]. Ferguson et al. [9] developed a sketch interface for military course-of-action diagrams, which supported queries using spatial relationships. Fogliaroni et al. [10] proposed several approaches to reduce the relation space and enable qualitative spatial queries in spatial databases to support query-by-sketch. Al-Salman et al. [1] developed an intuitive sketching tool for users to contribute and query information in disaster scenarios via their mobile devices.
Since the schematic nature of sketches makes qualitative representation methods fit naturally, some approaches appear in the literature which use qualitative techniques to describe the shape of a sketch [16,15]. Kuijpers et al. [16] developed an algorithm for polyline (and polygon) similarity calculus based on the doublecross (DC) [12] orientation model which was applied to query-by-sketch polyline databases and classification of terrain features. Gottfried [15] obtained sketches of images containing objects and river maps and used qualitative relations of orientation, inspired also by the DC model, between the line segments of the contour of a shape to calculate a similarity measure between the images. Thus, qualitative approaches have proved their effectiveness to image query-by-sketch retrieval.
In this paper, a novel approach for query-by-sketch based on qualitative shape information for image retrieval in databases is proposed and evaluated. The pro-posed qualitative shape similarity model is not based on the DC orientation model, as the previous approaches, but on qualitative features of shape given by Falomir et al. [7] which has shown good performance in fields such as mosaic assembling [8] or icon retrieval [19].
A crucial benefit of using a qualitative approach to process the sketches is the possibility to gather semantic rich information out of the sketches, which can be exploited in useful ways. In order to describe and recognize a sketch, a similarity measure is presented which is used to compare a sketch against a drawing in the database and it is also able to detect the differences between the shapes compared.
Moreover, it is worth noting that the qualitative descriptions presented can be translated to natural language and a narrative description can be provided to an end-user for reading or listening to by means of a speech synthesizer program [6].
These advantages allow the use of qualitative shape description techniques for the implementation of a learning system to support the teaching of geometric shapes to children. The proposed approach may be implemented in an Android system and then be used in a tablet, embedded in an application used to teach children how to draw a geometric shape. The sketch made by a child can be qualitatively described and compared with the ones already described in a database, and the differences can be described using natural language in order to explain the differences between the target geometric shape and the sketch. In this context, the main aim of this paper is to present the techniques that makes such an application feasible.
The steps of the approach are summarized in Fig. 1. First, the sketch is qualitatively described and its description is matched against a database of images of drawings using a similarity calculus. The resulting list of similar images is then presented to the user both graphically and using an automatically generated natural language description.
The remainder of this paper is organized as follows. The qualitative model for shape description used is outlined in Section 2. Section 3 explains how to obtain natural language descriptions from QSDs specifications. The shape similarity calculus is given in Section 4. The scenario, the performed tests and results obtained are shown in Section 5. Finally, conclusions and future work are drawn.

Qualitative Shape Description (QSD)
The Qualitative Shape Description (QSD) method [7] is based on the relevant points of the boundary of a shape. For bitmap images, this boundary is obtained using standard image segmentation algorithms, and then the slope of the pixels at the object boundary is analysed. For vectorial images, the relevant points are obtained by interpreting the drawing primitives.
Each of these relevant points is described by a set of four qualitative features as follows: -Compared Length (L) of the two edges connected by P, described as: {much shorter (msh), halflength (hl), a bit shorter (absh), similar length (sl), a bit longer (abl), double length (dl), much longer (ml)}; -Convexity (C), described as: {convex, concave}.
An example of the qualitative shape description of an object composed of 6 relevant points which connects straight lines and curves and defines different angles and lengths is given in Fig. 2.

Qualitative Object Description in Natural
Language (QODNL) According to geometric principles, objects described by qualitative features are characterized by a set of three elements: [Name, Regularity, Convexity] Regarding objects without curves, these elements are defined as follows. Name is given by the quantity of relevant points of the object (triangle, quadrilateral, pentagon, hexagon, heptagon, octagon, polygon); Regularity indicates if the object has all the same qualitative angles and all the edges of similar length (regular), or not (irregular); and Convexity indicates if the object has a concave angle (then it is concave) or not (then it is convex). Triangular objects are characterized as right/obtuse/acute according to the kind of angles, and as equilateral/isosceles/scalene according to the relation of length between the edges. Quadrilateral objects are characterized more accurately as square, rect-angle or rhombus depending on the compared length between the edges and on the kind of angles.
The characterization of the objects with curves are defined as follows. Name takes the next options depending on its properties: curved-polygon (it has at least one curvature-point and at least one line-line), polycurve (all the relevant points are curvature-points, curve-curve, curve-line or line-curve points), circle (a polycurve with four relevant points, two of them defined as semicircular) and ellipse (a polycurve with four relevant points, two of them defined as points of curvature). Regularity: circles and ellipses are considered regular and other objects irregular. Convexity of objects with curves is defined in the same way as for objects with straight edges. A more detailed description can be seen in [5].
In order to obtain a Qualitative Object Description in natural Language, the qualitative descriptors defined by the QSD approach are used and organized in a context-free grammar (G) built on the following parameters: where -V is an alphabet of symbols that are non-terminals; -Σ is an alphabet of terminal symbols (qualitative labels or words), disjoint with V ; -P ⊆ V × (V ∪ Σ) * is the set of production rules 1 ; -QODNL ∈ V is the initial symbol of the grammar; The grammar G(QODNL) [6], simplified here to show only the features of shape where λ is the empty string, is as follows: QODNL → Ob jID is a Regularity Convexity Name. Ob jectQSD Ob jectQSD → Its shape has M RegularEdges defining Amplitude | Its shape has M relevant points. RPsQSD | λ RegularEdges → equal edges | curves EC → a line to a curve | a curve to a line 2Curves → Point M joins two curves in a C and TC angle.
The language generated by the G(QODNL) grammar is defined as follows: The G(QODNL) language describes objects in two levels of detail: (1) a sentence describing the main features of an object within the image or (2) a detailed description including both the general details of the object and also all its features of shape: angles, curvature, length, etc. An illustrative example of the more general level of detail is given in Table 1.

Qualitative Shape Similarity
Freksa [13] determined that two qualitative terms are conceptual neighbours if "one can be directly transformed into another by continuous deformation". Therefore, angles acute and right are conceptual neighbours since an extension of the angle acute causes a direct transition to the angle right.
Hence, Conceptual Neighbourhood Diagrams (CNDs) can be described as graphs containing: (i) nodes that map to a set of individual relations defined on intervals and (ii) paths connecting pairs of adjacent nodes that map to continuous transformations which can have weights assigned to them in order to establish priorities. For each of the features in QSD, a CND is defined in Fig. 3. Dissimilarity matrices in Tables 2-5 map the pairs of nodes in each CND to the minimal path distance between them.

A Similarity between Qualitative Shape
Descriptions (QSDs) As explained in Section 2, the qualitative shape of an object is described by means of all its relevant points (RPs). Therefore, in order to define a similarity measure between shapes, first a similarity between relevant points must be obtained. Hence, given two relevant points, denoted by RP A and RP B , belonging to the shapes of the objects A and B respectively, a similarity between them, denoted by SimRP(RP A , RP B ), is defined as: where ds(i) and Ds(i) denote the dissimilarity between relevant points and the maximum dissimilarity with respect to the feature obtained from the dissimilarity matrix with I = {EC, A ∨ TC,C, L}, respectively. Hence, by dividing ds(i) and Ds(i) the proportion of dissimilarity related to feature of RP A and RP B is obtained, which is between 0 and 1. Furthermore, the parameter w i is the weight assigned to this feature, and it holds that w EC + w A + w L + w C = 1, w A = w TC and w i ≥ 0 for each f eature.
In order to compare two shapes A and B whose QSDs have the same number of relevant points (denoted by m), the similarity between A and B, denoted by SimQSD (A, B), is calculated from (1) as follows: Fixed an relevant point of A, RP i A , i = 1, ·· ·, m, the similarities between the pairs of relevant points of the set are calculated. Thus, In general, if the number of relevant points of the shapes A and B are n and m respectively, and assuming without loss of generality that n ≥ m, then there are n − m relevant points of A shape with no corresponding points in the B shape.
Let C the set of all possible way (combinations) to chose n − m relevant points of A. Hence, if c ∈ C, a new shape A c is considered such that A c is given by all the relevant points of A minus the n − m relevant points of A given by the c combination. Hence A c and B have the same number of relevant points and its similarity can be calculated as in the previous case.
Thus, the similarity between A and B is obtained as: More details and properties of this shape similarity calculus are given in [20].

Importance of the points of a sketch
Let F = {F i } i∈I a sketch set with I an index set, and the similarity SimQSD : F × F −→ R + between two sketches defined from the Qualitative Shape Description.
Let A ∈ F be a sketch with n relevant points, RP A i n i=1 . Given a fixed RP A i point, a new sketch A i is considered which is the same A sketch without the RP A i point. Removing a point can create a very difference between QSD(A) and QSD(A i ) (see Table 6).
Hence, a value s A i is defined as follows: This value is straightforward to interpret since i is high, that is, close to one, then QSD(A) and QSD(A i ) are very different. Hence, the elimination of the point RP A i has significantly modified A sketch, which implies that the RP A i point is very important in the Qualitative Shape Descriptions of A.
-If s A i is low, that is, close to zero, then QSD(A) and QSD(A i ) are not significantly different, which implies that the RP A i point is not very important in the QSD of A.
have been obtained. Hence, a weight of the RP A i point of A sketch, denoted by w A i , is given as follows: can be interpreted as being the same as the values s A i n i=1 , the only difference being that ∑ n i=1 w i = 1. an example of these weights is given in Table 6.

Cognitive Saliency of Each Qualitative Feature at a Relevant Point
The value ds(i) Ds(i) in (1) can be seen as the importance of changes in each feature of shape. Hence, from the dissimilarity matrices obtained from CNDs, the following maximums (Ds(i)) are obtained: for Convexity, 1; for Edge Connection, 2; for Angle and Type of Curvature, 4; and for Length, 6. As the value assigned to each change is 1, this means that each change in each feature has a different importance I in equation (1) and the following priorities among features are given: These priorities can be justified as being suitable for comparing shapes intuitively [7,5]. In Fig. 4 five shapes are shown (S1, S2, S3, S4 and S5) that exemplify these priorities. Convexity (C) is the feature that has the greatest priority because, when it changes, not only the boundary of the object changes, but also its interior (i.e. compare shapes S1 to S2 in which only the convexity of relevant point 2 changes). The Edge Connection (EC) is the second most important feature because it differentiates between curves and straight lines, which is also an important difference. For example, if we compare shapes S1 to S3, in which only the EC of relevant point 2 changes, we will see that they are more similar than S1 and S2 and than S2 and S3, in which both the EC and the C of 2 is different. The next most important feature is the Angle or Type of Curvature, because it characterises the shape of an object in a more significant way than the lengths of the edges, which usually depend on the angle they define. If we compare S3 and S4, the most perceptible difference is that the Angle of 2 is different, but the compared length between relevant points 3-4 and 4-0 is also different in both shapes, and this is less perceptible. Finally, note that it is also true that the more similar the number of relevant points between shapes, the higher the similarity, since S1-S4 are more similar to each other than any of them are to S5, which has one relevant point less than them.

Detecting the Differences in Shape by Correspondences of Relevant Points
The developed method, apart from calculating the similarity between two objects A and B with a different number of vertices, it also finds the correspondence of as many equivalent vertices of both shapes as possible. Therefore this method is able to detect the differences between the two shapes [7].
An example where the presented approach detects the 'extra' relevant points of a shape intuitively is given in Fig. 5. Hence, given the shapes Bone-1 and Bone-7 which have a similar shape, the calculation of the SimQSD provides the following results: -The SimQSD between shapes is 0.88. A high similarity is obtained since Bone-7 is exactly the same as Bone-1 with a bend in it.

Image Retrieval using Query-by-Sketch
In order to show the feasibility of the presented techniques, a prototype query-by-sketch tool have been implemented which: -Provides a user interface for simple sketching, which is suitable for future implementation in touch-based devices; -Uses the qualitative description and similarity approach presented to search in a database for images similar to the sketch; -In addition to the resulting list of ranked images, it presents an automatically obtained natural language description of each one, as well as of the sketch itself.
The user interface of the application is shown in Fig. 6. The drawing area where a user can draw an image by clicking and moving the mouse is placed in the left panel of the interface. The tool provides facilities for loading and saving sketches in a variety of formats.
The sketch made by the user is stored as an XML file in vector format (SVG). Clicking the button Compare and describe shows in the right panel the automatically obtained textual description of the sketch, and the images in the database in descending order of similarity to the sketch. Beside each image in the result list, the similarity between the sketch and the image appears, as well as its automatically obtained description in natural language.
In order to test the effectiveness of the query-bysketch approach, a database of 90 bitmap images is used, some of which are shown in Fig. 7. As a simplification, in this experiment the curves in the sketch are approximated to straight lines.
The approach works equally well in two potentially problematic cases: open shapes, as shown in Fig. 8, and shapes whose segments are sloppily joined, as shown in Fig. 9.
The experiments show that the application finds the most similar shape with a success rate of 90%. The algorithm also gets false positives, in the sense that the first image classified as more similar to the sketch is in fact not the most similar one; this is particularly the case if a figure has a strong semantics associated with it. However, in every test the most semantically similar image has been always positioned among the three first results, which is encouraging.

Conclusions and Future Work
The qualitative shape description scheme and the similarity calculus presented are promising approaches for calculating the similarity between a sketch and an image database. It serves, therefore, as proof-ofconcept for the idea of using a high level qualitative representation as the basis for a learning application for children. Specifically, the aim is to develop a tablet application for teaching geometric shapes to children.
However, as explained in the introduction, to be able to describe and compare cognitively a sketch, this is only the beginning. To be able to fully address the issues involved in creating this learning application, further research is required. Our plans for future work include: -The extension of the presented application in order to be able to work with curves and not only straight lines.
-The natural language generator must be enhanced in several ways. We are focusing in (i) enriching the generated language to be more suitable for a children-oriented application and (ii) to support the description of differences between shapes.
-To introduce the concept of point importance into the similarity approach, and evaluate its effects.
-Define an appropriate approach for teaching geometric shapes and implement it in a tablet application.
-Test the results with real users.

Input Image
Qualitative Description Natural Language Description