Home Authors Book Resources Course Academic TOSCA

Shape Retrieval Contest (SHREC'11) Datasets - Robustness benchmark

The common denominator of shape retrieval approaches is the creation of a shape descriptor or signature which captures the unique properties of the shape that distinguish it from shapes belonging to other classes on the one hand, and is invariant to a certain class of transformations a shape can undergo on the other. In rigid shape analysis, different types of invariance are rotation and translation. Dealing with non-rigid shapes requires compensating for the degrees of freedom resulting from deformations. In scanned shapes, connectivity and topological changes, noise, and holes are common plights. Other typical transformations encountered in shape retrieval problems, especially when dealing with Internet data include: scale, missing parts, different sampling and triangulation. The proposed benchmark evaluates the performance of shape retrieval on a large-scale dataset under a wide variety of different transformations.

The task

SHREC'11 Robustness track simulates a large-scale shape retrieval scenario, in which the queries include multiple modifications and transformations of the same shape. The benchmark allows evaluating how algorithms cope with certain classes of transformations and what is the strength of the transformations that can be dealt with.

The collection

The dataset used in this benchmark consists of 1237 shapes from three public domain collections: TOSCA shapes, Robert Sumner's shapes, and Princeton shapes. Each of the datasets is available in the public domain. The shapes in the dataset include simulated transformations of different types and strengths, as detailed in the following. The distribution is a zip file, containing 1237 shapes. Each shape is saved in .OFF format as a text file. The naming convention of the files is 000n.off, where n is an arbitrary shape number. The shape order is scrambled arbitrarily.

Retrieval dataset (51MB, zipped and password protected - contact michael.bronstein@usi.ch to obtain the password)
.OFF format reader (.m)
The dataset will be available for download soon.


The query sets consists of 13 shape classes taken from a subset of the dataset with simulated transformations applied to them. For each shape, transformations are split into 12 classes (isometry, topology, small and big holes, scaling, noise, shot noise, missing parts, sampling, affine, and rasterize). In each class, the transformation appears in five different strength levels. The total number of transformations per shape is 60 + 1 null shape, and the total query set size is 793 shapes.

Evaluation method

Participants will submit a distance matrix for the dataset, measuring the dissimilarity between each pair of shapes. Evaluation will be performed automatically by the organizers. Performance is evaluated using precision/recall characteristic. Mean average precision (mAP) is used as a single measure of performance. The performance is evaluated on the entire query set, in each transformation category, and for each transformation strength. This evaluation allows to see which algorithms are more sensitive to certain types of transformations, and also determine the transformations strength allowing for reasonable performance.

Submission format

Participants will submit a 1237x1237 matrix of pairwise shape distance values in zip with a single file in DIST format, containing 1237 rows of space-delimited distance values, as follows:

0.23998567 1.24572606 3.51428826 ... 0.156789

The order in rows and columns should be in accordance with the shape numbers in the dataset (i.e., the mn entry in the matrix corresponds to distance between the shapes 000m.off and 000n.off).

.DIST format writer (.m)

Each participant can submit up to three results corresponding to different methods or settings of their algorithms. Name the zip file in the following convention lastname_firstname_algorithm.zip. Participants will indicate typical running time for their algorithm.

Training data

A separate set with a total of 684 shapes is optionally provided for training. The set includes 13 shape classes with null shapes same as in the main dataset and different transformed shapes with representative transformations of different classes and strengths. The distribution is in xa zip file, containing 684 shapes. Each shape is saved in .OFF format as a text file. The naming convention of the files is 000n.xform.m.off, where n is the shape class number, xform is the transformation name and m is the transformation strength (1-5). 000n.null.0.off represents the null shape.

Retrieval training dataset (31MB, zipped)


3 Feb 2011 - Dataset published.
7 Feb 2011 - Submission of preliminary results (distance matrices).
10 Feb 2011 - Evaluation feedback to participants.
15 Feb 2011 - Submission of final results (distance matrices) and one-page description of the method(s).
20 Feb 2011 - Camera ready track papers submitted for printing.
10 April 2011 - EUROGRAPHICS Workshop on 3D Object Retrieval including SHREC'
Please contact michael.bronstein@usi.ch for additional instructions or to get evaluation of your results.