Semantically tied paired cycle consistency for any-shot sketch-based image retrieval
dc.contributor.author | Dutta, A | |
dc.contributor.author | Akata, Z | |
dc.date.accessioned | 2020-09-09T09:59:41Z | |
dc.date.issued | 2020-07-29 | |
dc.description.abstract | Low-shot sketch-based image retrieval is an emerging task in computer vision, allowing to retrieve natural images relevant to hand-drawn sketch queries that are rarely seen during the training phase. Related prior works either require aligned sketchimage pairs that are costly to obtain or inefficient memory fusion layer for mapping the visual information to a semantic space. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks, where we introduce the few-shot setting for SBIR. For solving these tasks, we propose a semantically aligned paired cycle-consistent generative adversarial network (SEM-PCYC) for any-shot SBIR, where each branch of the generative adversarial network maps the visual information from sketch and image to a common semantic space via adversarial training. Each of these branches maintains cycle consistency that only requires supervision at the category level, and avoids the need of aligned sketch-image pairs. A classification criteria on the generators’ outputs ensures the visual to semantic space mapping to be class-specific. Furthermore, we propose to combine textual and hierarchical side information via an auto-encoder that selects discriminating side information within a same end-to-end model. Our results demonstrate a significant boost in any-shot SBIR performance over the state-of-the-art on the extended version of the challenging Sketchy, TU-Berlin and QuickDraw datasets. | en_GB |
dc.description.sponsorship | European Union: Marie Skłodowska-Curie Grant | en_GB |
dc.description.sponsorship | European Research Council (ERC) | en_GB |
dc.identifier.citation | Published online 29 July 2020 | en_GB |
dc.identifier.doi | 10.1007/s11263-020-01350-x | |
dc.identifier.grantnumber | 665919 | en_GB |
dc.identifier.grantnumber | 853489 | en_GB |
dc.identifier.uri | http://hdl.handle.net/10871/122796 | |
dc.language.iso | en | en_GB |
dc.publisher | Springer Verlag | en_GB |
dc.rights | © The Author(s) 2020. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm ons.org/licenses/by/4.0/. | en_GB |
dc.title | Semantically tied paired cycle consistency for any-shot sketch-based image retrieval | en_GB |
dc.type | Article | en_GB |
dc.date.available | 2020-09-09T09:59:41Z | |
dc.identifier.issn | 0920-5691 | |
dc.description | This is the final version. Available from the publisher via the DOI in this record. | en_GB |
dc.identifier.journal | International Journal of Computer Vision | en_GB |
dc.rights.uri | http://creativecommons.org/licenses/by/4.0/ | en_GB |
dcterms.dateAccepted | 2020-06-19 | |
rioxxterms.version | VoR | en_GB |
rioxxterms.licenseref.startdate | 2020-06-19 | |
rioxxterms.type | Journal Article/Review | en_GB |
refterms.dateFCD | 2020-09-09T09:55:16Z | |
refterms.versionFCD | VoR | |
refterms.dateFOA | 2020-09-09T09:59:48Z | |
refterms.panel | B | en_GB |
Files in this item
This item appears in the following Collection(s)
Except where otherwise noted, this item's licence is described as © The Author(s) 2020. This article is licensed under a Creative Commons
Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as
long as you give appropriate credit to the original author(s) and the
source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material
in this article are included in the article’s Creative Commons licence,
unless indicated otherwise in a credit line to the material. If material
is not included in the article’s Creative Commons licence and your
intended use is not permitted by statutory regulation or exceeds the
permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecomm
ons.org/licenses/by/4.0/.