CycleGAN: Unpaired Image-to-Image Translationusing Cycle-Consistent Adversarial Networks
Introduction & Related Works
Image-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs.
Image-to-image translation은 말 그대로 pair image 간의 transformation function을 만드는 task이다. 그런데 이런 pair data가 부족해서 실현하기 어렵다. 그래서 이 논문에서는 pair data가 부족해도 GAN을 이용해서 학습할 수 있는 방법을 제안한다.
Our goal is to learn a mapping G : X → Y such that the distribution of images from G(X) is indistinguishable from the distribution Y using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping F : Y → X and introduce a cycle consistency loss to enforce F (G(X)) ≈ X (and vice versa).
하지만 이 transformation function이 under constrained라서, cycle을 만들어서 해소하고자 한다.
이 그림을 보여주고는 Monet 원화에서 photo를 생각할 수 있고, photo에서 Monet의 원화 스타일을 생각할 수 있지 않냐고 얘기한다. 그리고 이 '생각하는 과정'을 'translate'라고 표현한다.
In this paper, we present a method that can learn to do the same: capturing special characteristics of one image collection and figuring out how these characteristics could be translated into the other image collection, all in the absence of any paired training examples.
명확한 GT가 없는 것에서 image translation을 수행하는 GAN model을 만들려고 하는 것이다.
These issues call for adding more structure to our objective. Therefore, we exploit the property that translation should be “cycle consistent”, in the sense that if we translate, e.g., a sentence from English to French, and then translate it back from French to English, we should arrive back at the original sentence.
데이터도 부족하기도 하고, 앞선 문제도 있으니, 이를 해결하기 위해 cycle consistency loss를 주겠다 이말이다. cycle consistency loss는 이후에 referring image segmentation 등에도 사용된다.
이러한 연구의 기본적인 목적은 GT pair를 얻기 어렵기 때문이라고 언급된다.
Years of research in computer vision, image processing, computational photography, and graphics have produced powerful translation systems in the supervised setting, where example image pairs {xi, yi}N i=1 are available (Figure 2, left). However, obtaining paired training data can be difficult and expensive.
a mapping G : X → Y such that the output ˆy = G(x), x ∈ X, is indistinguishable from images y ∈ Y by an adversary trained to classify ˆy apart from y. In theory, this objective can induce an output distribution over ˆy that matches the empirical distribution pdata(y) (in general, this requires G to be stochastic) [16].
image translation을 위한 GAN의 이야기를 먼저 꺼낸다.
However, such a translation does not guarantee that an individual input x and output y are paired up in a meaningful way – there are infinitely many mappings G that will induce the same distribution over ˆy. Moreover, in practice, we have found it difficult to optimize the adversarial objective in isolation: standard procedures often lead to the wellknown problem of mode collapse, where all input images map to the same output image and the optimization fails to make progress [15].
하지만 mapping G를 찾는 것 자체가 ill-posed problem이다. 답의 개수가 무한하다. 그리고 GAN 자체가 mode collapse 문제 때문에 최적화가 어렵다... 이 mode collapse는 하나의 distribution mode에 대해서만 output이 나오는 것을 얘기한다.
These issues call for adding more structure to our objective. Therefore, we exploit the property that translation should be “cycle consistent”, in the sense that if we translate, e.g., a sentence from English to French, and then translate it back from French to English, we should arrive back at the original sentence [3].
이를 해결하기 위해 cycle consistency를 도입한다. 이를 통해 mapping G가 한정적이게 되고 mode collapse로 인한 최적화의 어려움이 덜어진다는 것 같다.
Mathematically, if we have a translator G : X → Y and another translator F : Y → X, then G and F should be inverses of each other, and both mappings should be bijections. ....... training both the mapping G and F simultaneously, and adding a cycle consistency loss [64] that encourages F (G(x)) ≈ x and G(F (y)) ≈ y. Combining this loss with adversarial losses on domains X and Y yields our full objective for unpaired image-to-image translation.
cycle consistency의 수학적 표현을 언급한 것이다.
Related Works
Generative Adversarial Networks (GANs) 연관있다. 그리고 Image-to-Image Translation 중에서 dataset을 사용하는 pix2pix를 기반으로 설계되었다. 하지만 이 연구에서는 Unpaired Image-to-Image Translation를 수행하는 것이 목적이다.
Unlike the above approaches, our formulation does not rely on any task-specific, predefined similarity function between the input and output, nor do we assume that the input and output have to lie in the same low-dimensional embedding space. This makes our method a general-purpose solution for many vision and graphics tasks.
기존 다른 방법론과는 다르게 더 general하고 task agnostic하다고 한다.
Cycle Consistency는 NLP에서는 자주 쓰이던 방식이다. 이를 통해 structured learning이 가능하다고 한다.
[13, 23, 52, 12] is another way to perform image-to-image translation, which synthesizes a novel image by combining the content of one image with the style of another image (typically a painting) based on matching the Gram matrix statistics of pre-trained deep features. Our primary focus, on the other hand, is learning the mapping between two image collections, rather than between two specific images, by trying to capture correspondences between higher-level appearance structures.
Neural Style Transfer 또한 연관된 연구이다. 이 때까지의 기존 방법은 Gram matrix를 활용했지만, 여기서는 higher level feature structure를 변경하고자 한다. 단순히 high frequency에 속하는 style만 변경하는 것이 목적이 아니란 것.
Methods
our model includes two mappings G : X → Y and F : Y → X. In addition, we introduce two adversarial discriminators DX and DY , where DX aims to distinguish between images {x} and translated images {F (y)}; in the same way, DY aims to discriminate between {y} and {G(x)}.
Cycle하는 architecture에 대한 언급.
Our objective contains two types of terms: adversarial losses [16] for matching the distribution of generated images to the data distribution in the target domain; and cycle consistency losses to prevent the learned mappings G and F from contradicting each other.
adversarial loss와 cycle consistency loss를 사용한다는 것이 핵심
아래 식과 같이 X->Y를 수행하는 generator와 discriminator에 대해서 adversarial loss를 지정한다. Y->X에 대해서도 지정한다.
Adversarial training can, in theory, learn mappings G and F that produce outputs identically distributed as target domains Y and X respectively (strictly speaking, this requires G and F to be stochastic functions) [15]. However, with large enough capacity, a network can map the same set of input images to any random permutation of images in the target domain, where any of the learned mappings can induce an output distribution that matches the target distribution. Thus, adversarial losses alone cannot guarantee that the learned function can map an individual input xi to a desired output yi.
실제로는 GAN 만을 이용해서는 mapping perturbation을 줄이기가 어렵다는 점을 지적, 그래서 cycle consistency loss를 이용한다는 것이다. 이를 아래의 식과 같이 표현한다. 첫항이 forward cycle consistency, 둘째항이 backward cycle consistency.
이 두 loss를 합하면 다음과 같이 최종 loss가 나온다. 우리는 이것을 최소화하는 generator를 찾으면 된다.
Notice that our model can be viewed as training two “autoencoders” [20]: we learn one autoencoder F ◦ G : X → X jointly with another G ◦ F : Y → Y . However, these autoencoders each have special internal structures: they map an image to itself via an intermediate representation that is a translation of the image into another domain. Such a setup can also be seen as a special case of “adversarial autoencoders” [34], which use an adversarial loss to train the bottleneck layer of an autoencoder to match an arbitrary target distribution. In our case, the target distribution for the X → X autoencoder is that of the domain Y.
이 method를 autoencoder로 볼 수 있는데, 대신에 중간에 다른 domain으로 보내는 autoencoder라는 것이다. 이를 adversarial autoencoder의 특수 케이스라고 볼 수 있다는 언급일 뿐.
Experiments
다음은 Cycle consistency가 대체로 잘 유지되었다는 증거인 figure이다.
다음은 다른 방법론과의 비교, 그리고 ablation study이다. 이를 통해 cycle consistency는 foward, backward를 같이 써야한다는 것을 알 수 있다. 그리고 cycle consistency 혼자만으로는 원하는 image translation이 어렵다는 것을 보였다.
