Image-to-Image Translation with Conditional Adversarial Nets
Phillip Isola
Jun-Yan Zhu
Tinghui Zhou
Alexei A. Efros
[Paper]
[GitHub]


Example results on several image-to-image translation problems. In each case we use the same architecture and objective, simply training on different data.

Abstract

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.


Try our code

 [GitHub]


Paper


Download [7MB]


[Bibtex]


Experiments

Here we show comprehensive results from each experiment in our paper. Please see the paper for details on these experiments.

Effect of the objective
Cityscapes
Facades

Effect of the generator architecture
Cityscapes

Effect of the discriminator patch scale
Cityscapes
Facades

Additional results
Map to aerial
Aerial to map
Semantic segmentation
Day to night
Edges to handbags
Edges to shoes
Sketches to handbags
Sketches to shoes


#pix2pix

People have used our code for many cool applications. Many of them are posted on twitter with the hashtag #pix2pix. Please feel free to check them out here.


Recent Related Work

Generative adversarial networks have been vigorously explored in the last two years, and many conditional variants have been proposed. Please see the discussion of related work in our paper. Below we point out two papers that especially influenced this work: the original GAN paper from Goodfellow et al., and the DCGAN framework, from which our code is derived.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks NIPS, 2014. [PDF]

Alec Radford, Luke Metz, Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks ICLR, 2016. [PDF]



Acknowledgements

We thank Richard Zhang, Deepak Pathak, and Shubham Tulsiani for helpful discussions. Thanks to Saining Xie for help with the HED edge detector. This work was supported in part by NSF SMA-1514512, NGA NURI, IARPA via Air Force Research Laboratory, Intel Corp, Berkeley Deep Drive, and hardware donations by Nvidia. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, AFRL or the U.S. Government.