Overview

Original GAN

Goodfellow et al. (2014) proposed the following first GAN architecture:

  • Prior: \(p_g(\mathbf{z}) = \mathcal{N}(\mathbf{0},\mathbf{I})\)
  • Generator: \(G(\mathbf{z})\)
  • Discriminator: \(D(\mathbf{x}) \in [0, 1]\)

The overall loss from their theory: \[ \mathcal{L} = \min_G \max_D \left\{ \mathbb{E}_{p_d(\mathbf{x})}\left[ \log D(\mathbf{x}) \right] + \mathbb{E}_{p_g(\mathbf{z})}\left[ \log(1 - D(G(\mathbf{z}))) \right] \right\} \] The discriminator loss (to maximize): \[ \mathcal{L}_D = \mathbb{E}_{p_d(\mathbf{x})} \left[ \log D(\mathbf{x}) \right] + \mathbb{E}_{p_g(\mathbf{z})}\left[ \log(1 - D(G(\mathbf{z}))) \right] \]

The theoretical generator loss (to minimize): \[ \mathcal{L}_G = \mathbb{E}_{p_g(\mathbf{z})}\left[ \log(1 - D(G(\mathbf{z}))) \right] \] The actual generator loss (to maximize) in experiments, to avoid saturation in the early stage of training: \[ \mathcal{L}_G = \mathbb{E}_{p_g(\mathbf{z})}\left[ \log(D(G(\mathbf{z}))) \right] \]

GAN Training Algorithm

The most widely adopted GAN training algorithm, which is proposed in this work, alternates between training the discriminator and the generator. That is, in each training iteration:

  • Repeat for n_critics iterations

    • Sample \(\mathbf{x}^{(1)}, \dots, \mathbf{x}^{(b)}\) from \(p_d(\mathbf{x})\), \(\mathbf{z}^{(1)}, \dots, \mathbf{z}^{(b)}\) from \(p_g(\mathbf{z})\).

    • Update the discriminator by: \[ \theta_D = \theta_D + \eta \, \nabla_{\theta_D}\left( \frac{1}{b}\sum_{i=1}^b \left[ \log D(\mathbf{x}^{(i)}) \right] + \frac{1}{b}\sum_{i=1}^b \left[ \log(1 - D(G(\mathbf{z}^{(i)}))) \right] \right) \]

  • Sample \(\mathbf{z}^{(1)}, \dots, \mathbf{z}^{(b)}\) from \(p_g(\mathbf{z})\).

  • Update the generator by: \[ \theta_G = \theta_G - \eta\,\nabla_{\theta_G}\left( \frac{1}{b} \sum_{i=1}^b \log(1 - D(G(\mathbf{z}^{(i)}))) \right) \] or alternatively, \[ \theta_G = \theta_G + \eta\,\nabla_{\theta_G}\left( \frac{1}{b} \sum_{i=1}^b \log(D(G(\mathbf{z}^{(i)}))) \right) \]

References

Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. “Generative Adversarial Nets.” In Advances in Neural Information Processing Systems, 2672–80.