GAN

Network Structure

StyleGAN

• 开源了高分辨率人脸数据集FFHQ
• 一种特征或许可控的高分辨率人脸生成网络
• 采用了从低分辨率到高分辨率的逐次生成
• 提出一种影响特征的变换方式AdaIN
• 

Evolutionary Generative Adversarial Networks

• 用进化算法找G。

Don't let your Discriminator be fooled

• Robust的Discriminator可以不用Wasserstein距离也能很好训练。这个很好理解，robust的训练方法保证了真实分布样本空间L-Lipschitz，而Discriminator则需要在生成分布空间及真实分布空间上L-Lipschitz。

Application

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

• 通过Network Dissection的方法，找到每个神经元对应的含义，从而控制GAN在指定区域生成指定内容。

Interpretability

Network Dissection: Quantifying Interpretability of Deep Visual Representations

• 一种自动匹配部分神经元语义含义的方法。方法如下：在一个有标注的segmentation的数据集上，看每个神经元对应的激活图像区域与标注划分的相似度，从而找到神经元含义。
• 一种新的评估神经网络性能的方法。
• 

Abstract

解决了GAN难训的问题。

Key Points

普通GAN有的问题及原因：

• D训练太好时，G没法学习。 采用CrossEntropy，当D收敛到最优时，G的梯度就没了。原因是当生成图片的分布与真实分布几乎没有交集时， KL散度或JS散度基本为定值。

当$f$为L-Lipschitz时，

所以

• 对应到网络结构中的解决方案就是——
• 判别器最后一层去掉激活函数
• 生成器和判别器的loss不取log(相对CrossEntropy而言)
• 每次更新判别器的参数之后把它们的绝对值截断到不超过一个固定常数c（尝试保证L-Lipschitz）
• -D_Loss可以反应训练进程

以下是实验的其他结果
* 不要用基于动量的优化算法（包括momentum和Adam），推荐RMSProp，SGD也行（无理论证明，后续发展证明这不影响）

后续发展

• 在Improved Training of Wasserstein GANs中，改进了限制L-Lipschitz的方法，从weight clipping变到gradient penalty.
• weight clipping和W度量导致网络所以权值要么是c要么是-c，从而影响网络表达能力。以及会导致梯度消失或者爆炸。
• gradient penalty通过设计一个loss来限制Lipschitz,采用$ReLU(|\nabla_x D(x)|_p - K)$加入损失函数来解决。
• 
• 其中$x\sim X$要从样本空间采样，过于困难。所以只考虑

心血来潮买了手环4NFC版

结果竟然用不了NFC，要我重新注册账号？？？但我小米账号显示的是中国。

找了半天，终于在互联网某角落发现解决方案：

小米手环是华米的，数据库不互通。所以需要重新注册的是华米账号，所以到这里来注销自己的华米账号再注册就好了。

https://user.huami.com/hm_account/2.0.0/index.html?loginPlatform=web&platform_app=com.xiaomi.hm.health&v=3.7.8#/

Background

• Solving low resource domain adaptation.
• Assume has few lable for Target Domain.

Structure

• the ultimate goal of our approach is to use the mapped Source → Target samples ($X_{S→T}$ ) to
  augment the limited data of the target domain ($X_T$ ).

• let $M_S$ and $M_T$ be the task specific models trained on domains $P_S(X, Y)$ and $P_T (X, Y)$.

Relaxed Cycle Consistency

.

In supervised case

Similarly, there's

In unsupervised case

Performance

$MNIST$ domain is limited to only 10 samples per class, denoted $MNIST-(10)$

硬核机器学习课

首先介绍什么是Compressed Sensing（压缩感知）

这一段参考自如何理解压缩感知(compressive sensing)？

• 在现有的传统的信号处理模式中，信号要采样、压缩然后再传输，接收端要解压再恢复原始信号。采样过程要遵循奈奎斯特采样定理，也就是采样速率不能小于信号最高频率的两倍，这样才能保证根据采样所得的信息可以完整地恢复出原始信号。压缩感知在接收端通过合适的重构算法就可以恢复出原始信号，因此可以避免在传统的信号处理模式中的数据浪费和资源浪费问题。
• 压缩感知最初提出时，是针对稀疏信号x，给出观测模型$y=\Phi x$时，要有怎么样的$\Phi$，通过什么样的方式可以从$y$中恢复出$x$。（PS：稀疏信号，是指在这个信号x中非零元素的个数远小于其中零元素的个数。）
• Tao他们还推导了Restricted Isotropic Property (RIP) (可以理解成空间映射能基本保持稀疏向量长度，“伸缩度”很小)等一系列理论，证明了为了达到完美恢复，采样矩阵和信号稀疏度需要满足的条件和相互之间的关系。
• 然而，很多信号本身并非稀疏的，比如图像信号。此时可以通过正交变换Ψ’，将信号投影到另外一个空间，而在这个空间中，信号$a=\Psi*x$(analysis model)变得稀疏了。然后我们可以由模型$y=\Phi*a$，即$y=Φ*Ψ*x$，来恢复原始信号x。(PS:正交变换——对一个由空间$R^n$

Background

Tile is clear, "Few-Shot Adversarial Domain Adaptation"

• They need labels for new domain, but only a few.

Structure

• Definition of Pairs:

• Trainning steps

DCD means domain-class discriminator(tells it's G1,G2,G3 or G4)

• loss1: classfication loss
• loss3: adversarial loss for discriminator
• loss4: adversarial loss for generator( mixup (G1,G2) and (G3,G4))

Perfomance

* FADA - n stands for our method when we use n labeled target samples per category in training*

Inspiration

The model trained on large data with more noise have more ability to adapte other domains.

Introduction

A muti-domain image-to-image translation model.

Model Structure

Loss consist of Adversarial Loss(D tries to distinguish whether a image x from domain C is real or not, while G tries to cheat it), Domain Classification Loss (D tries to tell where a image $x_{b}$ is from, while $G_{x->c}$ tries to let D tell the fake image is from $c$ instead of $b$), Reconstruction Loss for Generator.

The structure is adapted from CycleGAN.

Trainning

When trainning one domain, set other domains to be 0.

Performance

Background

Doing the same Domain Adaptation job as CYCADA, translate images between GTA5 and Cityscapes.

This is published after CYCADA.

Method

The main difference is that there's a Embedding space in the middle.

The loss consis of 6 parts:

• Normal Classfication (P1)

• Reconstruct part (P2)

• Discriminator z part (P3)

• Discriminator x,y part (P4)

• Cycle loss (P5)

• Complex Classfication (P6)

In total, there's 3 discriminator $D_z,D_x,D_y$ and 2 encoder $f_x,f_y$ and 2 decoder $g_x,g_y$ and a classifier $h$.

Performance

State of art performance when this paper was published.

In job above the encoder is LeNet.

Someting interesting: "Switching

Aims

To match the joint distribution $P(Y_s,X_s)$ with $P(Y_t,X_t)$.

Solution

Find a matrix $A$ to let $P(A^T X_s), P(A^T X_t)$ as close as possible.
So is $P(Y_s|A^T X_s), P(Y_t|A^T X_t)$.

For the first pair, we use a method called TCA to minimize

this equals to

where $M_0$ is

For the next part, because we don't have $Y_t$, so we train a classfier C to learn

Background

We have $(X_s,Y_s)$ and $X_t$, but $X_s, X_t$ are not from the same ditribution, i.e. $P(Y_s|X_s) \neq P(Y_t|X_t)$. To solve this, we use the following method.

Assumption

There exist a transformation $\phi$ that $P(Y_s|\phi(X_s)) \approx P(Y_t|\phi(X_t))$.

Solutioin

To find $\phi$, we try to minimize following distance called MMD(maximum mean discrepancy).

To solve