![](\"/media/paper-images/ADDA_CORAL/Network.png\")
\n \n
\n An illustration of our proposed method\ncombining Deep Coral and ADDA. Blue and orange\narrows denote data flows of source and target domain\nrespectively. Blue encoder and classifier are\npretrained and fixed.\n\t
\n\n Unsupervised domain adaptation techniques are essential for image classification tasks in the real world. As the domain of images, or the space of all possible images, is so enormous that models trained on any dataset will inevitably suffer from out of domain issues. One promising research direction is to use domain adaptation methods to adapt models trained on source domain to the target domain. Adversarial Discriminative Domain Adaptation (ADDA) is one typical adversarial learning based unsupervised domain adaptation method. Though it is proved to be effective on simple and small datasets, it requires sophisticated training strateies and is hard to converge at times. We propose to force align the distribution of the model’s output with that of an adapted model, which also serves as the initialization for the adversarial training. In this way, the adversarial process will be forced to search within a space with results at least as good as the initialization. Experiments on our proposed Tiny-16-Class-Imagenet show our method is effective and efficient in terms of accuracies and training time.
\nBy generalizability, we refer to the model’s ability to perform equally well on unseen data.\nThe word, “domain”, in this article denotes the space of input features X and the marginal distribution P(X). Specifically, for image classification tasks, the domain of training dataset is the set of all possible images and the marginal distribution in this dataset 6.\nIt is crucuial for models to be generalizable doing image classification tasks as the space of possible images is too big that any dataset can only capture one small fraction of it and if the model fails to generalize, then it is useless.\nDomain shift refers two domains being different, which is common. For example, when using a model trained with images taken in daylight, but used with images taken\nat night. Unsurprisingly, the model usually fails. Different patterns of perturbations like noises imposed\non images are another souce of domain shift.\nTo solve the problem of domain shift, one promising research area is domain adaptation, which aims to adapt a model trained on source domain to the target domain.\nIn this project, we investigate the unsupervised\ndomain adaptation problem, which does not require the target domain to be labeled.
\nExtensive domain adaptation\nalgorithms have been proposed to account for the\ndegradation in performance due to domain shift.\nDeep Coral 4 extends the unsupervised domain\nadaption method Coral to learn a nonlinear transformation\nthat is able to align correlations of layer\nactivations in deep neural networks. Adversarial Discriminative\nDomain Adaptation (ADDA) 5 combines\ndiscriminative model and generative adversarial\nnetworks to learn a discriminative mapping by fooling\na domain discriminator.
\nWe conduct experiments on two datasets:\nTiny-16-Class-ImageNet and MNIST-USPS2,3. Most experiments are done on the Tiny-16-Class-\nImageNet, which is self-produced following guidelines\nin 1.\nThe Tiny-16-Class-ImageNet has three\nsubsets: training set, validation set and test set, each containing 10015, 1269 and 10350 images respectively.\nAll three subsets have 16 general classes (like\nbear rather than brown bear), but with different domains.\nTraining and validation sets contain samples\nof different sub-classes (brown bear vs black bear). We apply different patterns of noises to generate different domains. Sample noises are illustrated in figure 2.\nTest set contains all samples from every sub-classes\n(brown bear, black bear, etc). We have also tested\nour proposed method on MNIST-USPS dataset.
\nWe adapt the idea of Deep Coral 4\nto simply align second-order statistics in the last layer\nof the backbone network by adding a coral loss. This\nmethod is simple yet effective and is very extensible.\nWe replace the backbone of the Deep Coral\nwith ResNet-50 pretrained on ImageNet when doing\nexperiments on the Tiny-16-Class-ImageNet. We\nuse the same SGD hyper-parametsers as in 4 The\nλ controlling the weight of the coral loss is set the\nsame with 4, except on MNIST-USPS dataset,\nwhere we set λ=1−num_epochsepoch.
\nWe also adopt the idea of ADDA by first learning a discriminative representation using data\nfrom the source domain and then learning another encoding\nthat maps the target domain to the source domain\nwith a domain-adversarial loss. We use ResNet-50 (excluding the last layer) as the backbone for encoder and a 3 layer MLP as\nthe discriminator with hidden size of 1024. The pretrained ResNet-50 will be freezed during adversarial training. Adam is\nused as the optimizer with β1=0.5 and β2=0.999.\nThe learning rate is set to be 0.0002 and the batch\nsize is 32. During the adaption stage, target encoder\nis updated every 4 steps.
\nWe propose a new method\nthat combines the Deep Coral and the ADDA methods,\nby using Deep Coral as the pretraining of the\nADDA, and aligning the target domains’ second order\nstatistics between the classification outputs of\nthe fixed pretrained encoder and the ADDA trained\ntarget encoder. The overall architecture is illustrated\nin Fig.1. During experiments, we find that vanilla\nADDA ruins the pretrained encoder due to the poorly\ntrained discriminator. To better use the initialization\nof the Deep Coral pretrained encoder while ensuring\nthe target encoder learned will generate similar features\nfor target and source domain, we use coral loss\nto only align the ADDA trained encoder’s classification output with that of the fixed pretrained encoder,\nand gradually decrease the coral loss’s weight.
\nThe underlying assumption we made here is that we assume the best possible solution lies near (with respect to learning using Adam) to the already good initialization in the solution space.
\nTable 1: Our Deep Coral+ADDA’s results on Tiny-16-Class-ImageNet and MINIST-USPS.
\nSetting | \nSource | \ntarget | \nAcc | \n
|---|---|---|---|
| ResNet-50 | \ntrain | \nval† | \n25.13% | \n
| ADDA | \ntrain | \nval† | \n48.32% | \n
| Deep Coral | \ntrain | \nval† | \n73.52% | \n
| Ours | \ntrain | \nval† | \n77.69% | \n
| LeNet | \nMINST | \nUSPS | \n25.13% | \n
| ADDA | \nMINST | \nUSPS | \n89.40% | \n
| Deep Coral | \nMINST | \nUSPS | \n54.30% | \n
| Ours | \nMINST | \nUSPS | \n94.56% | \n
†: validation set with uniform noise (0.5)
\nTable 2: Our Deep Coral+ADDA’s results on unseen test set of Tiny-16-Class-ImageNet.
\nSetting | \nTrain Source | \nTrain target | \nUnseen Target | \nAcc | \n
|---|---|---|---|---|
| ResNet-50 | \ntrain | \nNone | \nTest† | \n5.34% | \n
| ResNet-50-ImageNet | \ntrain | \nNone | \nTest† | \n13.14% | \n
| DeepCoral | \ntrain | \nval† | \nTest† | \n38.37% | \n
| Ours | \ntrain | \nval† | \nTest† | \n52.96% | \n
†: validation set with uniform noise (0.5)
\nExperiment results in Figure 3 shows ADDA and\nDeep Coral’s improvements on the target domain.\nDeep Coral generally outperform ADDA by a large\nmargin except on the High-Pass target domain. The\nfailure on this domain is mostly likely due to the drastic\ndomain shift between High-Pass and others, as illustrated\nin Figure 2 in the dataset section. Deep Coral has\nbetter generalizability to unseen domains. It’s most\nlikely because Deep Coral doesn’t alter the encoder\nmuch and the encoder is pretrained on the ImageNet\n(though without any added noises).
\nTable 1 shows our proposed Deep Coral+ADDA’s\nresults on the Tiny-16-Class-ImageNet and\nMNIST-USPS. We added uniform noise (0.5) to\nthe validation set making the domain shift to the\ntraining set even larger and the domain adaptation\ntask even harder. The high performance and concrete\nimprovements of our Deep Coral+ADDA method\nover other settings validate the effectiveness of our\nnovel modifications and designs. We also test our\nmethod on unseen and untrained target domain and\nobserve a significantly better results as shown in Table\n2 in the appendix.
\nRobert Geirhos, Carlos R Medina Temme, Jonas Rauber, Heiko H Schutt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. arXiv preprint arXiv:1808.08750, 2018.
\n↩\nYann LeCun, L\u0013eon Bottou, Yoshua Bengio, and Patrick Ha\u000bner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
\n↩\nYuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, BoWu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
\n↩\nBaochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443-450. Springer, 2016.
\n↩\nEric Tzeng, Judy Ho\u000bman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE con- ference on computer vision and pattern recogni- tion, pages 7167-7176, 2017.
\n↩\nWang, Mei, and Weihong Deng. “Deep visual domain adaptation: A survey.” Neurocomputing 312 (2018): 135-153.
\n↩\nWe combine Adversarial Discriminative Domain Adaptation (ADDA) with Deep CORAL to allow ADDA better utilize the pretrained initialization. Vanilla ADDA diverses drastically from the initialization resulting much poorer results in early epochs comparing to the initialization. It requires sophisticated fine-tuning for ADDA to give satisfying results. With our novel modifications ADDA-CORAL can be trained extremely faster and yields better results.
Bohua Wan, Cong Mu, Ruzhang Zhao, Zhuoying Li (Ordered by alphabetic)
","tags":["Computer Vision","Image Classification","Unsupervised","Adversarial","Deep Learning","Research"],"title":"Combining ADDA with Deep CORAL: Unsupervised Domain Adaptation for Image Classification","socialImage":{"publicURL":"/static/d0e03a59c26b7452974a9f8fabd18d99/demo.gif"}}}},"pageContext":{"slug":"/posts/combining-adda-with-deep-coral-unsupervised-domain-adaptation-for-image-classification"}},"staticQueryHashes":["251939775","401334301","41472230"]}