A Zero-Cost Darts Base on Multi-Step Optimization
DOI:
https://doi.org/10.54097/fcis.v5i3.13841Keywords:
Zero-Cost Darts, Multi-Step, Zen-NASAbstract
DARTS has achieved great result in Image classification field, the accuracy predictor and computation costs are the key of DNAS algorithm. Searching for a high-performance architecture always costs Large amount of computation. With a gradient-based bi-level optimization, DARTS using one-step optimization which makes the process available within a few GPU day, because of the one-step optimization , there exists a great gap between the architectures in search and evaluation. In this paper, we propose a zero-cost DARTS method which using multi-step optimization to address the above issues. To further reduce the computational requirements, we use the zen-score to estimate architectures in evaluation stage. Experiments on CIFAR-10 and our private data sets show that our algorithm play a certain role in solving the above problems.
Downloads
References
Zoph B, Le Q V. Neural architecture search with reinforcement learning[J]. arXiv preprint arXiv:1611.01578, 2016.
He K, Zhang X , Ren S ,et al.Deep Residual Learning for Image Recognition[J].IEEE, 2016.DOI:10.1109/CVPR.2016.90.
Simonyan K, Zisserman A .Very Deep Convolutional Networks for Large-Scale Image Recognition[J].Computer Science, 2014.DOI:10.48550/arXiv.1409.1556.
Lecun Y, Bottou L .Gradient-based learning applied to document recognition[J].Proceedings of the IEEE, 1998, 86(11):2278-2324.DOI:10.1109/5.726791.
Pham H, Guan M, Zoph B, et al. Efficient neural architecture search via parameters sharing[C]//International conference on machine learning. PMLR, 2018: 4095-4104.
Brock A, Lim T, Ritchie J M, et al. Smash: one-shot model architecture search through hypernetworks[J]. arXiv preprint arXiv:1708.05344, 2017.
Liu H, Simonyan K, Yang Y. Darts: Differentiable architecture search[J]. arXiv preprint arXiv:1806.09055, 2018.
Chu X, Zhou T , Zhang B ,et al.Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search [C]// European Conference on Computer Vision.Springer, Cham, 2020.DOI:10.1007/978-3-030-58555-6_28.
Hong W, Li G , Zhang W ,et al.DropNAS: Grouped Operation Dropout for Differentiable Architecture Search[C]//Twenty-Ninth International Joint Conference on Artificial Intelligence and Seventeenth Pacific Rim International Conference on Artificial Intelligence {IJCAI-PRICAI-20.2020. DOI:10. 24963/ ijcai.2020/318.
Abdelfattah M S, Mehrotra A , Dudziak U ,et al.Zero-Cost Proxies for Lightweight NAS[J]. 2021.DOI:10. 48550/ arXiv. 2101.08134.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages8580–8589, 2018.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations, 2020.
Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. Knas: Green neural architecture search. In International Conference on Machine Learning, pages 11613–11625. PMLR, 2021.
Lin M, Wang P , Sun Z ,et al.Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition[J]. 2021.DOI: 10. 13140/ RG.2.2.33579.98084.
Wang H, Yang R , Huang D ,et al.iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization [J].IEEE transactions on neural networks and learning systems, 2023.
Mok J , Na B , Kim J H ,et al.Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?[J]. 2022.DOI: 10. 48550/arXiv.2203.14577.
Xie S, Zheng H, Liu C, et al. SNAS: stochastic neural architecture search[J]. arXiv preprint arXiv:1812.09926, 2018.
Wang Y, Yang Y, Chen Y, et al. Textnas: A neural architecture search space tailored for text representation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2020, 34(05): 9242-9249.
Heuillet A, Tabia H , Arioui H ,et al.D-DARTS: Distributed Differentiable Architecture Search[J]. 2021.DOI:10. 48550/ arXiv. 2108.09306.
Liang H, Zhang S , Sun J ,et al.DARTS+: Improved Differentiable Architecture Search with Early Stopping[J]. 2019. DOI:10.48550/arXiv.1909.06035.
Xu Y, Xie L , Zhang X ,et al.PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search[C]// International Conference on Learning Representations.2020.
Chen J T Q. Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild[J].International Journal of Computer Vision, 2021, 129(3).
