MODULAR SEARCH SPACE FOR AUTOMATED DESIGN OF NEURAL ARCHITECTURE

Abstract

Abstract. The past years of research have shown that automated machine learning and neural
architecture search are an inevitable future for image recognition tasks. In addition, a crucial aspect of any
automated search is the predefined search space. As many studies have demonstrated, the modularization
technique may simplify the underlying search space by fostering successful blocks’ reuse. In this regard, the
presented research aims to investigate the use of modularization in automated machine learning. In this
paper, we propose and examine a modularized space based on the substantial limitation to seeded building
blocks for neural architecture search. To make a search space viable, we presented all modules of the space
as multisectoral networks. Therefore, each architecture within the search space could be unequivocally
described by a vector. In our case, a module was a predetermined number of parameterized layers with
information about their relationships. We applied the proposed modular search space to a genetic algorithm
and evaluated it on the CIFAR-10 and CIFAR-100 datasets based on modules from the NAS-Bench-201
benchmark. To address the complexity of the search space, we randomly sampled twenty-five modules and
included them in the database. Overall, our approach retrieved competitive architectures in averaged 8 GPU
hours. The final model achieved the validation accuracy of 89.1% and 73.2% on the CIFAR-10 and CIFAR-
100 datasets, respectively. The learning process required slightly fewer GPU hours compared to other
approaches, and the resulting network contained fewer parameters to signal lightness of the model. Such an
outcome may indicate the considerable potential of sophisticated ranking approaches. The conducted
experiments also revealed that a straightforward and transparent search space could address the
challenging task of neural architecture search. Further research should be undertaken to explore how the
predefined knowledge base of modules could benefit modular search space.