1. Introduction

• Manual network design
  • LeNet, AlexNet, VGG, ResNet, ...
  • Particular network instantiations and design principles can be generalized and applied to numerous settings
• Finding well-optimized networks
  • Neural Architecture Search (NAS)
  • High performance, but NO discovery of network design principles
• AnyNet
  • Unconstrained design space
• RegNet
  • A low-dimensional design space consisting of simple “regular” networks
  • Stage width and depths are determined by a quantized linear function

2. Related Work

• Manual network design
  • VGG, Inception, ResNet, ResNeXt, DenseNet, MobileNet, ...
• Automated network design
  • NAS
• Network scaling
  • EfficientNet
• Comparing networks
  • Comparing and analyzing populations of networks sampled from a design space
• Parameterization
  • An empirical study justifying the design choices & insights into structural design choices

3. Design Space Design

• 전체 모델에 적용하여 성능을 높일 수 있는 design principles를 발견하는 것이 목적
  • 초기에 어떠한 제약도 없는 design space에서 점진적으로 design하는 것을 제안 → design space design

3.1. Tools for Design Space Design

• I. Radosavovic et al., "On Network Design Spaces for Visual Recognition", ICCV 2019.

  • design space의 quality를 design space로부터 model들을 sampling해서 정량화

  • model error distribution으로 특징화

  • How to obtain a distribution of models?

    • low-compute, low-epoch training regime
      • e.g. use the 400M flop regime and train each sampled model for 10 epochs

  • The error empirical distribution function (EDF)

    

    $$ F(e)=\frac{1}{n}\sum^n_{i=1}\mathbf{1}[e_i<e] \tag{1} $$

    • $F(e)$ : $e$보다 낮은 error를 보인 모델들의 비율
    • empirical bootstrap
      1. 주어진 $n$개의 $(x_i,e_i)$ 쌍에서 25%만 추출
         • $x_i$ : model statistic (e.g. depth)
         • $e_i$ : error
      2. 최소 error를 갖는 쌍을 선택
      3. 1~2를 $10^4$번 반복
      4. 최소 $x$ 값에 대해 95% CI계산 → best value
• 요약
  • design space에서 $n$ 모델을 뽑고 학습하여 분포를 얻음
  • design space의 quality를 확인하기위해 error EDF를 계산하고 시각화
  • design space의 다양한 요인들 (e.g. depth, $w_4$,...)을 시각화하고, insight를 얻기 위해 empirical bootstrap 진행
  • 정해진 insight로 design space 정제

3.2. The AnyNet Design Space