SE3: Sequential Semantic Segmentation of Large Images with Minimized Memory

Abstract

Semantic segmentation results in pixel-wise perception accompanied with GPU computation and expensive memory, which makes trained models hard to apply to small devices in testing. Assuming the availability of hardware in training CNN backbones, this work converts them to a linear architecture enabling the inference on edge devices. Keeping the same accuracy as patch-mode testing, we segment images using a scanning line with the minimum memory. Exploring periods of pyramid network shifting on image, we perform such sequential semantic segmentation (SE3) with a circular memory to avoid redundant computation and preserve the same receptive field as patches for spatial dependency. In the experiments on large drone images and panoramas, we examine this approach in terms of accuracy, parameter memory, and testing speed. Benchmark evaluations demonstrate that, with only one-line computation in linear time, our designed SE3 network consumes a small fraction of memory to maintain an equivalent accuracy as the image segmentation in patches. Considering semantic segmentation for high-resolution images, particularly for data streamed from sensors, this method is significant to the real-time applications of CNN based networks on light-weighted edge devices.