Increasing CNN Representational Power Using Absolute Cosine Value Regularization

Abstract

The Convolutional Neural Network (CNN) is a mathematical model designed to distill input information into a more useful representation. This distillation process removes information over time through a series of dimensionality reductions, which ultimately, grant the model the ability to resist noise and generalize effectively. However, CNNs often contain elements that are ineffective at contributing towards useful representations. This paper aims at providing a remedy for this problem by introducing Absolute Cosine Value Regularization (ACVR). This is a regularization technique hypothesized to increase the representational power of CNNs by using a Gradient Descent Orthogonalization algorithm to force the vectors that constitute their filters at any given convolutional layer to occupy unique positions in ℝ n . This method should in theory, lead to a more effective balance between information loss and representational power, ultimately, increasing network performance. The following paper examines the mathematics and intuition behind this Regularizer, as well as its effects on the filters of a low-dimensional CNN.