Explicit matrix gradient expression for residual network

ABSTRACT

Residual Network (ResNet) is a distinguished network structure in deep learning, and its layers can be profound. We theoretically explore the mathematics characteristics of the ResNet, in particular, to pay attention to the gradient information, which is a powerful and straightforward mathematical tool for analysing the properties of ResNet, such as the gradients of the loss function with respect to the input and the weight parameters and the gradient of the entry of the logits with respect to the input. A theorem about the explicit matrix expression of gradients in Resnet is given in this work. A rigorous mathematical and logical derivation of the theorem is obtained in detail by the matrix derivative definition and matrix differentiation. We further provide explicit matrix expressions of some deep learning algorithms in ResNet, including backpropagation, gradient-based adversarial attacks, and gradient-based saliency maps. Furthermore, the reasons why the ResNet network works are analysed. Finally, experimental results are provided to verify the correctness and efficiency of the proposed theorem.

KEYWORDS:

• ResNet
• matrix gradient expression
• backpropagation
• saliency map
• adversarial example

Acknowledgements

This work is supported in part by the National Key R&D Program of China under Grant 2017YFB0802800.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. Pytorch 1.8.0 version.

2. Tensorflow 1.14.0 version.