Convoluational Layer Backpropagation

Numpy로 Deeplearing code를 구현하면서 공부한 내용입니다.

[github]

[blog]

위와 같은 예시에서 수식을 정리해보면

O=W∗X+b=(o11o12o21o22)o11=x11w11+x12w12+x13w13  x21w21+x22w22+x23w23  x31w31+x32w32+x33w33+bo12=x13w11+x14w12+x15w13  x23w21+x24w22+x25w23  x33w31+x34w32+x35w33+bo21=x31w11+x32w12+x33w13  x41w21+x42w22+x43w23  x51w31+x52w32+x53w33+bo22=x33w11+x34w12+x35w13  x43w21+x44w22+x45w23  x53w31+x54w32+x55w33+b O = W * X + b = \begin{pmatrix} o_{11} & o_{12} \\ o_{21} & o_{22} \end{pmatrix}\\ \begin{split} o_{11} &= x_{11} w_{11} + x_{12} w_{12} + x_{13} w_{13} \\ &\quad\; x_{21} w_{21} + x_{22} w_{22} + x_{23} w_{23} \\ &\quad\; x_{31} w_{31} + x_{32} w_{32} + x_{33} w_{33} + b\\ \\ o_{12} &= x_{13} w_{11} + x_{14} w_{12} + x_{15} w_{13} \\ &\quad\; x_{23} w_{21} + x_{24} w_{22} + x_{25} w_{23} \\ &\quad\; x_{33} w_{31} + x_{34} w_{32} + x_{35} w_{33} + b\\ \\ o_{21} &= x_{31} w_{11} + x_{32} w_{12} + x_{33} w_{13} \\ &\quad\; x_{41} w_{21} + x_{42} w_{22} + x_{43} w_{23} \\ &\quad\; x_{51} w_{31} + x_{52} w_{32} + x_{53} w_{33} + b\\ \\ o_{22} &= x_{33} w_{11} + x_{34} w_{12} + x_{35} w_{13} \\ &\quad\; x_{43} w_{21} + x_{44} w_{22} + x_{45} w_{23} \\ &\quad\; x_{53} w_{31} + x_{54} w_{32} + x_{55} w_{33} + b\\ \end{split}

backward시 다음과 같은 gradient값이 앞선 Layer로부터 들어오게 된다.

∂L∂O=(∂L∂o11∂L∂o12∂L∂o21∂L∂o22) \frac{\partial L}{\partial O} = \begin{pmatrix} \frac{\partial L}{\partial o_{11}} & \frac{\partial L}{\partial o_{12}} \\ \frac{\partial L}{\partial o_{21}} & \frac{\partial L}{\partial o_{22}} \end{pmatrix}

backpropagation을 진행하기 위해서 다음 Layer로 전달해줘야할 gradient ∂L∂X\frac{\partial L}{\partial X}와 weight, bias를 update하기 위한 ∂L∂W\frac{\partial L}{\partial W}, ∂L∂b\frac{\partial L}{\partial b}을 계산해야한다.

Input X에 대한 gradient ∂L∂X\frac{\partial L}{\partial X} 계산

먼저 ∂L∂X\frac{\partial L}{\partial X}의 xijx_{ij}에 대한 gradient matrix는 다음과 같이 표현된다.

∂L∂X=(∂L∂x11∂L∂x12∂L∂x13∂L∂x14∂L∂x15∂L∂x21∂L∂x22∂L∂x23∂L∂x24∂L∂x25∂L∂x31∂L∂x32∂L∂x33∂L∂x34∂L∂x35∂L∂x41∂L∂x42∂L∂x43∂L∂x44∂L∂x45∂L∂x51∂L∂x52∂L∂x53∂L∂x54∂L∂x55) \frac{\partial L}{\partial X} = \begin{pmatrix} \frac{\partial L}{\partial x_{11}} & \frac{\partial L}{\partial x_{12}} & \frac{\partial L}{\partial x_{13}} & \frac{\partial L}{\partial x_{14}} & \frac{\partial L}{\partial x_{15}} \\ \frac{\partial L}{\partial x_{21}} & \frac{\partial L}{\partial x_{22}} & \frac{\partial L}{\partial x_{23}} & \frac{\partial L}{\partial x_{24}} & \frac{\partial L}{\partial x_{25}} \\ \frac{\partial L}{\partial x_{31}} & \frac{\partial L}{\partial x_{32}} & \frac{\partial L}{\partial x_{33}} & \frac{\partial L}{\partial x_{34}} & \frac{\partial L}{\partial x_{35}} \\ \frac{\partial L}{\partial x_{41}} & \frac{\partial L}{\partial x_{42}} & \frac{\partial L}{\partial x_{43}} & \frac{\partial L}{\partial x_{44}} & \frac{\partial L}{\partial x_{45}} \\ \frac{\partial L}{\partial x_{51}} & \frac{\partial L}{\partial x_{52}} & \frac{\partial L}{\partial x_{53}} & \frac{\partial L}{\partial x_{54}} & \frac{\partial L}{\partial x_{55}} \\ \end{pmatrix}

x_11x\_{11}에 대해서만 계산해보면 다음과 같이 gradient의 합으로 구성되어있다.

∂L∂x11=∂L∂O∂O∂x11=∑i,j=(1,1)(2,2)∂L∂oij∂oij∂x11=∂L∂o11∂o11∂x11+∂L∂o12∂o12∂x11+∂L∂o21∂o21∂x11+∂L∂o22∂o22∂x11=∂L∂o11⋅w11+∂L∂o12⋅0+∂L∂o21⋅0+∂L∂o22⋅0 \begin{split} \frac{\partial L}{\partial x_{11}} &= \frac{\partial L}{\partial O} \frac{\partial O}{\partial x_{11}} \\ &= \sum^{(2,2)}_{i,j=(1,1)}{ \frac{\partial L}{\partial o_{ij}} \frac{\partial o_{ij}}{\partial x_{11}} } \\ &= \frac{\partial L}{\partial o_{11}} \frac{\partial o_{11}}{\partial x_{11}} + \frac{\partial L}{\partial o_{12}} \frac{\partial o_{12}}{\partial x_{11}} + \frac{\partial L}{\partial o_{21}} \frac{\partial o_{21}}{\partial x_{11}} + \frac{\partial L}{\partial o_{22}} \frac{\partial o_{22}}{\partial x_{11}} \\ &= \frac{\partial L}{\partial o_{11}} \cdot w_{11} + \frac{\partial L}{\partial o_{12}} \cdot 0 + \frac{\partial L}{\partial o_{21}} \cdot 0 + \frac{\partial L}{\partial o_{22}} \cdot 0 \\ \end{split}

다음 순서의 x에 대해서도 조금더 계산해보면

∂L∂x12=∂L∂o11⋅w12∂L∂x13=∂L∂o11⋅w13+∂L∂o12⋅w11∂L∂x14=∂L∂o12⋅w12∂L∂x15=∂L∂o12⋅w13... \begin{split} \frac{\partial L}{\partial x_{12}} &= \frac{\partial L}{\partial o_{11}} \cdot w_{12} \\ \frac{\partial L}{\partial x_{13}} &= \frac{\partial L}{\partial o_{11}} \cdot w_{13} + \frac{\partial L}{\partial o_{12}} \cdot w_{11} \\ \frac{\partial L}{\partial x_{14}} &= \frac{\partial L}{\partial o_{12}} \cdot w_{12} \\ \frac{\partial L}{\partial x_{15}} &= \frac{\partial L}{\partial o_{12}} \cdot w_{13} \\ ... \end{split}

계산되는 값들을 보면 forward 과정에서 convoluation했던 순서와 동일하게 같은 stride로 backporopagation에서도 weight와 gradient값으로 convolution 연산을 하고 있다.

Weight에 대한 gradient ∂L∂W\frac{\partial L}{\partial W} 계산

∂L∂W\frac{\partial L}{\partial W}를 matrix로 표현하면 다음과 같다.

∂L∂W=(∂L∂w11∂L∂w12∂L∂w13∂L∂w21∂L∂w22∂L∂w23∂L∂w31∂L∂w32∂L∂w33) \frac{\partial L}{\partial W} = \begin{pmatrix} \frac{\partial L}{\partial w_{11}} & \frac{\partial L}{\partial w_{12}} & \frac{\partial L}{\partial w_{13}} \\ \frac{\partial L}{\partial w_{21}} & \frac{\partial L}{\partial w_{22}} & \frac{\partial L}{\partial w_{23}} \\ \frac{\partial L}{\partial w_{31}} & \frac{\partial L}{\partial w_{32}} & \frac{\partial L}{\partial w_{33}} \\ \end{pmatrix}

원소 w_11w\_{11}에 대한 gradient는 다음과 같이 계산된다.

∂L∂w11=∂L∂O∂O∂w11=∑i,j=(1,1)(2,2)∂L∂oij∂oij∂w11=∂L∂o11∂o11∂w11+∂L∂o12∂o12∂w11+∂L∂o21∂o21∂w11+∂L∂o22∂o22∂w11=∂L∂o11⋅x11+∂L∂o12⋅x13+∂L∂o21⋅x31+∂L∂o22⋅x33 \begin{split} \frac{\partial L}{\partial w_{11}} &= \frac{\partial L}{\partial O} \frac{\partial O}{\partial w_{11}} \\ &= \sum^{(2,2)}_{i,j=(1,1)}{\frac{\partial L}{\partial o_{ij}} \frac{\partial o_{ij}}{\partial w_{11}}} \\ &= \frac{\partial L}{\partial o_{11}} \frac{\partial o_{11}}{\partial w_{11}} + \frac{\partial L}{\partial o_{12}} \frac{\partial o_{12}}{\partial w_{11}} + \frac{\partial L}{\partial o_{21}} \frac{\partial o_{21}}{\partial w_{11}} + \frac{\partial L}{\partial o_{22}} \frac{\partial o_{22}}{\partial w_{11}} \\ &= \frac{\partial L}{\partial o_{11}} \cdot x_{11} + \frac{\partial L}{\partial o_{12}} \cdot x_{13} + \frac{\partial L}{\partial o_{21}} \cdot x_{31} + \frac{\partial L}{\partial o_{22}} \cdot x_{33} \\ \end{split}

계산된 gradient는 w_11w\_{11}와 forward convolution 과정에서 한번이라도 연관된 값과 앞선 Layer의 gradient값을 곱한 값의 합과 같다.

X에 대한 gradient를 계산할 때와 마찬가지로 forward의 동일한 순서와 stride를 가지는 convolution으로 weight gradient값이 계산된다.

Bias b에 대한 gradient ∂L∂b\frac{\partial L}{\partial b} 계산

위에서 계산했던 W, X에 비해 bias의 gradient는 간단하게 다음과 같이 계산된다.

∂L∂b=∂L∂O∂O∂b=∑i,j=(1,1)(2,2)∂L∂oij∂oij∂b=∂L∂o11∂o11∂b+∂L∂o12∂o12∂b+∂L∂o21∂o21∂b+∂L∂o22∂o22∂b=∂L∂o11⋅1+∂L∂o12⋅1+∂L∂o21⋅1+∂L∂o22⋅1=∑i,j=(1,1)(2,2)∂L∂oij \begin{split} \frac{\partial L}{\partial b} &= \frac{\partial L}{\partial O} \frac{\partial O}{\partial b} \\ &= \sum^{(2,2)}_{i,j=(1,1)}{ \frac{\partial L}{\partial o_{ij}} \frac{\partial o_{ij}}{\partial b} } \\ &= \frac{\partial L}{\partial o_{11}} \frac{\partial o_{11}}{\partial b} + \frac{\partial L}{\partial o_{12}} \frac{\partial o_{12}}{\partial b} + \frac{\partial L}{\partial o_{21}} \frac{\partial o_{21}}{\partial b} + \frac{\partial L}{\partial o_{22}} \frac{\partial o_{22}}{\partial b} \\ &= \frac{\partial L}{\partial o_{11}} \cdot 1 + \frac{\partial L}{\partial o_{12}} \cdot 1 + \frac{\partial L}{\partial o_{21}} \cdot 1 + \frac{\partial L}{\partial o_{22}} \cdot 1 \\ &= \sum^{(2,2)}_{i,j=(1,1)}{ \frac{\partial L}{\partial o_{ij}} } \end{split}

모든 bias에 대한 ∂oij∂b\frac{\partial o_{ij}}{\partial b}이 1로 계산되므로 결국 들어온 gradient를 합한 값이 된다.

Reference