Vector, Matrix, and Tensor Derivatives

Tips

• if confusing, looking at single element
  • but don’t look at single element if you can look at the whole matrix.
  • otherwise, what’s the point we looking at matrix?
• Representing the important part of derivative arrays in a compact way is critical to efficient implementations of neural networks.
• the gradient of scalar with respect to a variable should have the same size as that variable

Jacobian Matrix

For $f:\R^n \to \R^m$, the Jacobian matrix $J \in \R^{m \times n}$

$$ W \in \R^{C \times D}, \bold x \in \R^D,\bold y \in \R^C \\ \frac{\partial \bold y}{\partial \bold x} \in \R^{C*D}, \left (\frac{\partial \bold y}{\partial \bold x} \right )_{i,j} \overset{def}{=} \frac{\partial y_i}{\partial x_j} \\

\bold y=W \bold x \to \frac{\partial \bold y}{\partial \bold x}=W \\

\bold z^T=\bold x^T W^T \to \frac{\partial \bold z}{\partial \bold x}=W $$

dL/dX

consider examples are row vectors

$$

Y=XW,[N \times M]=[N\times D][D\times M]\\

y_{i,j}=\sum_{k=1}^D x_{i,k} w_{k,j}\\

L \in R, \frac{\partial L}{\partial w_{k,j}} =\sum_{i=1}^n \frac{\partial L}{\partial y_{i,j}} \frac{\partial y_{i,j}}{\partial w_{k,j}} =\sum_{i=1}^n \frac{\partial L}{\partial y_{i,j}}x_{i,k} \\

\frac{\partial L}{\partial W}= X^T\frac{\partial L}{\partial Y}, [D \times M]=[D \times N][N \times M] \\ \frac{\partial L}{\partial X}= \frac{\partial L}{\partial Y}W^T, [N \times D]=[N \times M][M \times D] $$

Field $(F,+,*,0,1)$

• associative
• commutative
• distributive
• identity
• inverse