Matrix function derivative. Introduction

Question

The author of this question was close to determining the derivative of the function of dual variable, when we consider matrices isomorphic (algebraically and topologically) to dual numbers: $$(a+\epsilon b) \sim \begin{bmatrix} a & 0 \\ b & a \\ \end{bmatrix}.$$

So, using the fact we can define the derivative (in the Fréchet sense) for functions $F$ for with an argument in the form of such a matrix and a value in the form of such a matrix: $$F\big(\begin{bmatrix} x+s & 0 \\ y+t & x+s \\ \end{bmatrix}\big)-F\big(\begin{bmatrix} x & 0 \\ y & x \\ \end{bmatrix}\big)=\begin{bmatrix} u' & 0 \\ v' & u' \\ \end{bmatrix}\begin{bmatrix} s & 0 \\ t & s \\ \end{bmatrix}+o\bigg(\bigg|\bigg|\begin{bmatrix} s & 0 \\ t & s \\ \end{bmatrix}\bigg|\bigg|\bigg),$$ where $\bigg|\bigg|\begin{bmatrix} s & 0 \\ t & s \\ \end{bmatrix}\bigg|\bigg|=\max\{|s|,|t|\}$ and all elements of all matrices are real.

Therefore, the existence of such a matrix $\begin{bmatrix} u' & 0 \\ v' & u' \\ \end{bmatrix}$ (which we will call derivative at $\begin{bmatrix} x & 0 \\ y & x \\ \end{bmatrix}$) means differentiability of $F$ at $\begin{bmatrix} x & 0 \\ y & x \\ \end{bmatrix}$.

I'm interested in to what extent can this approach be generalized in defining a matrix-valued function of a matrix argument? I mean the case, when the derivative is an object of the same nature as variables (in opposed to the definition of the derivative of a function $f:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ which is a (Jacobian) matrix).

Can anyone share links to material with respect to such kind of derivatives?

Answer 1

That will quickly go wrong. One step more complex than your example is the case of complex valued functions of complex numbers. The matrix equivalent to those number is then: $$(a+i b) \sim \begin{bmatrix} a & b \\ -b & a \\ \end{bmatrix}.$$

And for the function we know that it needs not one, but two derivatives: $$F\big(\begin{bmatrix} x+s & y+t \\ -y-t & x+s \\ \end{bmatrix}\big)-F\big(\begin{bmatrix} x & y \\ -y & x \\ \end{bmatrix}\big)=\qquad\qquad\qquad \\[20pt] \begin{bmatrix} c & d \\ -d & c \\ \end{bmatrix}\begin{bmatrix} s & t \\ -t & s \\ \end{bmatrix} +\begin{bmatrix} u & v \\ -v & u \\ \end{bmatrix}\begin{bmatrix} s & -t \\ t & s \\ \end{bmatrix}+o\bigg(\bigg|\bigg|\begin{bmatrix} s & t \\ -t & s \\ \end{bmatrix}\bigg|\bigg|\bigg),$$ which we usually write as: $$ f(z+\Delta) - f(z) = c \ \Delta + u\ \Delta^* + o(|\Delta|)\\[15pt] \text{or:}\quad f(z+\Delta) - f(z) = \frac{df}{dz} \ \Delta + \frac{df}{dz^*}\ \Delta^* + o(|\Delta|)\\ $$.

As example take the following functions and their pair of derivatives: $$ \begin{matrix} f(z) & & df/dz & & df/dz^* \\[10pt] {\rm Re}(z) && \frac12 && \frac12 \\ {\rm Im}(z) && -\frac12\ i && \frac12\ i \\ z && 1 && 0 \\ z^2 && 2\,z && 0 \\ z^* && 0 && 1 \\ |z|^2 && z^* && z \\ |z| && \frac12 \frac{\Large z^*}{\Large |z|} && \frac12 \frac{\Large z}{\Large |z|} \\ |z|^3 && \frac32 |z| z^* && \frac32\, |z|\, z \\ && {\rm etc.} && \end{matrix}.$$

As can be seen, only functions that are analytical, like $z$, or $z^2$, have $df/dz^*=0$ so they need only $df/dz$ to describe their derivative (which we then call "the complex derivative"). Likewize, purely anti-analytical functions, like $z^*$, or $(z^*)^2$, need only $df/dz^*$. In general, however, two complex numbers are needed, or in matrix language: two matrices are needed to describe the first order variation for these matrix-valued functions of a matrix. (See also question 2126598.)

For more complex (larger) matrices the number further increase, there simply is much more information required than can be contained in one matrix to describe the derivative in those cases.

Answer 2

Let $(I,N)$ be the $2\times 2$ identity and nilpotent matrices, i.e. $\,N^2=0$.

Given two scalars $(x,y)$ construct the dual matrix and its differential
NB: $\,$ I use $(dx,dy)$ instead of $(s,t)$ $$\eqalign{ A &= xI+yN \\ dA &= dx\,I+dy\,N \\ A+dA &= (x+dx)\,I + (y+dy)\,N \\ }$$

Then apply the function and its derivative to the $A$ matrix $$\eqalign{ \def\bR#1{\Big(#1\Big)} \def\BR#1{\Big[#1\Big]} \def\LR#1{\left(#1\right)} F&\doteq \;f(A) = \;f\big(xI+yN\big) \;=\; \:f(x)\,I + yf'(x)\,N \\ F'&= f'(A) = f'\big(xI+yN\big) \;=\; f'(x)\,I + yf''(x)\,N \\ }$$ Now consider differentials and gradients of $F$ wrt the parameters $(x,y)$ $$\eqalign{ \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} dF &= f(A+dA) \;-\; f(A) \\ &= \BR{f(x+dx)-f(x)}\,I \;+\; \BR{(y+dy)\,f'(x+dx)-yf'(x)}\,N \\ &= \BR{f'(x)\,dx}\,I \;+\; \BR{yf''(x)\,dx + f'(x)\,dy}\,N \\ &= f'(A)\,dx \;+\; f'(x)\,N\,dy \\ \grad{F}{x} &= f'(A), \qquad\;\; f'(x)\,N = \grad{F}{y} \\ \\ }$$

---

Your claim is that the differential can be written using a simple matrix product $$\eqalign{ \def\tm{\times} \def\G{{\Gamma}} dF &= G\:dA \\ }$$ where $G$ is a matrix-valued gradient.

But the gradient $\,\G=\large\grad FA\,$ is known to be a fourth-order tensor $\LR{\G\in{\mathbb R}^{2\tm 2\tm 2\tm 2}}$
and a Double-Dot Product $(:)$ is required to calculate the differential $$\eqalign{ dF &= \G:dA \quad\iff\quad \G = \grad FA \quad\iff\quad \G_{ijkl} = \grad{F_{ij}}{A_{kl}} \\ }$$ This tensor-valued gradient can be related to the parametric gradients derived above by substituting $\,dA=\LR{I\,dx+N\,dy}$ $$\eqalign{ dF &= \G:I\,dx \;+\; \G:N\,dy \\ \grad Fx &= \G:I,\qquad\;\; \G:N=\grad Fy \\ }$$ Furthermore, since $N\,{\rm and}\,I$ are orthogonal under the $(:)$ product, this leads us to a simple expression for $\G$ $$\eqalign{ \def\s{\star} \G &= \frac12f'(A)\star I \;+\; f'(x)\,N\star N \\ }$$ where $(\s)$ is the dyadic (or tensor) product.

In terms of components, dyadic products are calculated as $$\eqalign{ \def\l{\ell} \def\R{{\large\Omega}} \R = P\s a\s b \quad\iff\quad \R_{ijk\l} = P_{ij}\, a_k\, b_\l \\ \\ }$$

---

$^{**}$An interesting use of dual numbers is to calculate the derivative of a $\sf real$ function $$\def\e{\varepsilon}f(x+\e) = f(x) + \e f'(x)$$

Answer 3

$\def\tsr#1#2#3{{{#1}^{#2}}_{#3}}$ $\def\prt{\partial}$ $\def\pdv#1#2{\frac{\prt #1}{\prt #2}}$

You are trying to show that $d{Y^j}_i = {D^k}_i d{X^j}_k$ where $D=\pdv{Y}{X}$; but in fact the only linear transformation that is consistent with the definition of the derivative is $d{Y^j}_i = {D^{jk}}_{il} d{X^l}_k$. A vector function $y(x)$ where $\def\R{\mathbb{R}}y \in \R^m$ and $x \in \R^n$ has a $m \times n$ Jacobian since we need to determine $\pdv{y_i}{ x_j}$ for all combinations of $i,j$. In the same way the derivative of $Y \in \R^{n \times m}$ w.r.t. $X \in \R^{k \times l}$ has to be a fourth-order tensor $D$ since we need to determine $D=\pdv{\tsr{Y}{j}{i}}{\tsr{X}{q}{r}}$ for all combinations of $i,j,q,r$. Using $d{Y^j}_i = {D^k}_i d{X^j}_k$ we can only get $\pdv{\tsr{Y}{j}{i}}{\tsr{X}{j}{k}}$ so $D$ is underdetermined.