Consider $g\in U(n)$ and $t\in T$, where $T$ is the diagonal maximal torus in $U(n)$.
Some common manifolds may be obtained as quotients of the $U(n)$ like the complex grassmannian, $Gr(k,n)=U(n)/U(k)\times U(n-k)$ and the manifold of complete flags in $\mathbb{C}^n$, $F_n=U(n)/T$, and there is an induced action of $T$ coming from the action of $U(n)$ in them. Counting the fixed points of this action yields ${n\choose k}$ and $n!$, respectively. These numbers are precisely $|w_n|/(\,|w_{n-k}|\cdot|w_{k}|\,)$ and $|w_n|$, respectively, where $|w_k|$ denotes the order of the Weyl group of $U(k)$, i.e., $w_k=N_{U(k)}(T)/T$.
From this there is a natural candidate to generalize the result for partial flag varieties $U(n)/U(k_1)\times\cdots\times U(k_d)$, where $k_1+\cdots+k_d=n$: the number of fixed points of this action will be $$ \frac{|w_n|}{|w_{k_1}|\cdots|w_{k_d}|}=\frac{n!}{k_1!\cdots k_d!}=\frac{|N_{U(n)}(T)/T|}{|N_{U(k_1)}(T_1)/T_1|\cdots|N_{U(k_d)}(T_d)/T_d|}, $$ where $T_i$ is the diagonal maximal torus in $U(k_i)$.
I would like some help on how to show this, in other words, how to show that there is a bijection between $$ \text{$g\in U(n)$ such that $g^*tg\in U(k_1)\times\cdots U(k_d)$ for all $t\in T$ (fixed points)} $$ and $$ \frac{N_{U(n)}(T)/T}{N_{U(k_1)}(T_1)/T_1\times\cdots\times N_{U(k_d)}/T_{k_d}}=\frac{N_{U(n)}(T)}{N_{U(k_1)}(T_1)\times\cdots\times N_{U(k_d)}}. $$ I'v tried brute forcing some conditions on the matrices $g$, yet I don't see anything that is of help, so I think a conceptual proof may be the way to go.