xmm16..31 (and their ymm/zmm equivalents) are new with AVX-512 and only accessible via EVEX prefixes, which have 2 extra bits to add to each of the ModRM fields, and a 5 more bits as an extra field for the third operand.
REX + legacy-SSE, and VEX for AVX1/2 encodings, can only access xmm/ymm0..15.
Wikipedia's EVEX article has a pretty good table that shows where the bits come from, which I transcribed some of:
| Addr mode |
Bit 4 |
Bit 3 |
Bits [2:0] |
Register type |
| REG |
EVEX.R' |
EVEX.R |
ModRM.reg |
General Purpose, Vector |
| RM |
EVEX.X |
EVEX.B |
ModRM.r/m |
GPR, Vector |
| NDS/NDD |
EVEX.V' |
EVEX.v3 |
EVEX.v2v1v0 |
Vector |
|
|
|
|
|
| Base |
0 |
EVEX.B |
SIB.base (or modrm) |
GPR |
| Index |
0 |
EVEX.X |
SIB.index |
GPR |
If the R/M operand is a vector register instead of a memory addressing mode, it uses both the X (index) and B (base) bits as extra register-number bits. Because that means there's no SIB.index field which could also need extension to select r8..r15.
In REX and VEX prefixes, The X bit goes unused when the source operand isn't memory with an indexed addressing mode. (https://wiki.osdev.org/X86-64_Instruction_Encoding#REX_prefix, but note in a register-number table earlier in that page showing X.Reg, X is just a placeholder for R or B, not REX.X; confusing choice on that page).
See also x86 BSWAP instruction REX doesn't follow Intel specs? for another diagram of using an extra register-number bit from a REX prefix.