The primary mechanism in Android for communicating with the radio is /dev/smd0, which implements an AT modem with many extended commands. I do not know how that device is implemented but I would recommend looking at its source. I believe it's essentially serial.
The OS (if I recall) has a daemon which monitors and interfaces with this device.
For example, when the phone needs the baseband to register on a mobile network, it can send AT+CREG=1 to that device, and the baseband will take care of it (and return 0). To dial a call it will send ATD8005551212 or something to that device. To pick up a call it will send ATA; to hang up it will send ATH, just like a modem.
The incoming call transaction on that device looks something like this. The baseband will print +CRING: VOICE on /dev/smd0, and the OS can send AT+CLCC to get the caller ID, which is formatted as an extended status message: +CLIP: "+18555551212",,,,"" or something.
I understand that sometimes the device name for this varies; it might be /dev/ttyUSB0, for instance, if the interface to the baseband is implemented as a USB serial port.
You'd be doing this at the system app layer, and not inside the kernel, which I believe would be the correct way of going about it.