2

Consider this assembly code on x86-64 MacOS:

global start
section .text
    start:
        mov rax, 0x02000005 ; MacOS syscall: "open"
        mov rdi, filename   ; open; 1st parameter: file path
        mov rsi, 0          ; open; 2nd parameter: flags
        mov rdx, 0          ; open; 3rd parameter: in case of a newly created file, the file permissions
        syscall
        mov rdi, rax        ; set the return value of open (the file descriptor or the error code)
                            ; as the 1st parameter of the next syscall, "exit": exit status
        mov rax, 0x02000001 ; MacOS syscall: "exit"
        syscall

section .data
    filename: db "doesntexist", 0 ; a non-existing filename

It calls just "open" and then "exit", returning the return code of the "open" syscall as the exit status.

For convenience, here's how to assemble and run it: nasm -f macho64 simple.asm && ld simple.o -static -o simple && ./simple. And right after running, to see the exit status: echo $?.

What surprises me, is that as of MacOS Monterey (12.2.1（21D62）), this program prints out 2 instead of a negative number converted to the exit status range. (Returning -1 would generate the exist status of 255; this is easy to test, tweaking above code, but I also made separately sure that the rax register on return is actually in the low positive range.) "open" is supposed to return a file descriptor or an error code, but a positive number corresponding to the file descriptor of stderr doesn't look like either of those. To be sure that the code above is working just touch doesntexist, run again and see that it returns a valid file descriptor of 3.

Officially, MacOS supports only doing syscalls through the OS-provided API, not directly via an ABI like this. Is this just a case of undocumented/unstable behaviour of MacOS syscalls, or do I have a mistake or a misunderstanding somewhere?

Edit: In the comments, there was a valid concern of whether the return value was turned from negative to positive in the conversion from the return value of the syscall (a 64-bit register) to the exit status of the process (0-255). Here's an attempt to prove that this isn't the case:

global start
section .text
    start:
        mov rax, 0x02000005 ; MacOS syscall: open
        mov rdi, filename   ; open; 1st parameter: file path
        mov rsi, 0          ; open; 2nd parameter: flags
        mov rdx, 0          ; open; 3rd parameter: in case of a newly created file, the file permissions
        syscall
        cmp rax, 0          ; if the result was negative or positive
        mov rax, 0x02000001 ; MacOS syscall: exit
        jg ok               ; if the result was positive, jump to "ok" and return 99
    error:
        mov rdi, 66
        syscall
    ok:
        mov rdi, 99
        syscall

section .data
    filename: db "doesntexist", 0 ; a non-existing filename

The process returns 99, which means that rax was positive.

GolDDranks
  • 3,272
  • 4
  • 22
  • 30
  • Are error status codes and `main` return values compatible types? If not and the `main` return value is a smaller number of bits, that will truncate, potentially turning a negative error number into a positive number, depending on the original error code value. – Erik Eidt Mar 13 '22 at 20:39
  • As I said in the main text, I separately made sure that the `rax` register was in the low positive range. – GolDDranks Mar 13 '22 at 20:41
  • (I guess I've better to post the code to do that too, lest I'd be wasting everyone's time.) – GolDDranks Mar 13 '22 at 20:44
  • 3
    MacOS raw system calls set CF on error, and I think the return value is the errno code. (Of course it won't always return `-1`, then you couldn't distinguish different errors. The global or per-thread `errno` variable is a user-space thing. The libc wrapper functions provide the POSIX API where errors return `-1`. BTW, on Linux, -4095 .. -1 are `-errno` return values codes, in-band instead of using CF) Anyway, is `ENOENT == 2` on MacOS? It is on x86-64 Linux. – Peter Cordes Mar 13 '22 at 21:35
  • (turns out yes, the linked duplicate actually quoted a MacOS man page which mentions `2` for `ENOENT`. And Nate's answer confirms my recollection of the *BSD system-call ABI on x86-64.) – Peter Cordes Mar 13 '22 at 21:43
  • A somewhat meta question: Is it possible for me to answer this question, based on the marked duplicate, now that I have improved understanding of the issue? (Can't find any way to do it from the GUI anymore...) I feel that would be valuable for the people stumbling upon this question via search engines. – GolDDranks Mar 13 '22 at 22:19

0 Answers0