0

I am trying to allocate 40 bytes of space in memory through calling the external C command malloc in x86 Assembly (AT&T/Intel syntax). However, when I debug my program, the EAX register has not changed after the malloc command is called (from my understanding, the procedure to use malloc is to put the number of bytes you want to allocate in the EDI register and then executing call malloc to put the pointer to the block of memory allocated in the EAX register). Below is my x86 Assembly code:

.extern malloc

.text
.global main
main:
    movl %esp, %ebp #for correct debugging
    # write your code here
    xorl  %eax, %eax
    
    movl $40, %edi
    call malloc
    
    ret

I am using 32-bit convention (not 64-bit) on Linux.

Compilation command:

gcc -m32 -Wall -g -c -o program.o program.s
Adam Lee
  • 436
  • 1
  • 14
  • 49
  • 5
    Yes, but 32 bit convention does not use `edi` to pass argument, that's 64 bit. 32 bit passes argument on the stack. You want `push $40; call malloc; mov %ebp; %esp; ret` – Jester Nov 23 '20 at 00:58
  • 1
    Are you targeting x86-64 or x86 (32-bit). The calling convention with EDI suggest 64-bit but you do stack related operations with ESP instead of RSP. Are you on Windows or MacOs/Linux/BSD? How are you observing the value in EAX? In a debugger or looking at the return value from the program (the return value from a program is 8 bits) – Michael Petch Nov 23 '20 at 00:59
  • 2
    Okay then @Jester is correct you have to push the parameters on the stack with the i386 System V ABI (they are pushed from right to left). As well the modern Linux ABI requires that the stack be properly aligned on at least a 16 byte boundary for calls to ABI compliant functions (which includes the C library). Improper stack alignment may work in some environments but may cause faults in other environments depending on how the functions (like the C library) are built. – Michael Petch Nov 23 '20 at 01:02
  • 3
    On a side note EBP is a callee saved register (non-volatile) so if you modify it in `main` you should be saving its value (push EBP on the stack at the start) and then restore it (pop EBP) before you `ret` from `main` – Michael Petch Nov 23 '20 at 01:06
  • @MichaelPetch Correct. – Adam Lee Nov 23 '20 at 17:23

1 Answers1

3
call malloc

where's my push?

push %edi
call malloc
add  %esp, 4 ; caller cleans up the stack

So they're telling me that a modern glibc is now imposing an byte stack alignment. I'm not in a position to confirm this, but you've just gotta do it. Would look like this now:

sub  %esp, 8
push %edi
call malloc
add  %esp, 12 ; caller cleans up the stack
Joshua
  • 40,822
  • 8
  • 72
  • 132
  • 1
    This still needs stack alignment, though. – Nate Eldredge Nov 23 '20 at 01:47
  • @NateEldredge: Last time I did it on 32 bit it sure didn't care. And if it cares now than my glibc programs compiled in 1999 don't work anymore. Glibc promised a fixed ABI. Did it change? – Joshua Nov 23 '20 at 01:54
  • 1
    The current version of the i386 System V ABI that Linux uses *does* on paper require / guarantee 16-byte stack alignment. But if your glibc isn't compiled with SSE2 enabled, then glibc itself is unlikely to ever depend on that. And even if it did, only for copying 4 int/pointer-sized things, not just a pair like in 64-bit mode. The main potential problem is for a callback into your own code, if you compiled with `-march=native`, or to another library compiled with SSE2 enabled. (ping @NateEldredge if you're curious) – Peter Cordes Nov 23 '20 at 01:57
  • 1
    Re: old binaries: `-mpreferred-stack-boundary=4` has been the default for a long time. It used to be just a good idea, but not enough people noticed that GCC's 32-bit SSE2 code-gen was making code that *depended* on it until binaries built that way were widespread, at which point the least-bad way forward was to make it "the law". See links at the top of [Why does the x86-64 / AMD64 System V ABI mandate a 16 byte stack alignment?](https://stackoverflow.com/q/49391001) for the sordid history of 32-bit: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40838#c91 is a larger version of this summary – Peter Cordes Nov 23 '20 at 02:02
  • @PeterCordes: Well there we go. My 32 bit CPUs don't have SSE and the compiler is equally old. So if I call up the 32 bit compiler I'm going to get 4 byte stack alignment. Modern machine's going to see 8 byte stack alignment. I only have *32 on my modern computer to run programs that literally _can't_ be compiled to 64 bit and the *32 libraries to sustain those programs. So the glibc _has_ to be compiled to support it or the binaries just crash. – Joshua Nov 23 '20 at 02:30
  • 1
    Although this question is Linux, MacOS uses the same 32-bit ABI (for versions of MacOS that still support that architecture) and malloc will like likely fail with a fault (much of the C library there will). I just tried it and confirmed without proper alignment it fails on aligned SSE instructions. I wouldn't rely on code that doesn't do proper alignment. I think one place on Linux that may fault is GLIBC's `printf` with a double parameter. – Michael Petch Nov 23 '20 at 02:31
  • 1
    @Joshua: *So if I call up the 32 bit compiler I'm going to get 4 byte stack alignment.* - that doesn't entirely follow. GCC probably defaulted to `-mpreferred-stack-boundary=4` or at least 3 before SSE existed, to allow 8-byte alignment for `double` locals on P5 Pentium. Anyway, the fact that *your* 32-bit setup is de-facto backwards compatible with old binaries that might misalign the stack doesn't mean you can safely tell everyone on the Internet to do the same in 32-bit code. This mistake / oversight by GCC developers that led to an ABI incompatibility sucks for everyone :( – Peter Cordes Nov 23 '20 at 02:37
  • Regarding your edit, the stack is misaligned by 4 when the call instruction transfer control to the function `main` because the return address was pushed. To got it aligned back on a 16 byte boundary you would have to reduce ESP by 12 by the time `malloc` is called. (or 12, 28, 44, etc) – Michael Petch Nov 23 '20 at 02:59
  • @MichaelPetch: Yeah I thought something was off. The modern way is to rearrange the whole function to use mov instructions but I'm not gonna do that in the middle of somebody else's hand-coded assembly. – Joshua Nov 23 '20 at 03:04
  • Also note that the OP wants to allocate a constant `40` bytes; they're only using EDI for arg-passing. So you should use `push $40`, not `push %edi`. Also, yes, this is correct stack alignment *if* you leave their broken `mov %esp, %ebp` without push/pop EBP around it. So if you're going to try to fix their whole function to be ABI-compliant, clobbering EBP is as big or bigger a problem as 16-byte stack alignment (especially in 32-bit code). – Peter Cordes Nov 23 '20 at 03:59
  • @PeterCordes: I'm ignoring register preservation because it's `main`. Any other function and I'd fix that too. – Joshua Nov 23 '20 at 04:05
  • Main's caller (from CRT code) tends to be forgiving, but it is legal in C for other functions to call `main`, in which case your program would likely break if main destroyed regs it shouldn't. Or a better argument is that future readers might want to use this code in other functions, so at least commenting on the ABI violations you're allowing would make sense. e.g. `# push %ebp # skipped in main because its caller usually doesn't care` – Peter Cordes Nov 23 '20 at 04:11
  • Or there's probably a duplicate we could use to close this question instead of putting effort into an answer. – Peter Cordes Nov 23 '20 at 04:13
  • @PeterCordes I have recently put in the compilation command I am using. – Adam Lee Nov 23 '20 at 05:42
  • With the code in your answer, is the pointer to allocated memory placed in the EAX register? – Adam Lee Nov 23 '20 at 05:59
  • 1
    For reference [the modern i386 System V ABI](https://www.uclibc.org/docs/psABI-i386.pdf), In the section _The Stack Frame_ it has this to say: _The **end of the input argument area** shall be **aligned on a 16 (32, if __m256 is passed on stack) byte boundary**. In other words, the value (%esp + 4) is always a multiple of 16 (32) when control is transferred to the function entry point_ – Michael Petch Nov 23 '20 at 13:22