how does the part work where you put the address of the /bin/sh string from the libc library on the stack? is it the first local variable of the system function?
It’s basically running
system("/bin/sh"); The payload from the point of corruption of the saved return address (saved eip) looks like “SSSSAAAABBBB”, where “SSSS” is the address of
system(), “AAAA” is what OP refers to as padding, and “BBBB” is the address of the “/bin/sh” string. Upon ret, the vulnerable function implicitly pops “SSSS” into eip and thereby enters
system() in libc. At this point, the stack pointer, esp, is pointing to “AAAA”.
system() immediately does two things that 32-bit x86 functions typically do in the absence of compiler optimizations: (1) it pushes the value of the frame pointer, ebp, onto the stack (let’s refer to this new stack object as “PPPP”), which of course adjusts esp; (2) it copies the value of esp into ebp to serve as the current frame pointer. The result is that both esp and ebp are now pointing to the start of “PPPPAAAABBBB”. ebp will keep pointing to “PPPP”, but esp will be further adjusted to deal with any local variables, alignment, and whatnot: “LLLL…LLLLPPPPAAAABBBB”.
system() will expect to find its first (and only) parameter at
0x8(%ebp), i.e., 8 bytes from the start of what ebp points to, which is “BBBB”. Why 8 bytes? ebp, as noted, is pointing to the start of “PPPP” (4 bytes), and “AAAA” (another 4 bytes) is where the saved eip would’ve been if
system() had been entered normally. OP uses “AAAA” as padding, but you can chain in a second library call by using its address instead of “AAAA” if desired.
You’ll notice that the challenge program doesn’t even call
system() itself, which is worth keeping in mind because some folks have overkilled exploits with ROP while being under the strange impression that there’s no
system() to return into if the target program doesn’t call
system(). Speaking of overkill, it may actually be possible to solve this challenge without any corruption of the saved return address at all, depending on whether a null-byte corruption of the saved frame pointer in the vulnerable function will have that saved frame pointer pointing to a suitable spot within the target buffer. This will affect
main()‘s conception of ebp and, in turn, the place from which
main() takes its saved return address at the end of the program. The displacement of the saved frame pointer will be a multiple of 4 between 0 and 252 inclusive. It’s possible to influence the displacement via environment variables and command-line arguments (even though the challenge program uses no command-line arguments), both of which will influence the value of
main()‘s original ebp.