How to easily diagnose problems due to access to unmapped mmap regions?

Question

I've recently found a segfault that neither Valgrind, nor Address Sanitizer could give any useful info about. It happened because the faulty program munmapped a file and then tried to access the formerly mmapped region.

The following example demonstrates the problem:

#include <stdio.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>

int main()
{
    const int fd=open("/tmp/test.txt", O_RDWR);
    if(fd<0) abort();
    const char buf[]="Hello";
    if(write(fd, buf, sizeof buf) != sizeof buf) abort();

    char*const volatile ptr=mmap(NULL,sizeof buf,PROT_READ,MAP_SHARED,fd,0);
    if(!ptr) abort();
    printf("1%c\n", ptr[0]);

    if(close(fd)<0) abort();
    printf("2%c\n", ptr[0]);

    if(munmap(ptr, sizeof buf)<0) abort();
    printf("3%c\n", ptr[0]); // Cause a segfault
}

With Address Sanitizer I get the following output:

1H
2H
AddressSanitizer:DEADLYSIGNAL
=================================================================
==8503==ERROR: AddressSanitizer: SEGV on unknown address 0x7fe7d0836000 (pc 0x55bda425c055 bp 0x7ffda5887210 sp 0x7ffda5887140 T0)
==8503==The signal is caused by a READ memory access.
    #0 0x55bda425c054 in main /tmp/test/test1.c:22
    #1 0x7fe7cf64fb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #2 0x55bda425bcd9 in _start (/tmp/test/test1+0xcd9)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /tmp/test/test1.c:22 in main

And here's the relevant part of output with Valgrind:

1H
2H
==8863== Invalid read of size 1
==8863==    at 0x108940: main (test1.c:22)
==8863==  Address 0x4029000 is not stack'd, malloc'd or (recently) free'd
==8863== 
==8863== 
==8863== Process terminating with default action of signal 11 (SIGSEGV)
==8863==  Access not within mapped region at address 0x4029000
==8863==    at 0x108940: main (test1.c:22)

Compare this with the case when a malloced region is accessed after free. Test program:

#include <stdio.h>
#include <string.h>
#include <malloc.h>

int main()
{
    const char buf[]="Hello";
    char*const volatile ptr=malloc(sizeof buf);
    if(!ptr)
    {
        fprintf(stderr, "malloc failed");
        return 1;
    }
    memcpy(ptr,buf,sizeof buf);
    printf("1%c\n", ptr[0]);
    free(ptr);
    printf("2%c\n", ptr[0]); // Cause a segfault
}

Output with Address Sanitizer:

1H
=================================================================
==7057==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000000010 at pc 0x55b8f96b5003 bp 0x7ffff5179b70 sp 0x7ffff5179b60
READ of size 1 at 0x602000000010 thread T0
    #0 0x55b8f96b5002 in main /tmp/test/test1.c:17
    #1 0x7f4298fd8b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #2 0x55b8f96b4c49 in _start (/tmp/test/test1+0xc49)

0x602000000010 is located 0 bytes inside of 6-byte region [0x602000000010,0x602000000016)
freed by thread T0 here:
    #0 0x7f42994b3b4f in free (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10bb4f)
    #1 0x55b8f96b4fca in main /tmp/test/test1.c:16
    #2 0x7f4298fd8b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

previously allocated by thread T0 here:
    #0 0x7f42994b3f48 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0x10bf48)
    #1 0x55b8f96b4e25 in main /tmp/test/test1.c:8
    #2 0x7f4298fd8b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

Output with Valgrind:

1H
==6888== Invalid read of size 1
==6888==    at 0x108845: main (test1.c:17)
==6888==  Address 0x522d040 is 0 bytes inside a block of size 6 free'd
==6888==    at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6888==    by 0x108840: main (test1.c:16)
==6888==  Block was alloc'd at
==6888==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6888==    by 0x1087D2: main (test1.c:8)

My question: is there any way to make Valgrind or a Sanitizer, or some other Linux-compatible tool output useful diagnostic about the context of access to munmapped region (like where it had been mmapped and munmapped), similar to the above given output for the access-after-free?

Answer 1

valgrind (and I guess asan does the same) can output a 'use after free' error because it maintains a list of 'recently freed' blocks. Such blocks are logically freed, but they are not returned (directly) to the usable memory for further malloc calls. instead they are marked unaddressable. The size of this 'recently freed' block list can be tuned using

--freelist-vol=<number>          volume of freed blocks queue     [20000000]
--freelist-big-blocks=<number>   releases first blocks with size>= [1000000]

It would be possible to use a similar technique for munmap-ed memory: rather than physically unmap it, it could be kept in a list of recently unmapped blocks, be logically unmapped, but marked unaddressable.

Note that you could simulate that in your program by having a function my_unmap that does not really do the unmap, but rather use the client requests of valgrind to mark this memory as unaddressable.

Answer 2

is there any way to make Valgrind or a Sanitizer, or some other Linux-compatible tool output useful diagnostic

I know of no such tool, although it would be relatively easy to make one.

Your problem is sufficiently different from heap corruption problems which require specialized tools, and probably doesn't need such a tool.

The major difference is the "action at a distance" aspect: with heap corruption, the code in which the problem manifests is often very far removed from the code in which the problem originates. Hence the need to track memory state, to have red zones, etc.

In your case, the access to munmapped memory results in immediate crash. So if you just log every mmap and munmap that your program performs, you'll only have to look back for the last munmap that "covered" the address on which you crashed.

In addition, most programs perform relatively few mmap and munmap operations. If your program performs so many that you can't log them all, it's likely that it shouldn't actually do that (mmap and munmap are relatively very expensive system calls).