Lecture for Linux conference

Abstract

Ever wonder how memory is given to your app? Where does it come from? Where does it go? This is tutorial for programmers working in and around the Linux kernel. Topics include file mapping, dynamic loading and linking, the /dev/null device, inter-app security, virtual memory and fork() management. These topics may sound diverse, but are bound together by a single concept.

This is an intermediate level tutorial. C coding experience is required.

1.5 hours Next

The Point (Oblio & Arrow)

File mapping as basic principle results in

demand loading
interapp security
virtual memory

File Mapping I

basic principle
often taught last
mapping is a link between block of memory and a block of file

just an entry in the map table
see below

Consider app launch

What does the kernel have to do in order to launch an app?

read the head of the file (first 80 bytes); contains a map of the file layout
usually an executable is made up of

text (code, ro data)
data (data)

some is initialized data
all the 0 data is clustered at the end, and indicated by the "in-memory" size > than the "on-disk" size

actually, there are two maps, one coarse (for loading) and one detailed (for linking)

the kernel simply creates two mmaps: one for code and one for data

see File Mapping II below

Here is a sample dump of a common little program

/usr/bin/dir:     file format elf32-i386
/usr/bin/dir
architecture: i386, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x08048f90

Program Header:
    PHDR off    0x00000034 vaddr 0x08048034 paddr 0x08048034 align 2**2
         filesz 0x000000a0 memsz 0x000000a0 flags r-x
  INTERP off    0x000000d4 vaddr 0x080480d4 paddr 0x080480d4 align 2**0
         filesz 0x00000013 memsz 0x00000013 flags r--
    LOAD off    0x00000000 vaddr 0x08048000 paddr 0x08048000 align 2**12
         filesz 0x00006a88 memsz 0x00006a88 flags r-x
    LOAD off    0x00006a88 vaddr 0x0804fa88 paddr 0x0804fa88 align 2**12
         filesz 0x000001f8 memsz 0x00000444 flags rw-
 DYNAMIC off    0x00006bf8 vaddr 0x0804fbf8 paddr 0x0804fbf8 align 2**2
         filesz 0x00000088 memsz 0x00000088 flags rw-

Dynamic Section:
  NEEDED      libc.so.6
  INIT        0x8048c10
  FINI        0x804d3e0
  HASH        0x80480e8
  STRTAB      0x804879c
  SYMTAB      0x804831c
  STRSZ       0x290
  SYMENT      0x10
  DEBUG       0x0
  PLTGOT      0x804fb18
  PLTRELSZ    0x1a0
  PLTREL      0x11
  JMPREL      0x8048a6c
  REL         0x8048a2c
  RELSZ       0x40
  RELENT      0x8

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .interp       00000013  080480d4  080480d4  000000d4  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .hash         00000234  080480e8  080480e8  000000e8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .dynsym       00000480  0804831c  0804831c  0000031c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .dynstr       00000290  0804879c  0804879c  0000079c  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .rel.got      00000008  08048a2c  08048a2c  00000a2c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .rel.bss      00000038  08048a34  08048a34  00000a34  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .rel.plt      000001a0  08048a6c  08048a6c  00000a6c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .init         0000002c  08048c10  08048c10  00000c10  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  8 .plt          00000350  08048c3c  08048c3c  00000c3c  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  9 .text         00004444  08048f90  08048f90  00000f90  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 10 .fini         0000001c  0804d3e0  0804d3e0  000053e0  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 11 .rodata       0000168c  0804d3fc  0804d3fc  000053fc  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 12 .data         00000080  0804fa88  0804fa88  00006a88  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 13 .ctors        00000008  0804fb08  0804fb08  00006b08  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 14 .dtors        00000008  0804fb10  0804fb10  00006b10  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 15 .got          000000e0  0804fb18  0804fb18  00006b18  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 16 .dynamic      00000088  0804fbf8  0804fbf8  00006bf8  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 17 .bss          0000024c  0804fc80  0804fc80  00006c80  2**3
                  ALLOC
 18 .comment      000000f0  00000000  00000000  00006c80  2**0
                  CONTENTS, READONLY
 19 .note         000000f0  000000f0  000000f0  00006d70  2**0
                  CONTENTS, READONLY

load process:

you can find the source for this in /usr/src/linux/fs/binfmt_elf.c "point of no return"
map the .text (code)

mark it executable, read-only

map the .data

mark it non-executable, read-write, copy on write (COW)
in Linux, COW is PROT_WRITE and MAP_DENYWRITE
first one describes the memory protection, and the second one describes the mmap binding back to the original file
the original file is read-only

lengthen the .data

to realize .bss, mmap to /dev/zero

mmap the stack to /dev/zero

mark it grow-down
this is actually done in setup_arg_pages()

NOTE: so far, we've done 4 mmap() calls these are fast (10's of us) no memory is brought in yet

env[] is copied; argv[] is copied

triggers page fault for stack, page copy

now simply exec the start address

triggers page fault, file load

first data read will trigger file load
first data write will/may trigger page copy

we're running!

Appreciate:

only the blocks which are required are actually loaded
the above is for a statically linked exe

File Mapping II

Linux not only allows mapping, it practically insists on it

each piece of memory has a file association

except for the kernel itself

anything that has an entry in the filesystem driver list can be used as a "file"

most files are on the main disk or the swap disk
/dev/null is a "special file"

a "file" is managed by a code module that know how to:

                        open, close
                        read, write
                        lseek
                        mmap

mappings are held in a global file mapping table
structure has

                address, len
                filename, file offset
                flags

        [really, has
                vm_start, vm_end
                vm_page_prot
                vm_flags
                various pointer to next
                        sibling by virt address
                        avl tree of sibling by address
                vm_ops: array of functions
                vm_next/prev_share: points to inode
                vm_offset
        ]

for the purpose of this lecture, consider the file map table to look like this:

Address	Len	File	FileOffset	Flags
0x08048000	0x7000	/bin/dir	0	code: ro, exec
0x0804F000	0x1000	/bin/dir	0x7000	data: rw, no-exec, COW
0x40000000	0x14000	/lib/ld-linux.so.2	0	code: ro, exec
...	...	...	...	...

adding a new entry to this table is quick
note that both sides of the table do not have to be created at the same time

we can have memory blocks (Address!=0) which exist before a file has been assigned to it (File==0) [this is the MAP_ANON case]
we can have file blocks (File!=0) which exist with without any memory associated with them (Address==0) [this is the swapped-out case or the not-yet-loaded case]

The mmap Flags

The mmap() system call is used heavily by the kernel and is also available to applications. There are two sets of flags involved in a mmap() call.

The PROT_* flags manage the memory access.

there are only READ WRITE & EXEC
these are tied in tightly to the hardware memory map unit

The MAP_* flags manage the file binding

MAP_READ, MAP_WRITE should match the file props

Other flags can be attached to a map

GROWSDOWN: used for stack
DENYWRITE: used for COW
EXECUTABLE:
SHARED:
PRIVATE: used for COW
ANONYMOUS:

there is no source file but eventually, there will be a swap file!
starts mapped to /dev/zero, COW

The /dev/zero Device

there is a 4k block of zero's declared at power up
when I said that filemapping was a one-to-one relationship, I lied:

each device has a mmap() function which determines how the page tables are set
it is one(file block) to many(memory block)

the /dev/zero device effectively has a mmap() function
large amounts of zero's are quickly created
funny things happen to zero page access

see zerotest.c

Back to the load process

the bss

created quickly
guaranteed clean

the stack

created quickly
grows through a kernel mm mechanism
guaranteed clean

thus we have inter-app security
through the map-fault-load mechanism, only the minimum code & data blocks are loaded

The C runtime library (CRT)

the kernel knows little beyond the facts that we have covered so far
the kernel does not deal with calls to malloc()
instead, the CRT handles malloc()

the first call excites the CRT to request a pool of memory from the kernel
then the CRT subdivides and issues the pieces
there is only one kernel call to ask for memory: sbrk() requests that the brk marker be moved; usually upwards; syscall 2d

run strace dir to watch the brk marker move!

in GLIBC2, large malloc's are realized by mmaps of /dev/zero

Typical Load Map

Small static app

Address Contents

0x08048000 code

0x08052000 data

0x0805A000 bss (zero data)

0x08072000 end of data (brk marker)

0xBFFFE000 stack

Dynamic Linking

To save disk space and load time, apps can share libraries on disk
Almost all dyn-linking code is in ld-linux.so.2, not in the kernel!
The kernel knows that if an app is marked dyn-link, there is an interpeter/helper which must get loaded first

usually, this is /lib/ld-linux.so.2
you can check by running ldd filename
or objdump -s --section=.interp filename

This little app

reads and parses the app header
mmaps each library (code, data & possibly a bss)
fixes up the links to data references
marks subroutine calls as "not-yet-available"
jumps to the app

check out readelf and LD_DEBUG

Small dyn-link app

Address Contents

0x08000000 code

0x08002000 data

0x0800A000 bss (zero data)

0x08012000 end of data (brk marker)

0x40000000 beginning of mmap area
ld.so (aka ld-linux.so.2)

0x40014000 libc.so.6

0x400f2000 end of libc.so.6

0xBFFFC000 stack, already enlarged

lib/libc.so.6:     file format elf32-i386
/lib/libc.so.6
architecture: i386, flags 0x00000150:
HAS_SYMS, DYNAMIC, D_PAGED
start address 0x00017d68

Program Header:
    PHDR off    0x00000034 vaddr 0x00000034 paddr 0x00000034 align 2**2
         filesz 0x000000a0 memsz 0x000000a0 flags r-x
  INTERP off    0x000e5ded vaddr 0x000e5ded paddr 0x000e5ded align 2**0
         filesz 0x00000013 memsz 0x00000013 flags r--
    LOAD off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**12
         filesz 0x000e5e00 memsz 0x000e5e00 flags r-x
    LOAD off    0x000e5e00 vaddr 0x000e6e00 paddr 0x000e6e00 align 2**12
         filesz 0x00003ea0 memsz 0x00007038 flags rw-
 DYNAMIC off    0x000e9bf0 vaddr 0x000eabf0 paddr 0x000eabf0 align 2**2
         filesz 0x000000b0 memsz 0x000000b0 flags rw-

Dynamic Section:
  NEEDED      ld-linux.so.2
  SONAME      libc.so.6
  INIT        0x17ae4
  FINI        0xda510
  HASH        0xd4
  STRTAB      0x9848
  SYMTAB      0x2c08
  STRSZ       0x786f
  SYMENT      0x10
  PLTGOT      0xea1b4
  PLTRELSZ    0xd70
  PLTREL      0x11
  JMPREL      0x151b0
  REL         0x11f10
  RELSZ       0x32a0
  RELENT      0x8
  VERDEF      0x11e60
  VERDEFNUM   0x4
  VERNEED     0x11ee0
  VERNEEDNUM  0x1
  VERSYM      0x110d8

Version definitions:
1 0x01 0x0865f4e6 libc.so.6
2 0x00 0x0d696910 GLIBC_2.0
3 0x00 0x0d696911 GLIBC_2.1
        GLIBC_2.0 
4 0x00 0x09691f71 GLIBC_2.1.1
        GLIBC_2.1 

Version References:
  required from ld-linux.so.2:
    0x0d696911 0x00 06 GLIBC_2.1
    0x0d696910 0x00 05 GLIBC_2.0

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .hash         00002b34  000000d4  000000d4  000000d4  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .dynsym       00006c40  00002c08  00002c08  00002c08  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .dynstr       0000788f  00009848  00009848  00009848  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .gnu.version  00000d88  000110d8  000110d8  000110d8  2**1
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .gnu.version_d 00000080  00011e60  00011e60  00011e60  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .gnu.version_r 00000030  00011ee0  00011ee0  00011ee0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .rel.data     00002ab0  00011f10  00011f10  00011f10  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .rel__libc_subinit 00000018  000149c0  000149c0  000149c0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  8 .rel__libc_subfreeres 00000070  000149d8  000149d8  000149d8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  9 .rel.eh_frame 00000068  00014a48  00014a48  00014a48  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 10 .rel__libc_atexit 00000008  00014ab0  00014ab0  00014ab0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 11 .rel.got      000006f0  00014ab8  00014ab8  00014ab8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 12 .rel.ctors    00000008  000151a8  000151a8  000151a8  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 13 .rel.plt      00000d70  000151b0  000151b0  000151b0  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 14 .plt          00001af0  00015f20  00015f20  00015f20  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 15 .text         000c2af8  00017a10  00017a10  00017a10  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 16 .fini         00000041  000da510  000da510  000da510  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 17 .rodata       0000b88d  000da560  000da560  000da560  2**5
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 18 .interp       00000013  000e5ded  000e5ded  000e5ded  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
 19 .data         000030bc  000e6e00  000e6e00  000e5e00  2**5
                  CONTENTS, ALLOC, LOAD, DATA
 20 __libc_subinit 0000000c  000e9ebc  000e9ebc  000e8ebc  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 21 __libc_subfreeres 00000038  000e9ec8  000e9ec8  000e8ec8  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 22 __libc_atexit 00000004  000e9f00  000e9f00  000e8f00  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 23 .eh_frame     0000029c  000e9f04  000e9f04  000e8f04  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 24 .ctors        0000000c  000ea1a0  000ea1a0  000e91a0  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 25 .dtors        00000008  000ea1ac  000ea1ac  000e91ac  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 26 .got          00000a3c  000ea1b4  000ea1b4  000e91b4  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 27 .dynamic      000000b0  000eabf0  000eabf0  000e9bf0  2**2
                  CONTENTS, ALLOC, LOAD, DATA
 28 .bss          00003198  000eaca0  000eaca0  000e9ca0  2**5
                  ALLOC
 29 .stab         0022c8b4  00000000  00000000  000e9ca0  2**2
                  CONTENTS, READONLY, DEBUGGING
 30 .stabstr      0007b589  00000000  00000000  00316554  2**0
                  CONTENTS, READONLY, DEBUGGING
 31 .comment      0000d44f  00000000  00000000  00391add  2**0
                  CONTENTS, READONLY
 32 .note         0000459c  0000d44f  0000d44f  0039ef2c  2**0
                  CONTENTS, READONLY
 33 .rel.stab     0000ee30  000119ec  000119ec  003a481c  2**2
                  CONTENTS, READONLY, DEBUGGING
 34 .gnu.warning.sigstack 0000004d  00020820  00020820  003a34e0  2**5
                  CONTENTS, READONLY
 35 .gnu.warning.sigreturn 0000003b  00020880  00020880  003a3540  2**5
                  CONTENTS, READONLY
 36 .gnu.warning.siggetmask 00000039  000208c0  000208c0  003a3580  2**5
                  CONTENTS, READONLY
 37 .gnu.warning.getcontext 0000003c  00020900  00020900  003a35c0  2**5
                  CONTENTS, READONLY
 38 .gnu.warning.setcontext 0000003c  00020940  00020940  003a3600  2**5
                  CONTENTS, READONLY
 39 .gnu.warning.makecontext 0000003d  00020980  00020980  003a3640  2**5
                  CONTENTS, READONLY
 40 .gnu.warning.swapcontext 0000003d  000209c0  000209c0  003a3680  2**5
                  CONTENTS, READONLY
 41 .gnu.warning.gets 00000039  00020a00  00020a00  003a36c0  2**5
                  CONTENTS, READONLY
 42 .gnu.warning.fexecve 00000039  00020a40  00020a40  003a3700  2**5
                  CONTENTS, READONLY
 43 .gnu.warning.setlogin 0000003a  00020a80  00020a80  003a3740  2**5
                  CONTENTS, READONLY
 44 .gnu.warning.getwd 0000003a  00020ac0  00020ac0  003a3780  2**5
                  CONTENTS, READONLY
 45 .gnu.warning.sstk 00000036  00020b00  00020b00  003a37c0  2**5
                  CONTENTS, READONLY
 46 .gnu.warning.revoke 00000038  00020b40  00020b40  003a3800  2**5
                  CONTENTS, READONLY
 47 .gnu.warning.gtty 00000036  00020b80  00020b80  003a3840  2**5
                  CONTENTS, READONLY
 48 .gnu.warning.stty 00000036  00020bc0  00020bc0  003a3880  2**5
                  CONTENTS, READONLY
 49 .gnu.warning.chflags 00000039  00020c00  00020c00  003a38c0  2**5
                  CONTENTS, READONLY
 50 .gnu.warning.fchflags 0000003a  00020c40  00020c40  003a3900  2**5
                  CONTENTS, READONLY
 51 .gnu.warning.madvise 00000039  00020c80  00020c80  003a3940  2**5
                  CONTENTS, READONLY
 52 .gnu.warning.getmsg 00000038  00020cc0  00020cc0  003a3980  2**5
                  CONTENTS, READONLY
 53 .gnu.warning.getpmsg 00000039  00020d00  00020d00  003a39c0  2**5
                  CONTENTS, READONLY
 54 .gnu.warning.putmsg 00000038  00020d40  00020d40  003a3a00  2**5
                  CONTENTS, READONLY
 55 .gnu.warning.putpmsg 00000039  00020d80  00020d80  003a3a40  2**5
                  CONTENTS, READONLY
 56 .gnu.warning.fattach 00000039  00020dc0  00020dc0  003a3a80  2**5
                  CONTENTS, READONLY
 57 .gnu.warning.fdetach 00000039  00020e00  00020e00  003a3ac0  2**5
                  CONTENTS, READONLY

Summary

File mapping in the kernel gives us

demand loading

only load from the disk what is required, when it is required
fast creation of bss, stack
code, data & libraries shared between instances
simple code in the kernel

inter-app security

bss, malloc() & stack spaces are initialized clean

virtual memory

large amounts of memory can be allocated, guaranteed zeros
only written memory is actually instantiated
the swap mechanism is flexible

Address	Contents
0x08048000	code
0x08052000	data
0x0805A000	bss (zero data)
0x08072000	end of data (brk marker)
0xBFFFE000	stack

Address	Contents
0x08000000	code
0x08002000	data
0x0800A000	bss (zero data)
0x08012000	end of data (brk marker)
0x40000000	beginning of mmap area ld.so (aka ld-linux.so.2)
0x40014000	libc.so.6
0x400f2000	end of libc.so.6
0xBFFFC000	stack, already enlarged