From time to time you hit into something, which proves that even the old dog can learn new things.
Did you ever need to mmap() the device memory?
I recently work on one of the implementations of ODP (Dataplane framework similar to DPDK) and naturally I need to mmap of device or physical address space into userspace process.
Mapping of PCI BAR0, simple stuff right?
addr = mmap (NULL, BAR0_SIZE, PROT_READ, MAP_PRIVATE, mem_fd, BAR0_OFFSET);
so why I use MAP_PRIVATE and what is wrong with it?
Usually when you map piece of memory and you do not want to share the memory page with other processes, you use MAP_PRIVATE. It's fully legitimate with typical memory as when in conjunction with PROT_READ, kernel use copy-on-write and allocate the page when it is first time referenced.
Wait... what I just said? It is copied ...
So remember, never use MAP_PRIVATE when mapping PCI BARS, because it will COPY the BAR memory. In other words references to this mapped memory will not reach the PCI. You will not operate on device address space anymore, instead you will operate on copy of PCI address space inside of RAM.
