Experimenting with hugepages to reduce CPU and memory resources

Experimenting with hugepages to reduce CPU and memory resources

Computer server
Image courtesy of Renjith Krishnan / FreeDigitalPhotos.net
Most computers today have support for virtual memory. An application (i.e. process) running on such computers sees its address space as one large range of contiguous addresses, even if its memory chunks may be scattered around the physical memory (RAM). This means that when the process requests a particular memory location, the computers must figure out which physical memory location this corresponds to. The mapping of virtual memory to physical memory is stored in the page tables.

For processes that uses lots of memory, the virtual to physical memory mapping (i.e. pagetable) will need to hold a lot of mappings (called page table entries, or PTEs), and may grow to very large sizes. Having very large pagetables claims extra resources on the system, as more memory are needed to hold the page tables, and more CPU cycles must be used to search the pagetable. The system may benefit from keeping the number of page table entries at a minimum.

This is where hugepages comes in handy. Using hugepages, we increase the memory chunks allocated by the process, thus reducing number of memory mappings needed by the process. Instead of one page table entry mapping for example 4 kB of data (which is the default size on many systems), each entry may map for example 4 MB of data.

To test how a process memory usage affects the pagetable on one of our linux servers at work, I created two tiny applications written i C. The first allocates 19 GB of memory using regular memory allocations, and the second one allocated the same amount of memory using hugepages.

For simplicity, we’ll start with the latter program, hereby referred to as the hugepages application. To prepare the server for using hugepages, I added this line to /etc/sysctl.conf:

The current (default) regular page size and hugepage size was found here:

Based on the default hugepage size of the system, the number of hugepages allocated would set me up with enough hugepages (20 GB to be exact) to map the entire hugepages application.

After a reboot, I could see that the hugepages were preallocated by the system, and the 20 GB of memory were already mapped:

The pagetables size for mapping 19 GB of memory using hugepages is relatively small:

(Note that this number also includes the page table entries for mapping the other processes run on this server at that time. I didn’t calculate the exact number. )

This is the contents of memory_hugepages.c, the hugepages application I ran in my experiment:

As one can see, the application allocated 19 GB of memory, filled it with dummy data, and slept for five minutes on order for me to retrieve the page table data needed before the application free up the memory.

I ran the application like this:

…and as soon as the memory had been allocated I could see that both the size of the page table and the amount of used memory were approximately the same as before:

Next, I repeated the experiment, but by running the application that uses regular memory allocations. This is the contents of memory_normal.c:

I ran the application, and after it had allocated the memory (the same amount as for the hugepages application), is could see that the page table size had grown almost 40 MB (subtract the initial page tables size from the number below to find the exact increase in size):

As we can see, using regular memory allocations for mapping an application of this size requires a lot more resources than mapping the same memory using hugepages. So if you are running applications which require lots and lots of memory, you may want to consider using hugepages if the application supports this feature.

Leave a Reply

%d bloggers like this: