HugeTLBfs

See Huge Pages (hugetlbfs) for assorted on-line documentation about using large pages to avoid excessive load on the TLB.

Here we do not claim to be complete for all target architectures. This is meant to serve as an example for Linux targets. The following steps are what we did on one RHEL5 system.

NOTE: If you have kernel documentation installed, see: /usr/share/doc/kernel-doc-<version>/Documentation/vm/hugetlbpage.txt

First steps (most of this must be done by the super-user):

• Determine that our kernel supports hugetlbfs.
  • grep Hugepagesize /proc/meminfo
• Ensure that we have adequate huge pages (this example is for an 8GB buffer pool) on reboot:
  • echo "vm.nr_hugepages=4096" >> /etc/sysctl.conf To dynamically allocate pages,
  • echo 4096 > /proc/sys/vm/nr_hugepages
• Create a group for users of the hugetlbfs using your sys admin applications. My group is called "ssm" and has gid 55555.
• Add users to the ssm group.
• Create the mount point for a hugetlbfs:
  • mkdir -p /mnt/huge
• Mount a pseudo filesystem of type hugetlbfs at that mount point; do this on reboot:
  • echo "none /mnt/huge hugetlbfs rw,gid=55555,size10g,mode=0770 0 0" > /etc/fstab

• Reboot.

Warning:

If you have configured the storage manager for use with hugetlbfs and have not taken the above steps to ensure that your system actually has the hugetlbfs pages available for use, your storage manager (or the 'make check' tests) will likely croak. Unfortunately, we have not yet figured out a way to determine ahead of time, programmatically, whether things will go well or not, before we try to write to the huge pages.

Second steps (this can be done by users in the ssm group):

• Edit the default path for the hugetlbfs node in config/shore.def.
• Create a file in the hugetlbfs, owned by the ssm group, whose name matches the default path in config/shore.def, e.g.,
  • touch /etc/huge/SSM-BUFPOOL
• Configure and build the storage manager with --with-hugetlbfs.
• To run using the hugetlbfs is now the default; to run without it, change the sm_hugetlbfs_path run-time option to the value "NULL"; here is an example from the .shoreconfig file in the smsh directory:
  • *.server.*.sm_hugetlbfs_path: NULL

Note that your buffer pool size will have to be set to a multiple of the huge page size for your system. Thus, if your huge pages are 2 MB you will get an error from mmap if you use a 3 MB buffer pool.

On the whole, the use of mmap with the hugetlbfs is not reliable and, although at process end, all huge pages are supposed to be returned to the system, we have seen cases in which pages were "lost" and the mmap thereafter failing, repaired only on reboot, so consider this feature for performance experiments only on systems that do not require high availability.

---