Searching for the Sorting Record: Experience with NOW-Sort
Searching for the Sorting Record: Experience with NOW-Sort
We present our experiences in developing and tuning the performance of
NOW-Sort, a parallel, disk-to-disk sorting algorithm. NOW-Sort currently
holds two world records in database-industry standard benchmarks.
Critical to the tuning process was the setting of expectations,
which tell the programmer both where to tune and when
to stop. We found three categories of useful tools: tools that
help set expectations and configure the application to different hardware
parameters, visualization tools that animate performance counters, and
search tools that track down performance anomalies. All such tools must
interact well with all layers of the underlying software (e.g., the
operating system), as well as with applications that leverage modern OS
features, such as threads and memory-mapped I/O.