The Non-Uniform Memory Architecture (NUMA) systems are common in enterprise computing today: almost all high-end large-memory systems are NUMA machines, and even the most common mid-range servers (32 to 40 cores, under 500G of memory) are usually NUMA systems. For something so widely used, one would expect NUMA and its impact on program performance to be well understood. Sadly, it’s not. NUMA systems present a Non-Universal Menagerie of Attributes and their behavior is devilishly complex. Practical consequences range from “ignore the fact that it’s NUMA and you’re fine” to “the program runs much faster on 16 CPUs than on 32.” To make the matters worse, reliable measurements are hard to collect, the act of measuring often influences the behavior, the standard profiling tools are at best inadequate and at worst misleading, and concrete knowledge is usually inferred from somewhat opaque behavior. In this talk, I present my experiences and lessons learned from working with NUMA systems. If you never asked yourself, “how did they stuff so much memory into this box?,” worry not, we’ll get you up to speed with the introduction to NUMA. We will then discuss the performance restrictions and problems unique to NUMA systems, and learn how to identify and troubleshoot NUMA-related issues. I will show how the standard and often relied-on profiling tools can mislead you when working on a NUMA system and how to recognize the danger signs. I will also show the solutions we came up with for several very different types of problems: poor scaling, large overhead, and low memory performance. Overall, if you ever work on a NUMA system, this talk just might save you days or weeks of debugging, profiling, and experimentation.
View Slides
Fedor G. Pikus is a Technical Fellow and the Director of the Advanced Projects Team at Siemens Digital Industries Software. His responsibilities include planning the long-term technical direction of Calibre products, directing and training the engineering teams, overseeing the design and architecture of the software, and researching new design and software technologies.
His earlier positions included Chief Scientist at Mentor Graphics (acquired by Siemens Software), Senior Software Engineer at Google, and Chief Software Architect for Calibre PERC, LVS, and DFM. He joined Mentor Graphics in 1998, making the switch from academic research in computational physics to the software industry.
Fedor is a recognized expert in high-performance computing and C++. He is an O'Reilly author and has written three books on C++ and software design. He is a regular instructor at the CppCon Academy, leading two of the best-attended classes, and has presented his work at CppNow, CppCon, CppNorth, SD West, DesignCon, and in various software development journals. Fedor holds over 30 patents and has authored over 100 papers and conference presentations on physics, EDA, software design, and the C++ language.