Context-Switching Weirdness

Updated January 31, 2004

© Copyright 2004 by CyberLife Labs, LLC
All rights reserved

This page exists to document a minor research project whose purpose is to understand exactly why two very similar computers are exhibiting such wildly different context-switching performance characteristics.

Background

CyberLife Labs uses FreeBSD almost exclusively for its scientific, production and administrative needs. One machine in particular (appserver.geo) has always been known to run very fast, but it was never given much thought until recently. As an exercise, the BYTE Benchmark suite was installed and run on that unit along with another, nearly identical but slightly faster machine (beastie.lab). Most of the results were as expected, with beastie.lab edging out appserver.geo by a narrow margin. However, the context-switching results clocked appserver.geo at four-times the rate of it's slightly faster cousin. Considering the types of jobs these two machines typically run (very process-switching oriented), that difference explained why appserver.geo always seemed to get things done quicker. The only question was why.

System Configurations

The following table lists the configuration details of the two systems:

Both systems were run with the exact same kernel configuration, compiled with the same compiler settings.

BYTE Benchmark Results

The following table lists the raw figures output from the BYTE Benchmark suite:

All tests were run in single-user mode to eliminate any chance of interference from other processes. The first nine tests all measure CPU performance. As is expected, beastie.lab performed about 4% faster than appserver.geo. The pipe-throughput, file-read and recursion tests are highly CPU dependent and thus show a similar 4% difference. The file-write and file-copy tests rate significantly higher on appserver.geo, but this is due to the use of an ATA100 disk versus the ATA33 on beastie.lab. (This was verified by forcing appserver.geo down to ATA33 and re-running the benchmark, at which point the file-write and file-copy tests were identical between the systems.) All other tests are highly dependent on context-switching and thus rate much higher on appserver.geo.

So why does appserver.geo task-switch so much better? It's not a CPU issue as the benchmarks correctly show beastie.lab to be the faster system. It's not a disk issue since, again, the benchmarks show beastie.lab to be faster (when both run UDMA33). Even if there were a large difference in disk performance, the context-switching tests don't use the disk subsystem at all. It's not a network issue since the NICs were completely shutdown. appserver.geo has twice the memory capacity of beastie.lab, but as with the disk subsystem, the context-switching tests are not memory-intensive. That leaves only memory and cache performance.

RAMSPEED Benchmark

To determine if memory or cache was affecting the results, Alasir's RAMSPEED benchmark was used to directly stress-test both subsystems. The following table shows the results (in KB/s):

Once again, beastie.lab shows an approximate 4% advantage over appserver.geo both with cache hits (block sizes <= 64KB) and cache misses (block sizes > 64KB), just as it should. This would suggest that neither memory nor cache are the culprit.

What's Next?

We've tested virtually everything we can think of and are still no closer to the answer than when we started. We figure the next step is to track down some of the FreeBSD kernel developers and find out exactly what affects context-switching performance. Hopefully that will give us an idea of where we should be looking.