Java-based Middleware Notes

• Overview of 3-Tier Systems
  

  One class of common, current commercial workloads are 3-tier systems: Tier 1 typically consists of "thin-clients" such as a web browsers or dumbish terminals. These connects to Tier 2, which is a web server with dynamic content incorporating business logic and presentation software, which in turn connects to Tier 3, a database. Tier 2, the "Middleware" is often written in Java, and is part of a "rapidly-growing market," with performance characteristics which are different from many other kinds of commercial workloads such as on-line transaction processing (OLTP).

  Two Java based middleware benchmarks are examined in this paper: SPECjbb and ECperf (now SPECjAppServer2001). Both intend to highlight the characteristics of Java middleware, but SPECjbb is self-contained (combining all three tiers into one relatively easy to setup and run application) whereas ECperf is just the middleware - it requires driver clients and a database back-end - making it more accurately represent a real system, yet much more complicated to setup, tune, and run.

• Common Characteristics
  

  SPECjbb and ECperf share a number of characteristics, some of which are different that those of other commercial workloads such as OLTP.

  • Low CPI and low memory stall times compared to OLTP
  • Smaller working sets compared to OLTP
  • Many cache misses are satisfied by cache-to-cache transfers, making this an essential machine parameter to optimize

• Different Characteristics
  

  SPECjbb, by nature of being self-contained, has some different characteristics compared to ECperf.

  • Particularly as the workloads scale, SPECjbb's data working set increases nearly linearly (for awhile), while ECperf's levels off. This is due to the structures used to model a database increase in size as the workload increases.
  • ECperf tends to have a much larger instruction working set.

• Trade-off Between Benchmark Simplicity and "Accuracy"
  

  These differences highlight the danger of using one benchmark as the only representative of a class of workloads: it is easy to draw incorrect conclusions, in this case about the memory footprint as the workloads scale. These differences also point out a characteristic of benchmarks in general - the trade-off between a benchmark's simplicity and the accuracy with which it represents a "real" workload. The results of many benchmarks are completely representative of only themselves, and can easily be taken out of context by architects and marketers alike.