Several studies have claimed that disks are a main bottleneck in the performance of busy proxies. We wanted to analyse the impact of spreading the cached files over multiple disks on proxy performance. We expected that by increasing the number of disks, queueing overheads would be reduced, the time spent servicing each disk request would be shortened and, ultimately this would reflect on the overall performance of the proxy.
Only two of the four proxy servers in our testbed allowed us to specify multiple directories for cache storage: Squid and Proxy N. In this section, we present results for both Squid and Proxy N when the cache storage is spread over one and two disks. We also compare these results with those collected when caching was disabled. For these experiments, we used the same cache size (75 MB), since we are interested only in undertanding the impact of one extra disk in proxy performance. Table 1 shows these results for Squid. Surprisingly, there is no improvement in the overall performance when we add an extra disk. Actually, there is a small slowdown in latency of both phases, but hit ratio remains around the same. Table 2 shows results for Proxy N. For Proxy N, the extra disk guaranteed an improvement of 10% in client latency. Hit ratio remained around the same. However, the number of errors occurred in the first phase decreased by 20%. This is probably a side effect of the improvement in latency: because requests are handled faster, new requests can be taken out from the pending connections queue faster and the probability of finding this queue full decreases.
Comparing Proxy N's latency for two disk and no caching experiments, it
can be seen that adding the extra disk indeed alleviates the disk bottleneck
for this proxy. In the no-caching experiment, there is minimal load on
the disk.
In the caching experiment, when the proxy has only one disk arm to use, the
client latency in the first phase is increased by 46%, mostly contributed
by the disk bottleneck.
However, when the proxy has two disk arms to use, the latency increase is
limited to only 30%.
Furthermore, when the proxy has two disk arms to use, the processing of
cache hits is sped up and the second phase latency is significantly
improved.
However, despite the improvements, we were very surprised with these results, since we expected a more drastic improvement in latency. During those experiments, we also collected disk, processor and paging activities using vmstat and iostat in order to have a better picture of how the system resources are consumed by the proxies.
Table 3 show the main statistics collected by these tools for Proxy N. Disk is clearly a bottleneck, since it was busy over 94% of time. With the extra disk, read and write requests were spread over two disks and, as a consequence, service time (svc_t) which is the average time spenting servicing a request dropped significantly, due mainly to reduction of queueing delay. The average queue length ( wait ) drops to almost 0 for both disks. The total number of read and write operations is bigger for the two-disk experiment as a consquence of less contention to disk access. Disk throughput increases as well as disk bandwidth. Processor and paging activity remains almost the same, with slight increase in cpu utilization, both in terms of user and system modes. However, the statistics show that the load is not evenly distributed between the disks: disk one received a bigger number of requests. This may be one explanation for the less-than-expected improvements from the two disk arms, and we are still investigating the issues.
Table 4 shows similar results for Squid. It is clear that the disk bottleneck was reduced when one extra disk was added. The service time (svc_t) was reduced by almost 50% for both disks. The queue length ( actv) and busy time were also reduced. As a consequence, disk throughtput and bandwidth increased. However, this improvements were not reflected in the client latency. Processor utilization remained about the same but paging activity increases when the extra disk is added. We are still in the process of finding out why Squid behaves sub-optimally, and why Squid and Proxy N behaves so differently.