Assignment 2

Due Monday, September 24 at 11:59pm

Overview

This assignment will introduce you to the services offered by major Infrastructure-as-a-Service (IaaS) providers and allow you to gain experience using these services.  You will provision compute, storage, and networking resources in Amazon Web Services and Windows Azure and evaluate the performance of a subset of these services.  You will produce a short report detailing your experimental methodology, performance observations, and takeaways.

Learning Outcomes

After completing this assignment, students should:

  1. Be able to identify the types of the services offered by IaaS cloud providers
  2. Understand how to provision compute, storage, and network services
  3. Be able to design experiments that illustrate the capabilities of specific IaaS services

Details

IaaS providers offer a variety of different services, including compute, networking, storage, database, and messaging. You will evaluate a subset of these services on Amazon Web Services (AWS) and Windows Azure.

Compute & Network

Amazon Elastic Compute Cloud (EC2) and Windows Azure provide individual virtual machines with several tiers of processing, memory, and network capabilities.  You should evaluate both the compute and network performance of these different tiers.

Due to the constraints of our grants you only need to consider:

  1. In AWS: Small, Medium, and Large Standard Instances
  2. In Azure: Small VMs

You can use standard benchmarks and measurement tools for conducting measurements.  You may consider using:

  1. SPECjvm2008 (http://www.spec.org/jvm2008/)
  2. Any of several open source benchmarking tools (http://wikipedia.org/wiki/Benchmark_%28computing%29#Open_source_benchmarks)
  3. Ping
  4. iPerf (available via apt in Ubuntu)

You should be sure to address at least the following questions:

  1. How do standard compute benchmarks perform on specific types of EC2 instances?  Is the increase in performance with larger instances commensurate with the increase in resources (cores, EC2 compute units, memory, etc) these instances offer?
  2. How does the performance of compute benchmarks on a specific type of EC2 instance compare with the performance of your local machine?  (Machines in the mumble lab in the CS building would be a reasonable comparison point to a large EC2 instance.)
  3. What network performance is experienced between instances of the same type?  Different types?  Is this performance consistent over time (e.g., time of day, different pairs of instances, etc.)?
  4. How does the compute and network performance of a small EC2 instance compare to the compute and network performance of a small Azure VM?

You may find it useful to refer to the paper “CloudCmp: Comparing Public Cloud Providers” by Ang Li et al. published in IMC 2010.

Storage

Amazon Web Services recently announced Provisioned IOPS for its Elastic Block Storage (EBS) service (http://aws.amazon.com/about-aws/whats-new/2012/07/31/announcing-provisioned-iops-for-amazon-ebs/).  You should compare the performance of Provisioned IOPS to the regular EBS service.  You can use standard benchmarks and tools for conducting performance measurements. You should be sure to address at least the following questions:

  1. How much performance variation exists with regular EBS?  Is the performance consistent over time (e.g., time of day) and space (e.g., parts of an EBS volume, different EBS volumes)?
  2. How does the measured performance with Provisioned IOPS compare to the stated performance (e.g., is the performance slightly better than the stated performance)?
  3. Do the compute, memory,and networking capabilities of a compute instance affect the performance of Provisioned IOPS or regular EBS: e.g., does the CPU become a bottleneck on a micro instance, or does storage performance degrade if you are heavily using the network interface on the instance for other tasks?

Deliverables

You should provide a brief write-up (3 to 4 pages using standard ACM SIG Proceedings Alternate format, http://www.acm.org/sigs/publications/proceedings-templates#aL2) detailing your experimental methodology, performance observations, and takeaways.  Your takeaways should provide insights into how applications using these services would perform, how you believe the services could be improved, and/or what design considerations should be taken into account when leveraging these services.

Clarifications

  1. You should submit a PDF of your report (by Monday, September 24 at 11:59pm) by placing it in your assign2 hand-in directory for the course: ~cs838-1/handin/$USER/assign2/
  2. Please include, at the end of your report, how many EC2 grant credits you have remaining.  This will help us ensure the course assignments can reasonably be completed within the limits of the free usage tiers and grants.  You can find this information on the Activity Summary page for your AWS account (https://portal.aws.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=activity-summary).  Note that the activity summary is updated daily, so it may take up to 24 hours for all usage to appear.
    EDIT: The EC2 credits remaining are not updated until the end of the billing period. Instead include in your report the number of used credits listed under Details > AWS Service Charges, which is updated daily.
  3. If you are taking the qual this semester (which is on the same day the assignment is due) and are concerned about timing, please come talk to Aaron (CS 7376).
  4. The assignment asks you to answer the question: "How do standard compute benchmarks perform on specific types of EC2 instances?" It is not necessary to run a long compute benchmark. A few smaller benchmarks that illustrate performance differences would be sufficient.
  5. The assignment asks you to answer the question: "Is [network] performance consistent over time (e.g., time of day, different pairs of instances, etc.)?" It is not necessary to run an instance for many hours to obtain this measure. You can either measure network performance at fine granularity over the course of an hour, or you can measure network performance for a few minutes at different points in the day (e.g., morning, afternoon, and night). Furthermore, you do not need to measure performance over time for all types of instances, one type is sufficient.
  6. Keep in mind your report should be 3 to 4 pages, so you only need to gather enough data to provide a cursory answer to each of the questions posed. If you provide a more in-depth answer to some of the questions, then your answers to other questions can be less in-depth.