Assignment 0

Finish it soon so that you can start Assignment #1

Overview

This assignment will verify you can access Microsoft Azure and introduce you to it's management interfaces.  You will setup a cluster of 4 VMs using Azure Command-Line Interface (Azure CLI). This cluster will be used to carry out the remaining assignments during the course of the semester.

Learning Outcomes

After completing this assignment, students should:

  1. Be able to use Azure's management interface
  2. Be able to use Azure CLI to setup a cluster
  3. Understand the pay-as-you-go billing model

Details

Create Accounts

Microsoft Azure offers a one-month free trial for any new customer and provides Azure credits of $200 at no charge, for a month. Students are encouraged to use these credits to carry out the course assignments.

Initially, only one student in a group should sign up for the free trial using your wisc id (Eg: badger@wisc.edu). The other student should sign up for the free trial a month later. You can sign up for the free trial here. You will need a phone number (to receive your confirmation code) and a credit/debit card (which will be charged for any usage beyond the free trial).

Install Azure CLI

The Azure CLI is a set of open-source, cross-platform commands for working with the Azure platform. It provides an easy way to setup a cluster using custom scripts. We will be using the Azure CLI to set up the cluster. In order to install the Azure CLI on your PC, you need to perform either of the following steps:

  1. Mac OS X: Use the MAC OS X installer to setup the CLI

  2. Windows: Use the Windows installer to setup the CLI

  3. Linux:
    1. Install the required packages:
    1. sudo apt-get update
    2. sudo apt-get install nodejs
    3. sudo apt-get install npm
    1. Download the linux tar file
    2. Run the following command: sudo npm install -g <path to downloaded tar file>

  4. Other platforms: Follow the Azure CLI documentation to setup the CLI

In order to check if the Azure CLI installation was successful, you should type azure help in the command line. If the installation was successful, the expected output is the CLI usage information.

Cluster Deployment

Microsoft Azure provides two models of deployement - Classic model and Resource Manager model. The latter deployment model introduced the concept of a resource group. A resource group enables the deployment, management, and monitoring of all resources that share a common life cycle, in a single coordinated operation. You will be using the Resource Manager deployment model to setup your cluster.

The cluster that you will deploy consists of 4 VMs connected by a virtual network. Each VM runs Ubuntu 14.04 and has 3 disks associated with it - OS disk, data disk and temporary scratch disk.

To deploy the cluster, you need to download the deployment scripts archive and perform the following steps:

  1. Untar the deployment archive (tar -xvzf deployment-archive.tar.gz) in an appropriate location. The archive consists of the following files:

  1. cluster-deployment-template.json: JSON file that defines the cluster resources that need to be deployed
  2. cluster-deployment-parameters.json: JSON file that defines the parameters passed to the template during deployment
  3. cluster-deployer.sh: Helper script to deploy the cluster as a resource group

  1. Modify the value of the adminPassword parameter in cluster-deployment-parameters.json file. This password will be used to log into the VMs of the cluster. The default password is Ubuntu123$.

  2. Log in to Azure from the Azure CLI by typing the azure login command in the command prompt. This command requires you to log in interactively through a web portal.

  3. To begin the deployment, run the cluster deployment helper script: ./cluster-deployer.sh group<group_number>. Replace <group_number> with your group number.

The cluster deployment would take a few minutes. Once the deployment is successful, verify you can connect to the VMs using SSH by typing the following command in the command prompt: ssh ubuntu@cs838fall2016group<group_number><vm_number>.eastus.cloudapp.azure.com. The <vm_number> placeholder takes values 1, 2, 3 or 4. Replace <group_number> and <vm_number> with appropriate values.

Cluster Deployment Clarifications

The cluster deployment may fail incase the CPU core limit is reached(Expected Error: error: QuotaExceeded : Operation results in exceeding quota limits of Core. Maximum allowed: 4, Current in use: 0, Additional requested: 16). By default, for a free trial Azure may limit the number of cores in a region to 4. You can check the limit for your account by typing the command: azure vm list-usage eastus. The cluster deployment requires 16 cores and the limit on the number of cores can be increased via the Microsoft Azure Portal:

  1. Click on Help+Support->New Support Request->Basics and select:

  1. Issue type as Quota
  2. Subscription corresponding to the free trial
  3. Quota type as Cores
  4. Support plan as Quotas - Support Included

  1. Next, click on Problem and select:

  1. Appropriate severity
  2. Deployment model as Resource Manager
  3. Location as East US
  4. SKU family as Dv2 Series
  5. New quota as 16

  1. Proceed to give your contact information and submit the request.

It normally takes a few hours for the request to be processed. Once the core limit is increased, you can go ahead and retry the cluster deployment.

Data Disk Provisioning

The cluster deployment template associates a 50GB data disk with each VM. In order to use the data disk, you need to SSH into each VM to partition, format, and mount the data disk. You need to perform the following steps in every VM of the cluster:

  1. First, run the df -H command to find the corresponding disk devices of your OS disk and temporary disk. The OS disk will be mounted at the path / and the temporary disk will be mounted at the path /mnt. Next, find the data disk, using the command dmesg | grep SCSI. The output should list 3 attached disks. Notice the name of third disk apart from the OS disk and temporary disk. From now onwards we will refer to the data disk as sdc.

  2. Partition the disk, make it a primary disk on partition 1, and accept the other defaults. Type sudo fdisk /dev/sdc command in the command prompt and perform the following steps:

  1. Type n to create a new partition
  2. Type p to create a primary partition
  3. Type 1 as the partition number
  4. Press enter twice to accept the default values for the remaining parameters
  5. Type w to write these changes to disk

  1. Write ext4 file system to the partition by typing sudo mkfs -t ext4 /dev/sdc1 in the command prompt.

  2. Create a directory to mount the file system: sudo mkdir /workspace .

  3. Mount the directory: sudo mount /dev/sdc1 /workspace .

  4. Verify that the data disk is mounted properly: df -H. One of the entries in the output should contain /dev/sdc1.

To ensure that the data disk is remounted automatically after a reboot, it must be added to the /etc/fstab file. You need to perform the following steps:

  1. Find the UUID of /dev/sdc1 using the blkid utility: sudo -i blkid. In the following, we refer to the UUID of the data disk as <data_disk_uuid>.

  2. Open the /etc/fstab file: sudo vi /etc/fstab and add the following line to the file:

UUID=<data_disk_uuid>    /workspace    ext4    defaults,discard    1    2

For more detailed instructions regarding data disk provisioning, please refer the Azure CLI documentation regarding the same.

Clean-Up

To ensure that the $200 Azure credits are not wasted, students must always deallocate all the 4 VMs when the cluster is not in use. Merely stopping the VMs is of no good - you will still be charged for compute hours as well as storage. However, when the VMs are deallocated, you will be only charged for the storage of the OS disk (/dev/sda) and the data disk (/dev/sdc). Another thing to note is that when the VMs are deallocated, any data on the temporary disk (/dev/sdb mounted on /mnt) will be lost.

In order to deallocate the 4 VMs, type the following command: azure vm deallocate group<group_number> vm1;azure vm deallocate group<group_number> vm2;azure vm deallocate group<group_number> vm3;azure vm deallocate group<group_number> vm4. Replace <group_number> with your group number.

In order to start the 4 VMs after deallocation, type the following command: azure vm start group<group_number> vm1;azure vm start group<group_number> vm2;azure vm start group<group_number> vm3;azure vm start group<group_number> vm4. Replace <group_number> with your group number.

Confirm Billing

You can manage you cluster and keep a track of the Azure credits using the Microsoft Azure Portal. View your account usage in Azure. (It may take up to a day for the usage to be registered.) Confirm that you incurred no actual charges apart from the deduction in the Azure credits.

Deliverables

You do not need to submit anything for this assignment. It is assumed you will complete this assignment for your own benefit.

Clarifications