Complexity and Big-O Notation

Contents

• Introduction
  • Test Yourself #1
  • Test Yourself #2
• Big-O Notation
• How to Determine Complexities
  • Test Yourself #3
  • Test Yourself #4
• Best-case and Average-case Complexity
  • Test Yourself #5
• When do Constants Matter?
• Answers to Self-Study Questions

An important question is: How efficient is an algorithm or piece of code? Efficiency covers lots of resources, including:

• CPU (time) usage
• memory usage
• disk usage
• network usage

All are important but we will mostly talk about CPU time in 367. Other classes will discuss other resources (e.g., disk usage may be an important topic in a database class).

Be careful to differentiate between:

1. Performance: how much time/memory/disk/... is actually used when a program is run. This depends on the machine, compiler, etc. as well as the code.
2. Complexity: how do the resource requirements of a program or algorithm scale, i.e., what happens as the size of the problem being solved gets larger.

The time required by a method is proportional to the number of "basic operations" that it performs. Here are some examples of basic operations:

• one arithmetic operation (e.g., +, *).
• one assignment
• one test (e.g., x == 0)
• one read
• one write (of a primitive type)

Some methods perform the same number of operations every time they are called. For example, the size method of the Sequence class always performs just one operation: return numItems; the number of operations is independent of the size of the sequence. We say that methods like this (that always perform a fixed number of basic operations) require constant time.

Other methods may perform different numbers of operations, depending on the value of a parameter or a field. For example, for many of the Sequence methods, the number of operations depends on the size of the sequence. We call the important factor (the parameters and/or fields whose values affect the number of operations performed) the problem size or the input size.

When we consider the complexity of a method, we don't really care about the exact number of operations that are performed; instead, we care about how the number of operations relates to the problem size. If the problem size doubles, does the number of operations stay the same? double? increase in some other way? For constant-time methods like the size method, doubling the problem size does not affect the number of operations (which stays the same).

Furthermore, we are usually interested in the worst case: what is the most operations that might be performed for a given problem size (other cases -- best case and average case -- are discussed below). For example, the addBefore method (for a sequence implemented using an array) has to move the current item and all of the items that come after the current item one place to the right in the array. In the worst case, all of the items in the array must be moved. Therefore, in the worst case, the time for addBefore is proportional to the number of items in the sequence, and we say that the worst-case time for addBefore is linear in the number of items in the sequence. For a linear-time method, if the problem size doubles, the number of operations also doubles.

Assume that sequences are implemented using an array. For each of the following Sequence methods, say whether (in the worst case) the number of operations is independent of the size of the sequence (is a constant-time method), or is proportional to the size of the sequence (is a linear-time method):

• the constructor
• addAfter
• start
• getCurrent
• lookup

solution

Constant and linear times are not the only possibilities. For example, consider method createSeq:

Sequence createSeq( int N ) {
  Sequence s = new Sequence();
  for (int k=1; k<=N; k++) s.addBefore(new Integer(k));
  return s;
}

s.addBefore( new Integer(1) );
s.addBefore( new Integer(2) );
s.addBefore( new Integer(3) );
    ...
s.addBefore( new Integer(N) );

Recall that we don't care about the exact time, just how the time depends on the problem size. For method createSeq, the "problem size" is the value of N (because the number of operations will be different for different values of N). It is clear that the time for the N calls (and therefore the time for method createSeq) is not independent of N (so createSeq is not a constant-time method). Is it proportional to N (linear in N)? That would mean that doubling N would double the number of operations performed by createSeq. Here's a table showing the value of 0+1+2+...+(N-1) for some different values of N:

Clearly, the value of the sum does more than double when the value of N doubles, so createSeq is not linear in N. In the following graph, the bars represent the lengths of the sequence (0, 1, 2, ..., N-1) for each of the N calls.

The value of the sum (0+1+2+...+(N-1)) is the sum of the areas of the individual bars. You can see that the bars fill about half of the square. The whole square is an N-by-N square, so its area is N²; therefore, the sum of the areas of the bars is about N²/2. In other words, the time for method createSequence is proportional to the square of the problem size; if the problem size doubles, the number of operations will quadruple. We say that the worst-case time for createSeq is quadratic in the problem size.

Consider the following three algorithms for determining whether anyone in the room has the same birthday as you.

Algorithm 1: You say your birthday, and ask whether anyone in the room has the same birthday. If anyone does have the same birthday, they answer yes.

Algorithm 2: You tell the first person your birthday, and ask if they have the same birthday; if they say no, you tell the second person your birthday and ask whether they have the same birthday; etc, for each person in the room.

Algorithm 3: You only ask questions of person 1, who only asks questions of person 2, who only asks questions of person 3, etc. You tell person 1 your birthday, and ask if they have the same birthday; if they say no, you ask them to find out about person 2. Person 1 asks person 2 and tells you the answer. If it is no, you ask person 1 to find out about person 3. Person 1 asks person 2 to find out about person 3, etc.

Question 1: What is the factor that can affect the number of questions asked (the "problem size")?

Question 2: In the worst case, how many questions will be asked for each of the three algorithms?

Question 3: For each algorithm, say whether it is constant, linear, or quadratic in the problem size in the worst case.

solution

• a constant-time method is "order 1": O(1)
• a linear-time method is "order N": O(N)
• a quadratic-time method is "order N squared": O(N²)

Formal definition:

A function T(N) is O(F(N)) if for some constant c and for values of N greater than some value n₀: T(N) <= c * F(N)

For example, consider T(N) = 3 * N² + 5. We can show that T(N) is O(N²) by choosing c = 4 and n₀ = 2. This is because for all values of N greater than 2:

T(N) is not O(N), because whatever constant c and value n₀ you choose, I can always find a value of N greater than n₀ so that 3 * N² + 5 is greater than c * N.

How to Determine Complexities

1. Sequence of statements

   statement 1;
   statement 2;
     ...
   statement k;
   

   (Note: this is code that really is exactly k statements; this is not an unrolled loop like the N calls to addBefore shown above.) The total time is found by adding the times for all statements:

   total time = time(statement 1) + time(statement 2) + ... + time(statement k)

   If each statement is "simple" (only involves basic operations) then the time for each statement is constant and the total time is also constant: O(1). In the following examples, assume the statements are simple unless noted otherwise.

2. if-then-else statements

   if (cond) {
       sequence of statements 1
   }
   else {
       sequence of statements 2
   }
   

   Here, either sequence 1 will execute, or sequence 2 will execute. Therefore, the worst-case time is the slowest of the two possibilities: max(time(sequence 1), time(sequence 2)). For example, if sequence 1 is O(N) and sequence 2 is O(1) the worst-case time for the whole if-then-else statement would be O(N).

3. for loops

   for (i = 0; i < N; i++) {
       sequence of statements
   }
   

   The loop executes N times, so the sequence of statements also executes N times. Since we assume the statements are O(1), the total time for the for loop is N * O(1), which is O(N) overall.

4. Nested loops

   for (i = 0; i < N; i++) {
       for (j = 0; j < M; j++) {
           sequence of statements
       }
   }
   

   The outer loop executes N times. Every time the outer loop executes, the inner loop executes M times. As a result, the statements in the inner loop execute a total of N * M times. Thus, the complexity is O(N * M). In a common special case where the stopping condition of the inner loop is j < N instead of j < M (i.e., the inner loop also executes N times), the total complexity for the two loops is O(N²).

   ---

   TEST YOURSELF #3

   What is the worst-case complexity of the each of the following code fragments?

   1. Two loops in a row:

      for (i = 0; i < N; i++) {
          sequence of statements
      }
      for (j = 0; j < M; j++) {
          sequence of statements
      }
      

      How would the complexity change if the second loop went to N instead of M?

   2. A nested loop followed by a non-nested loop:

      for (i = 0; i < N; i++) {
          for (j = 0; j < N; j++) {
              sequence of statements
          }
      }
      for (k = 0; k < N; k++) {
          sequence of statements
      }
      

   3. A nested loop in which the number of times the inner loop executes depends on the value of the outer loop index:

      for (i = 0; i < N; i++) {
          for (j = i; j < N; j++) {
              sequence of statements
          }
      }
      

   solution

   ---

5. Statements with method calls:

   When a statement involves a method call, the complexity of the statement includes the complexity of the method call. Assume that you know that method f takes constant time, and that method g takes time proportional to (linear in) the value of its parameter k. Then the statements below have the time complexities indicated.

   f(k);  // O(1)
   g(k);  // O(k)
   

   When a loop is involved, the same rule applies. For example:

   for (j = 0; j < N; j++) g(N);
   

   has complexity (N²). The loop executes N times and each method call g(N) is complexity O(N).

   ---

   TEST YOURSELF #4

   For each of the following loops with a method call, determine the overall complexity. As above, assume that method f takes constant time, and that method g takes time linear in the value of its parameter.

   1. for (j = 0; j < N; j++) f(j);
   
   2. for (j = 0; j < N; j++) g(j);
   
   3. for (j = 0; j < N; j++) g(k);
   

   solution

   ---

Some methods may require different amounts of time on different calls, even when the problem size is the same for both calls. For example, we know that if addBefore is called with a sequence of length N, it may require time proportional to N (to move all of the items and/or to expand the array). This is what happens in the worst case. However, when the current item is the last item in the sequence, and the array is not full, addBefore will only have to move one item, so in that case its time is independent of the length of the sequence; i.e., constant time.

In general, we may want to consider the best and average time requirements of a method as well as its worst-case time requirements. Which is considered the most important will depend on several factors. For example, if a method is part of a time-critical system like one that controls an airplane, the worst-case times are probably the most important (if the plane is flying towards a mountain and the controlling program can't make the next course correction until it has performed a computation, then the best-case and average-case times for that computation are not relevant -- the computation needs to be guaranteed to be fast enough to finish before the plane hits the mountain).

On the other hand, if occasionally waiting a long time for an answer is merely inconvenient (as opposed to life-threatening), it may be better to use an algorithm with a slow worst-case time and a fast average-case time, rather than one with so-so times in both the average and worst cases.

For addBefore, for a sequence of length N, the worst-case time is O(N), the best-case time is O(1), and the average-case time (assuming that each item is equally likely to be the current item) is O(N), because on average, N/2 items will need to be moved.

Note that calculating the average-case time for a method can be tricky. You need to consider all possible values for the important factors, and whether they will be distributed evenly.

Assume that sequences are implemented using arrays. Fill in the missing complexities from the table below for the methods, assuming a sequence length of N. Include an explanation of what conditions lead to each of the complexities you fill in.

operation	Complexity
	best	worst	average
size	O(1)	O(1)	O(1)
addAfter
addBefore	O(1): last item is current and array is not full	O(N): first item is current and/or array is full	O(N): assuming each item equally likely to be current
removeCurrent
start
getCurrent
advance
isCurrent

solution

Recall that when we use big-O notation, we drop constants and low-order terms. This is because when the problem size gets sufficiently large, those terms don't matter. However, this means that two algorithms can have the same big-O time complexity, even though one is always faster than the other. For example, suppose algorithm 1 requires N² time, and algorithm 2 requires 10 * N² + N time. For both algorithms, the time is O(N²), but algorithm 1 will always be faster than algorithm 2. In this case, the constants and low-order terms do matter in terms of which algorithm is actually faster.

However, it is important to note that constants do not matter in terms of the question of how an algorithm "scales" (i.e., how does the algorithm's time change when the problem size doubles). Although an algorithm that requires N² time will always be faster than an algorithm that requires 10*N² time, for both algorithms, if the problem size doubles, the actual time will quadruple.

When two algorithms have different big-O time complexity, the constants and low-order terms only matter when the problem size is small. For example, even if there are large constants involved, a linear-time algorithm will always eventually be faster than a quadratic-time algorithm. This is illustrated in the following table, which shows the value of 100*N (a time that is linear in N) and the value of N²/100 (a time that is quadratic in N) for some values of N. For values of N less than 10⁴, the quadratic time is smaller than the linear time. However, for all values of N greater than 10⁴, the linear time is smaller.