criterion performance measurements

overview

want to understand this report?

traverse1000/ListR1000

5.50 μs
5.55
5.60
5.65
5.70
5.75
5.80
traverse1000/ListR1000 time densities
mean
5.50 μs
5.55
5.60
5.65
5.70
5.75
5.80
traverse1000/ListR1000 times
lower bound estimate upper bound
Mean execution time 5.527 μs 5.534 μs 5.547 μs
Standard deviation 28.47 ns 46.97 ns 72.33 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

traverse1000/IMR1000

33.5 μs
34.0
34.5
35.0
traverse1000/IMR1000 time densities
mean
33.5 μs
34.0
34.5
35.0
traverse1000/IMR1000 times
lower bound estimate upper bound
Mean execution time 34.47 μs 34.55 μs 34.61 μs
Standard deviation 314.9 ns 358.5 ns 411.1 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

traverse1000/SeqR1000

23.25 μs
23.50
23.75
24.00
24.25
24.50
traverse1000/SeqR1000 time densities
mean
23.25 μs
23.50
23.75
24.00
24.25
24.50
traverse1000/SeqR1000 times
lower bound estimate upper bound
Mean execution time 23.74 μs 23.77 μs 23.81 μs
Standard deviation 129.6 ns 167.8 ns 236.7 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

traverse1000/VecR1000

37.5 μs
38.0
38.5
39.0
39.5
40.0
40.5
traverse1000/VecR1000 time densities
mean
37.5 μs
38.0
38.5
39.0
39.5
40.0
40.5
traverse1000/VecR1000 times
lower bound estimate upper bound
Mean execution time 38.16 μs 38.27 μs 38.38 μs
Standard deviation 512.5 ns 570.3 ns 702.6 ns

Outlying measurements have slight (7.5%) effect on estimated standard deviation.

traverse1000/SeqL1000

23.4 μs
23.6
23.8
24.0
24.2
traverse1000/SeqL1000 time densities
mean
23.4 μs
23.6
23.8
24.0
24.2
traverse1000/SeqL1000 times
lower bound estimate upper bound
Mean execution time 23.48 μs 23.49 μs 23.52 μs
Standard deviation 36.93 ns 86.64 ns 188.1 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

traverse1000/VecL1000

18.3 μs
18.4
18.5
18.6
18.7
18.8
18.9
traverse1000/VecL1000 time densities
mean
18.3 μs
18.4
18.5
18.6
18.7
18.8
18.9
traverse1000/VecL1000 times
lower bound estimate upper bound
Mean execution time 18.37 μs 18.37 μs 18.39 μs
Standard deviation 35.48 ns 64.37 ns 131.1 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

traverse1000/ListLS1000

4.30 μs
4.35
4.40
4.45
4.50
4.55
4.60
traverse1000/ListLS1000 time densities
mean
4.30 μs
4.35
4.40
4.45
4.50
4.55
4.60
traverse1000/ListLS1000 times
lower bound estimate upper bound
Mean execution time 4.303 μs 4.310 μs 4.322 μs
Standard deviation 24.99 ns 42.67 ns 69.01 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

traverse1000/SeqLS1000

46.0 μs
46.5
47.0
47.5
traverse1000/SeqLS1000 time densities
mean
46.0 μs
46.5
47.0
47.5
traverse1000/SeqLS1000 times
lower bound estimate upper bound
Mean execution time 46.27 μs 46.30 μs 46.35 μs
Standard deviation 117.0 ns 182.6 ns 346.0 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

traverse1000/VecLS1000

17.90 μs
18.00
18.10
18.20
18.30
traverse1000/VecLS1000 time densities
mean
17.90 μs
18.00
18.10
18.20
18.30
traverse1000/VecLS1000 times
lower bound estimate upper bound
Mean execution time 17.99 μs 18.00 μs 18.01 μs
Standard deviation 42.56 ns 64.16 ns 98.00 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt3_1000/List1000

29.50 ns
29.60
29.70
29.80
29.90
indexAt3_1000/List1000 time densities
mean
29.50 ns
29.60
29.70
29.80
29.90
indexAt3_1000/List1000 times
lower bound estimate upper bound
Mean execution time 29.59 ns 29.60 ns 29.61 ns
Standard deviation 43.38 ps 54.98 ps 83.64 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt3_1000/IM1000

49.25 ns
49.50
49.75
50.00
50.25
50.50
50.75
indexAt3_1000/IM1000 time densities
mean
49.25 ns
49.50
49.75
50.00
50.25
50.50
50.75
indexAt3_1000/IM1000 times
lower bound estimate upper bound
Mean execution time 49.73 ns 49.81 ns 49.89 ns
Standard deviation 357.5 ps 396.8 ps 435.6 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt3_1000/Seq1000

45.00 ns
45.25
45.50
45.75
46.00
46.25
indexAt3_1000/Seq1000 time densities
mean
45.00 ns
45.25
45.50
45.75
46.00
46.25
indexAt3_1000/Seq1000 times
lower bound estimate upper bound
Mean execution time 45.04 ns 45.06 ns 45.10 ns
Standard deviation 81.35 ps 138.2 ps 274.1 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt3_1000/Vec1000

28.2 ns
28.3
28.4
28.5
28.6
28.7
28.8
28.9
indexAt3_1000/Vec1000 time densities
mean
28.2 ns
28.3
28.4
28.5
28.6
28.7
28.8
28.9
indexAt3_1000/Vec1000 times
lower bound estimate upper bound
Mean execution time 28.26 ns 28.27 ns 28.30 ns
Standard deviation 44.89 ps 77.23 ps 152.6 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt30_1000/List1000

131.0 ns
131.5
132.0
132.5
indexAt30_1000/List1000 time densities
mean
131.0 ns
131.5
132.0
132.5
indexAt30_1000/List1000 times
lower bound estimate upper bound
Mean execution time 131.4 ns 131.5 ns 131.5 ns
Standard deviation 222.0 ps 268.8 ps 346.1 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt30_1000/IM1000

53.5 ns
54.0
54.5
55.0
55.5
56.0
56.5
indexAt30_1000/IM1000 time densities
mean
53.5 ns
54.0
54.5
55.0
55.5
56.0
56.5
indexAt30_1000/IM1000 times
lower bound estimate upper bound
Mean execution time 54.09 ns 54.17 ns 54.30 ns
Standard deviation 371.2 ps 517.5 ps 703.3 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt30_1000/Seq1000

86.0 ns
86.5
87.0
87.5
indexAt30_1000/Seq1000 time densities
mean
86.0 ns
86.5
87.0
87.5
indexAt30_1000/Seq1000 times
lower bound estimate upper bound
Mean execution time 86.21 ns 86.24 ns 86.30 ns
Standard deviation 142.3 ps 208.2 ps 377.0 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt30_1000/Vec1000

28.1 ns
28.2
28.3
28.4
28.5
28.6
28.7
indexAt30_1000/Vec1000 time densities
mean
28.1 ns
28.2
28.3
28.4
28.5
28.6
28.7
indexAt30_1000/Vec1000 times
lower bound estimate upper bound
Mean execution time 28.20 ns 28.21 ns 28.23 ns
Standard deviation 49.29 ps 70.55 ps 122.7 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt30_10000/Seq10000

86.0 ns
86.2
86.4
86.6
86.8
87.0
indexAt30_10000/Seq10000 time densities
mean
86.0 ns
86.2
86.4
86.6
86.8
87.0
indexAt30_10000/Seq10000 times
lower bound estimate upper bound
Mean execution time 86.23 ns 86.25 ns 86.29 ns
Standard deviation 137.4 ps 160.4 ps 207.7 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt30_10000/Vec10000

33.0 ns
33.1
33.2
33.3
33.4
33.5
33.6
indexAt30_10000/Vec10000 time densities
mean
33.0 ns
33.1
33.2
33.3
33.4
33.5
33.6
indexAt30_10000/Vec10000 times
lower bound estimate upper bound
Mean execution time 33.10 ns 33.12 ns 33.14 ns
Standard deviation 79.65 ps 97.52 ps 137.6 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt3000_10000/IM1000

58 ns
59
60
61
62
63
64
65
indexAt3000_10000/IM1000 time densities
mean
58 ns
59
60
61
62
63
64
65
indexAt3000_10000/IM1000 times
lower bound estimate upper bound
Mean execution time 63.72 ns 63.95 ns 64.02 ns
Standard deviation 201.1 ps 586.5 ps 1.317 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexAt3000_10000/Seq10000

198 ns
200
202
204
206
indexAt3000_10000/Seq10000 time densities
mean
198 ns
200
202
204
206
indexAt3000_10000/Seq10000 times
lower bound estimate upper bound
Mean execution time 198.8 ns 199.3 ns 199.8 ns
Standard deviation 2.168 ns 2.549 ns 2.886 ns

Outlying measurements have slight (5.4%) effect on estimated standard deviation.

indexAt3000_10000/Vec10000

33.1 ns
33.2
33.3
33.4
33.5
33.6
33.7
33.8
33.9
indexAt3000_10000/Vec10000 time densities
mean
33.1 ns
33.2
33.3
33.4
33.5
33.6
33.7
33.8
33.9
indexAt3000_10000/Vec10000 times
lower bound estimate upper bound
Mean execution time 33.20 ns 33.22 ns 33.25 ns
Standard deviation 72.63 ps 112.7 ps 174.5 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexRange501-550_10000/Seq10000

8.45 μs
8.50
8.55
8.60
indexRange501-550_10000/Seq10000 time densities
mean
8.45 μs
8.50
8.55
8.60
indexRange501-550_10000/Seq10000 times
lower bound estimate upper bound
Mean execution time 8.545 μs 8.549 μs 8.552 μs
Standard deviation 12.76 ns 17.97 ns 30.80 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexRange501-550_10000/Vec10000

1.90 μs
1.95
2.00
2.05
indexRange501-550_10000/Vec10000 time densities
mean
1.90 μs
1.95
2.00
2.05
indexRange501-550_10000/Vec10000 times
lower bound estimate upper bound
Mean execution time 1.883 μs 1.885 μs 1.892 μs
Standard deviation 4.515 ns 16.67 ns 38.45 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexRange5001-5500_100000/Seq10000

109.5 μs
110.0
110.5
111.0
111.5
indexRange5001-5500_100000/Seq10000 time densities
mean
109.5 μs
110.0
110.5
111.0
111.5
indexRange5001-5500_100000/Seq10000 times
lower bound estimate upper bound
Mean execution time 109.7 μs 109.8 μs 109.8 μs
Standard deviation 213.1 ns 288.1 ns 471.2 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

indexRange5001-5500_100000/Vec10000

21.8 μs
22.0
22.2
22.4
22.6
indexRange5001-5500_100000/Vec10000 time densities
mean
21.8 μs
22.0
22.2
22.4
22.6
indexRange5001-5500_100000/Vec10000 times
lower bound estimate upper bound
Mean execution time 21.80 μs 21.81 μs 21.84 μs
Standard deviation 40.86 ns 97.39 ns 212.4 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

appendValue1000/IM1000

67 ns
68
69
70
71
72
73
appendValue1000/IM1000 time densities
mean
67 ns
68
69
70
71
72
73
appendValue1000/IM1000 times
lower bound estimate upper bound
Mean execution time 67.25 ns 67.32 ns 67.55 ns
Standard deviation 162.0 ps 551.5 ps 1.265 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

appendValue1000/Seq1000

25.0 ns
25.2
25.4
25.6
25.8
26.0
appendValue1000/Seq1000 time densities
mean
25.0 ns
25.2
25.4
25.6
25.8
26.0
appendValue1000/Seq1000 times
lower bound estimate upper bound
Mean execution time 25.09 ns 25.12 ns 25.16 ns
Standard deviation 143.1 ps 178.0 ps 237.0 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

appendValue1000/Vec1000

22.2 ns
22.4
22.6
22.8
23.0
23.2
23.4
appendValue1000/Vec1000 time densities
mean
22.2 ns
22.4
22.6
22.8
23.0
23.2
23.4
appendValue1000/Vec1000 times
lower bound estimate upper bound
Mean execution time 22.33 ns 22.35 ns 22.38 ns
Standard deviation 83.16 ps 130.3 ps 247.9 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

appendValue100000/IM100000

71.0 ns
71.5
72.0
72.5
73.0
73.5
74.0
74.5
appendValue100000/IM100000 time densities
mean
71.0 ns
71.5
72.0
72.5
73.0
73.5
74.0
74.5
appendValue100000/IM100000 times
lower bound estimate upper bound
Mean execution time 71.67 ns 71.72 ns 71.83 ns
Standard deviation 177.2 ps 337.7 ps 696.6 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

appendValue100000/Seq100000

25.0 ns
25.5
26.0
26.5
27.0
27.5
appendValue100000/Seq100000 time densities
mean
25.0 ns
25.5
26.0
26.5
27.0
27.5
appendValue100000/Seq100000 times
lower bound estimate upper bound
Mean execution time 25.29 ns 25.34 ns 25.46 ns
Standard deviation 234.2 ps 385.3 ps 597.8 ps

Outlying measurements have slight (8.5%) effect on estimated standard deviation.

appendValue100000/Vec100000

22.65 ns
22.70
22.75
22.80
22.85
22.90
22.95
23.00
23.05
appendValue100000/Vec100000 time densities
mean
22.65 ns
22.70
22.75
22.80
22.85
22.90
22.95
23.00
23.05
appendValue100000/Vec100000 times
lower bound estimate upper bound
Mean execution time 22.75 ns 22.76 ns 22.77 ns
Standard deviation 43.48 ps 52.33 ps 74.64 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

updateIndex5_1000/IM1000

129 ns
130
131
132
133
134
135
updateIndex5_1000/IM1000 time densities
mean
129 ns
130
131
132
133
134
135
updateIndex5_1000/IM1000 times
lower bound estimate upper bound
Mean execution time 130.0 ns 130.0 ns 130.2 ns
Standard deviation 263.6 ps 568.2 ps 1.217 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

updateIndex5_1000/Seq1000

30.5 ns
31.0
31.5
32.0
32.5
33.0
updateIndex5_1000/Seq1000 time densities
mean
30.5 ns
31.0
31.5
32.0
32.5
33.0
updateIndex5_1000/Seq1000 times
lower bound estimate upper bound
Mean execution time 31.25 ns 31.31 ns 31.39 ns
Standard deviation 269.8 ps 352.5 ps 497.0 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

updateIndex5_1000/Vec1000

82 ns
83
84
85
86
87
88
updateIndex5_1000/Vec1000 time densities
mean
82 ns
83
84
85
86
87
88
updateIndex5_1000/Vec1000 times
lower bound estimate upper bound
Mean execution time 82.26 ns 82.33 ns 82.55 ns
Standard deviation 242.7 ps 586.8 ps 1.289 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

updateIndex5000_100000/IM100000

185 ns
190
195
200
updateIndex5000_100000/IM100000 time densities
mean
185 ns
190
195
200
updateIndex5000_100000/IM100000 times
lower bound estimate upper bound
Mean execution time 185.5 ns 185.7 ns 186.3 ns
Standard deviation 823.9 ps 1.855 ns 3.991 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

updateIndex5000_100000/Seq100000

31.25 ns
31.50
31.75
32.00
32.25
32.50
32.75
updateIndex5000_100000/Seq100000 time densities
mean
31.25 ns
31.50
31.75
32.00
32.25
32.50
32.75
updateIndex5000_100000/Seq100000 times
lower bound estimate upper bound
Mean execution time 31.36 ns 31.38 ns 31.43 ns
Standard deviation 77.49 ps 149.7 ps 310.9 ps

Outlying measurements have no (1.0%) effect on estimated standard deviation.

updateIndex5000_100000/Vec100000

68 ns
69
70
71
72
73
updateIndex5000_100000/Vec100000 time densities
mean
68 ns
69
70
71
72
73
updateIndex5000_100000/Vec100000 times
lower bound estimate upper bound
Mean execution time 68.34 ns 68.45 ns 68.67 ns
Standard deviation 426.5 ps 773.6 ps 1.268 ns

Outlying measurements have no (1.0%) effect on estimated standard deviation.

build1000/IM1000

123 μs
124
125
126
127
128
129
130
131
build1000/IM1000 time densities
mean
123 μs
124
125
126
127
128
129
130
131
build1000/IM1000 times
lower bound estimate upper bound
Mean execution time 125.9 μs 126.2 μs 126.5 μs
Standard deviation 1.375 μs 1.527 μs 1.764 μs

Outlying measurements have no (1.0%) effect on estimated standard deviation.

build1000/Seq1000

57.0 μs
57.5
58.0
58.5
59.0
59.5
60.0
60.5
61.0
build1000/Seq1000 time densities
mean
57.0 μs
57.5
58.0
58.5
59.0
59.5
60.0
60.5
61.0
build1000/Seq1000 times
lower bound estimate upper bound
Mean execution time 58.82 μs 59.03 μs 59.24 μs
Standard deviation 955.3 ns 1.050 μs 1.163 μs

Outlying measurements have moderate (10.4%) effect on estimated standard deviation.

build1000/Vec1000

36.0 μs
37.0
38.0
39.0
40.0
build1000/Vec1000 time densities
mean
36.0 μs
37.0
38.0
39.0
40.0
build1000/Vec1000 times
lower bound estimate upper bound
Mean execution time 37.80 μs 37.96 μs 38.12 μs
Standard deviation 744.1 ns 834.4 ns 959.8 ns

Outlying measurements have moderate (15.2%) effect on estimated standard deviation.

understanding this report

In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.

Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)

A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.