criterion performance measurements
overview
want to understand this report?
traverse1000 | |
indexAt3_1000 | |
indexAt30_1000 | |
indexAt30_10000 | |
indexAt3000_10000 | |
indexRange501-550_10000 | |
indexRange5001-5500_100000 | |
appendValue1000 | |
appendValue100000 | |
updateIndex5_1000 | |
updateIndex5000_100000 | |
build1000 |
traverse1000/ListR1000
5.50 μs 5.55 5.60 5.65 5.70 5.75 5.80
mean |
5.50 μs 5.55 5.60 5.65 5.70 5.75 5.80
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 5.527 μs | 5.534 μs | 5.547 μs |
Standard deviation | 28.47 ns | 46.97 ns | 72.33 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
traverse1000/IMR1000
33.5 μs 34.0 34.5 35.0
mean |
33.5 μs 34.0 34.5 35.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 34.47 μs | 34.55 μs | 34.61 μs |
Standard deviation | 314.9 ns | 358.5 ns | 411.1 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
traverse1000/SeqR1000
23.25 μs 23.50 23.75 24.00 24.25 24.50
mean |
23.25 μs 23.50 23.75 24.00 24.25 24.50
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 23.74 μs | 23.77 μs | 23.81 μs |
Standard deviation | 129.6 ns | 167.8 ns | 236.7 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
traverse1000/VecR1000
37.5 μs 38.0 38.5 39.0 39.5 40.0 40.5
mean |
37.5 μs 38.0 38.5 39.0 39.5 40.0 40.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 38.16 μs | 38.27 μs | 38.38 μs |
Standard deviation | 512.5 ns | 570.3 ns | 702.6 ns |
Outlying measurements have slight (7.5%) effect on estimated standard deviation.
traverse1000/SeqL1000
23.4 μs 23.6 23.8 24.0 24.2
mean |
23.4 μs 23.6 23.8 24.0 24.2
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 23.48 μs | 23.49 μs | 23.52 μs |
Standard deviation | 36.93 ns | 86.64 ns | 188.1 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
traverse1000/VecL1000
18.3 μs 18.4 18.5 18.6 18.7 18.8 18.9
mean |
18.3 μs 18.4 18.5 18.6 18.7 18.8 18.9
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 18.37 μs | 18.37 μs | 18.39 μs |
Standard deviation | 35.48 ns | 64.37 ns | 131.1 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
traverse1000/ListLS1000
4.30 μs 4.35 4.40 4.45 4.50 4.55 4.60
mean |
4.30 μs 4.35 4.40 4.45 4.50 4.55 4.60
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 4.303 μs | 4.310 μs | 4.322 μs |
Standard deviation | 24.99 ns | 42.67 ns | 69.01 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
traverse1000/SeqLS1000
46.0 μs 46.5 47.0 47.5
mean |
46.0 μs 46.5 47.0 47.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 46.27 μs | 46.30 μs | 46.35 μs |
Standard deviation | 117.0 ns | 182.6 ns | 346.0 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
traverse1000/VecLS1000
17.90 μs 18.00 18.10 18.20 18.30
mean |
17.90 μs 18.00 18.10 18.20 18.30
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 17.99 μs | 18.00 μs | 18.01 μs |
Standard deviation | 42.56 ns | 64.16 ns | 98.00 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt3_1000/List1000
29.50 ns 29.60 29.70 29.80 29.90
mean |
29.50 ns 29.60 29.70 29.80 29.90
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 29.59 ns | 29.60 ns | 29.61 ns |
Standard deviation | 43.38 ps | 54.98 ps | 83.64 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt3_1000/IM1000
49.25 ns 49.50 49.75 50.00 50.25 50.50 50.75
mean |
49.25 ns 49.50 49.75 50.00 50.25 50.50 50.75
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 49.73 ns | 49.81 ns | 49.89 ns |
Standard deviation | 357.5 ps | 396.8 ps | 435.6 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt3_1000/Seq1000
45.00 ns 45.25 45.50 45.75 46.00 46.25
mean |
45.00 ns 45.25 45.50 45.75 46.00 46.25
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 45.04 ns | 45.06 ns | 45.10 ns |
Standard deviation | 81.35 ps | 138.2 ps | 274.1 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt3_1000/Vec1000
28.2 ns 28.3 28.4 28.5 28.6 28.7 28.8 28.9
mean |
28.2 ns 28.3 28.4 28.5 28.6 28.7 28.8 28.9
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 28.26 ns | 28.27 ns | 28.30 ns |
Standard deviation | 44.89 ps | 77.23 ps | 152.6 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt30_1000/List1000
131.0 ns 131.5 132.0 132.5
mean |
131.0 ns 131.5 132.0 132.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 131.4 ns | 131.5 ns | 131.5 ns |
Standard deviation | 222.0 ps | 268.8 ps | 346.1 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt30_1000/IM1000
53.5 ns 54.0 54.5 55.0 55.5 56.0 56.5
mean |
53.5 ns 54.0 54.5 55.0 55.5 56.0 56.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 54.09 ns | 54.17 ns | 54.30 ns |
Standard deviation | 371.2 ps | 517.5 ps | 703.3 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt30_1000/Seq1000
86.0 ns 86.5 87.0 87.5
mean |
86.0 ns 86.5 87.0 87.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 86.21 ns | 86.24 ns | 86.30 ns |
Standard deviation | 142.3 ps | 208.2 ps | 377.0 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt30_1000/Vec1000
28.1 ns 28.2 28.3 28.4 28.5 28.6 28.7
mean |
28.1 ns 28.2 28.3 28.4 28.5 28.6 28.7
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 28.20 ns | 28.21 ns | 28.23 ns |
Standard deviation | 49.29 ps | 70.55 ps | 122.7 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt30_10000/Seq10000
86.0 ns 86.2 86.4 86.6 86.8 87.0
mean |
86.0 ns 86.2 86.4 86.6 86.8 87.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 86.23 ns | 86.25 ns | 86.29 ns |
Standard deviation | 137.4 ps | 160.4 ps | 207.7 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt30_10000/Vec10000
33.0 ns 33.1 33.2 33.3 33.4 33.5 33.6
mean |
33.0 ns 33.1 33.2 33.3 33.4 33.5 33.6
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 33.10 ns | 33.12 ns | 33.14 ns |
Standard deviation | 79.65 ps | 97.52 ps | 137.6 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt3000_10000/IM1000
58 ns 59 60 61 62 63 64 65
mean |
58 ns 59 60 61 62 63 64 65
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 63.72 ns | 63.95 ns | 64.02 ns |
Standard deviation | 201.1 ps | 586.5 ps | 1.317 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexAt3000_10000/Seq10000
198 ns 200 202 204 206
mean |
198 ns 200 202 204 206
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 198.8 ns | 199.3 ns | 199.8 ns |
Standard deviation | 2.168 ns | 2.549 ns | 2.886 ns |
Outlying measurements have slight (5.4%) effect on estimated standard deviation.
indexAt3000_10000/Vec10000
33.1 ns 33.2 33.3 33.4 33.5 33.6 33.7 33.8 33.9
mean |
33.1 ns 33.2 33.3 33.4 33.5 33.6 33.7 33.8 33.9
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 33.20 ns | 33.22 ns | 33.25 ns |
Standard deviation | 72.63 ps | 112.7 ps | 174.5 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexRange501-550_10000/Seq10000
8.45 μs 8.50 8.55 8.60
mean |
8.45 μs 8.50 8.55 8.60
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 8.545 μs | 8.549 μs | 8.552 μs |
Standard deviation | 12.76 ns | 17.97 ns | 30.80 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexRange501-550_10000/Vec10000
1.90 μs 1.95 2.00 2.05
mean |
1.90 μs 1.95 2.00 2.05
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 1.883 μs | 1.885 μs | 1.892 μs |
Standard deviation | 4.515 ns | 16.67 ns | 38.45 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexRange5001-5500_100000/Seq10000
109.5 μs 110.0 110.5 111.0 111.5
mean |
109.5 μs 110.0 110.5 111.0 111.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 109.7 μs | 109.8 μs | 109.8 μs |
Standard deviation | 213.1 ns | 288.1 ns | 471.2 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
indexRange5001-5500_100000/Vec10000
21.8 μs 22.0 22.2 22.4 22.6
mean |
21.8 μs 22.0 22.2 22.4 22.6
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 21.80 μs | 21.81 μs | 21.84 μs |
Standard deviation | 40.86 ns | 97.39 ns | 212.4 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
appendValue1000/IM1000
67 ns 68 69 70 71 72 73
mean |
67 ns 68 69 70 71 72 73
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 67.25 ns | 67.32 ns | 67.55 ns |
Standard deviation | 162.0 ps | 551.5 ps | 1.265 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
appendValue1000/Seq1000
25.0 ns 25.2 25.4 25.6 25.8 26.0
mean |
25.0 ns 25.2 25.4 25.6 25.8 26.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 25.09 ns | 25.12 ns | 25.16 ns |
Standard deviation | 143.1 ps | 178.0 ps | 237.0 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
appendValue1000/Vec1000
22.2 ns 22.4 22.6 22.8 23.0 23.2 23.4
mean |
22.2 ns 22.4 22.6 22.8 23.0 23.2 23.4
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 22.33 ns | 22.35 ns | 22.38 ns |
Standard deviation | 83.16 ps | 130.3 ps | 247.9 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
appendValue100000/IM100000
71.0 ns 71.5 72.0 72.5 73.0 73.5 74.0 74.5
mean |
71.0 ns 71.5 72.0 72.5 73.0 73.5 74.0 74.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 71.67 ns | 71.72 ns | 71.83 ns |
Standard deviation | 177.2 ps | 337.7 ps | 696.6 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
appendValue100000/Seq100000
25.0 ns 25.5 26.0 26.5 27.0 27.5
mean |
25.0 ns 25.5 26.0 26.5 27.0 27.5
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 25.29 ns | 25.34 ns | 25.46 ns |
Standard deviation | 234.2 ps | 385.3 ps | 597.8 ps |
Outlying measurements have slight (8.5%) effect on estimated standard deviation.
appendValue100000/Vec100000
22.65 ns 22.70 22.75 22.80 22.85 22.90 22.95 23.00 23.05
mean |
22.65 ns 22.70 22.75 22.80 22.85 22.90 22.95 23.00 23.05
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 22.75 ns | 22.76 ns | 22.77 ns |
Standard deviation | 43.48 ps | 52.33 ps | 74.64 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
updateIndex5_1000/IM1000
129 ns 130 131 132 133 134 135
mean |
129 ns 130 131 132 133 134 135
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 130.0 ns | 130.0 ns | 130.2 ns |
Standard deviation | 263.6 ps | 568.2 ps | 1.217 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
updateIndex5_1000/Seq1000
30.5 ns 31.0 31.5 32.0 32.5 33.0
mean |
30.5 ns 31.0 31.5 32.0 32.5 33.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 31.25 ns | 31.31 ns | 31.39 ns |
Standard deviation | 269.8 ps | 352.5 ps | 497.0 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
updateIndex5_1000/Vec1000
82 ns 83 84 85 86 87 88
mean |
82 ns 83 84 85 86 87 88
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 82.26 ns | 82.33 ns | 82.55 ns |
Standard deviation | 242.7 ps | 586.8 ps | 1.289 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
updateIndex5000_100000/IM100000
185 ns 190 195 200
mean |
185 ns 190 195 200
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 185.5 ns | 185.7 ns | 186.3 ns |
Standard deviation | 823.9 ps | 1.855 ns | 3.991 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
updateIndex5000_100000/Seq100000
31.25 ns 31.50 31.75 32.00 32.25 32.50 32.75
mean |
31.25 ns 31.50 31.75 32.00 32.25 32.50 32.75
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 31.36 ns | 31.38 ns | 31.43 ns |
Standard deviation | 77.49 ps | 149.7 ps | 310.9 ps |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
updateIndex5000_100000/Vec100000
68 ns 69 70 71 72 73
mean |
68 ns 69 70 71 72 73
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 68.34 ns | 68.45 ns | 68.67 ns |
Standard deviation | 426.5 ps | 773.6 ps | 1.268 ns |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
build1000/IM1000
123 μs 124 125 126 127 128 129 130 131
mean |
123 μs 124 125 126 127 128 129 130 131
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 125.9 μs | 126.2 μs | 126.5 μs |
Standard deviation | 1.375 μs | 1.527 μs | 1.764 μs |
Outlying measurements have no (1.0%) effect on estimated standard deviation.
build1000/Seq1000
57.0 μs 57.5 58.0 58.5 59.0 59.5 60.0 60.5 61.0
mean |
57.0 μs 57.5 58.0 58.5 59.0 59.5 60.0 60.5 61.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 58.82 μs | 59.03 μs | 59.24 μs |
Standard deviation | 955.3 ns | 1.050 μs | 1.163 μs |
Outlying measurements have moderate (10.4%) effect on estimated standard deviation.
build1000/Vec1000
36.0 μs 37.0 38.0 39.0 40.0
mean |
36.0 μs 37.0 38.0 39.0 40.0
|
lower bound | estimate | upper bound | |
---|---|---|---|
Mean execution time | 37.80 μs | 37.96 μs | 38.12 μs |
Standard deviation | 744.1 ns | 834.4 ns | 959.8 ns |
Outlying measurements have moderate (15.2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. In each section, we display two charts, each with an x axis that represents measured execution time. These charts are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. Measurements are displayed on the y axis in the order in which they occurred.
Under the charts is a small table displaying the mean and standard deviation of the measurements. We use a statistical technique called the bootstrap to provide confidence intervals on our estimates of these values. The bootstrap-derived upper and lower bounds on the mean and standard deviation let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.