criterion performance measurements
overview
want to understand this report?
head/bad/noloc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.232328835871741e-8 | 3.26348191176203e-8 | 3.3056343648438915e-8 |
| Standard deviation | 9.851232637781709e-10 | 1.2046151317545238e-9 | 1.6738002778701571e-9 |
Outlying measurements have severe (0.586661655258986%) effect on estimated standard deviation.
head/bad/loc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.028625997066534e-8 | 8.089824238170393e-8 | 8.175212287066612e-8 |
| Standard deviation | 1.8013163906492962e-9 | 2.4358528351694863e-9 | 3.203719034162452e-9 |
Outlying measurements have moderate (0.4676092782872557%) effect on estimated standard deviation.
head/good/noloc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.756338159650434e-9 | 8.823872946075667e-9 | 8.93904621420681e-9 |
| Standard deviation | 1.9606351986036923e-10 | 2.8920588398128427e-10 | 4.881398246287841e-10 |
Outlying measurements have severe (0.5498630742794954%) effect on estimated standard deviation.
head/good/loc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.1945467992104107e-8 | 1.2029662877043155e-8 | 1.2139004864385782e-8 |
| Standard deviation | 2.499738669611164e-10 | 3.1700033486547307e-10 | 4.220823391230783e-10 |
Outlying measurements have moderate (0.43522112509760574%) effect on estimated standard deviation.
loop/bad/noloc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.2576845645516664e-8 | 3.278926887621527e-8 | 3.315706255158604e-8 |
| Standard deviation | 6.903481780642184e-10 | 9.649373920866892e-10 | 1.5402348731956818e-9 |
Outlying measurements have moderate (0.4722826264689864%) effect on estimated standard deviation.
loop/bad/loc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0594265114282946e-7 | 1.0693468711711164e-7 | 1.0832143559281681e-7 |
| Standard deviation | 2.811974805114114e-9 | 3.815626083775801e-9 | 5.220951256571829e-9 |
Outlying measurements have severe (0.5453195721223985%) effect on estimated standard deviation.
loop/good/noloc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0911581374270278e-8 | 1.104938984277707e-8 | 1.1256969959711376e-8 |
| Standard deviation | 4.162130038809786e-10 | 5.703698987181945e-10 | 8.208984493978659e-10 |
Outlying measurements have severe (0.7519237854876427%) effect on estimated standard deviation.
loop/good/loc
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.891941745379741e-8 | 3.931056346971441e-8 | 3.9724594818681977e-8 |
| Standard deviation | 1.0559014317884832e-9 | 1.3516622466682546e-9 | 1.7441358009852193e-9 |
Outlying measurements have severe (0.5495858306238336%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.