tree: 7ffa16e509bc706ca54645fb2d9c1e7aa82e3f02 [search]
  1. .gitignore
  2. accumulate_profile.py
  3. arith_eval.ml
  4. binarytrees.ghc-6.hs
  5. binarytrees.lean
  6. binarytrees.lean.args
  7. binarytrees.lean.expected.out
  8. binarytrees.ocaml-2.ml
  9. binarytrees.swift
  10. compile.sh
  11. const_fold.hs
  12. const_fold.lean
  13. const_fold.lean.args
  14. const_fold.lean.expected.out
  15. const_fold.ml
  16. const_fold.sml
  17. const_fold.swift
  18. cross.nix
  19. cross.yaml
  20. deriv.hs
  21. deriv.lean
  22. deriv.lean.args
  23. deriv.lean.expected.out
  24. deriv.ml
  25. deriv.sml
  26. deriv.swift
  27. disable-st.patch
  28. full-stdlib.exec.yaml
  29. ghc-gc.py
  30. lean-gc.py
  31. Makefile
  32. mlkit-gc.py
  33. nixpkgs.nix
  34. ocaml-gc.py
  35. perf.py
  36. qsort.hs
  37. qsort.lean
  38. qsort.lean.args
  39. qsort.lean.expected.out
  40. qsort.ml
  41. qsort.sml
  42. qsort.swift
  43. rbmap.hs
  44. rbmap.lean
  45. rbmap.lean.args
  46. rbmap.lean.expected.out
  47. rbmap.library.lean
  48. rbmap.ml
  49. rbmap.sml
  50. rbmap.swift
  51. rbmap2.lean
  52. rbmap3.lean
  53. rbmap4.lean
  54. rbmap500k.lean
  55. rbmap_checkpoint.hs
  56. rbmap_checkpoint.lean
  57. rbmap_checkpoint.lean.args
  58. rbmap_checkpoint.lean.expected.out
  59. rbmap_checkpoint.ml
  60. rbmap_checkpoint.sml
  61. rbmap_checkpoint.swift
  62. rbmap_checkpoint2.lean
  63. rbmap_checkpoint2.sml
  64. rbmap_checkpoint_cpp_lean3.cpp
  65. rbmap_checkpoint_cpp_std.cpp
  66. rbmap_cpp_lean3.cpp
  67. rbmap_cpp_std.cpp
  68. README.md
  69. report.py
  70. run.sh
  71. speedcenter.exec.velcom.yaml
  72. speedcenter.yaml
  73. test_single.sh
  74. unionfind.lean
  75. unionfind.lean.args
  76. unionfind.lean.expected.out
  77. unionfind_clean.lean
tests/bench/README.md

Lean Benchmark Suites

This folder contains multiple small Lean programs for benchmarking used by two separate benchmark suites based on the temci benchmarking tool:

  • The light-weight “Speedcenter” suite benchmarks the current build of Lean. It can be used for quick comparisons on the cmdline and powers the Lean Speedcenter website.
  • The heavy-weight “Cross” suite benchmarks multiple Lean configurations and other functional compilers against each other and generates CSV and HTML reports from that. It was created for the paper “Counting Immutable Beans - Reference Counting Optimized for Purely Functional Programming” (IFL19).

Speedcenter Suite

Requirements:

  • A local Lean build in ../../build/release. Build at least the bin target.
  • temci. Using Nix, open a nix-shell in the project root directory to add a compatible version to your PATH. Alternatively, try pip3 install git+https://github.com/parttimenerd/temci.git.

To execute the suite and save the results in base.yaml, run (in this folder)

temci exec --config speedcenter.yaml --out base.yaml

Other interesting exec flags:

  • use --runs N to modify the default number of 10 runs per benchmark
  • use --included_blocks fast to excluded slow benchmarks like the stdlib benchmark. You can replace fast with any benchmark name or label in speedcenter.exec.yaml.

If you have multiple saved result files, you can compare them with

temci report --config speedcenter.yaml report1.yaml report2.yaml ...

Cross Suite

We recommend using Nix for building/obtaining all Lean variants and used compilers in a reproducible way. After installing Nix, running the benchmarks is as easy as

nix-shell cross.nix --pure --run make

This will record 50 runs for each benchmark configuration (this can be changed with runs in cross.yaml), generate results in report_lean.csv and report_cross.csv, and print them to stdout in a tabulated format. It will also generate HTML reports in report/ comparing the time-based benchmarks.

In order to reduce noise in the benchmarking data, you may instead want to try calling make inside a temci shell:

nix-shell cross.nix --pure --run "temci short shell --sudo --preset usable --cpuset_active make"

Using root powers, this will temporarily configure your machine similarly to the LLVM benchmarking recommendations and move all your other processes to a single CPU core.