tree: d6366734cbaf862a3707ffb3dca6d383e9d81436 [search]
  1. inundation/
  2. mergeSort/
  3. qsort/
  4. .gitignore
  5. accumulate_profile.py
  6. arith_eval.ml
  7. big_do.lean
  8. big_omega.lean
  9. binarytrees.ghc-6.hs
  10. binarytrees.lean
  11. binarytrees.lean.args
  12. binarytrees.lean.expected.out
  13. binarytrees.ocaml-2.ml
  14. binarytrees.st.hs
  15. binarytrees.st.lean
  16. binarytrees.st.mlton-2.sml
  17. binarytrees.st.sml
  18. binarytrees.st.swift
  19. binarytrees.swift
  20. binarytrees5.ml
  21. binarytrees5_multicore.ml
  22. bv_decide_inequality.lean
  23. bv_decide_mod.lean
  24. bv_decide_mul.lean
  25. bv_decide_realworld.lean
  26. compile.sh
  27. const_fold.hs
  28. const_fold.lean
  29. const_fold.lean.args
  30. const_fold.lean.expected.out
  31. const_fold.ml
  32. const_fold.sml
  33. const_fold.swift
  34. cross.yaml
  35. dag_hassorry_issue.lean
  36. dag_hassorry_issue.lean.args
  37. dag_hassorry_issue.lean.expected.out
  38. deriv.hs
  39. deriv.lean
  40. deriv.lean.args
  41. deriv.lean.expected.out
  42. deriv.ml
  43. deriv.sml
  44. deriv.swift
  45. ex-50-50-1.leq
  46. flake.lock
  47. flake.nix
  48. full-stdlib.exec.yaml
  49. ghc-gc.py
  50. identifier_completion.lean
  51. identifier_completion_didOpen.log
  52. identifier_completion_initialization.log
  53. identifier_completion_runner.lean
  54. ilean_roundtrip.lean
  55. lean-gc.py
  56. liasolver.lean
  57. liasolver.lean.args
  58. liasolver.lean.expected.out
  59. Makefile
  60. mlkit-gc.py
  61. nat_repr.lean
  62. nat_repr.lean.args
  63. nat_repr.lean.expected.out
  64. ocaml-gc.py
  65. omega_stress.lean
  66. parser.lean
  67. perf.py
  68. qsort.hs
  69. qsort.lean
  70. qsort.lean.args
  71. qsort.lean.expected.out
  72. qsort.ml
  73. qsort.sml
  74. qsort.swift
  75. rbmap.hs
  76. rbmap.lean
  77. rbmap.lean.args
  78. rbmap.lean.expected.out
  79. rbmap.ml
  80. rbmap.sml
  81. rbmap.swift
  82. rbmap2.lean
  83. rbmap3.lean
  84. rbmap500k.lean
  85. rbmap_checkpoint.hs
  86. rbmap_checkpoint.lean
  87. rbmap_checkpoint.lean.args
  88. rbmap_checkpoint.lean.expected.out
  89. rbmap_checkpoint.ml
  90. rbmap_checkpoint.sml
  91. rbmap_checkpoint.swift
  92. rbmap_checkpoint2.lean
  93. rbmap_checkpoint2.sml
  94. rbmap_checkpoint_cpp_lean3.cpp
  95. rbmap_checkpoint_cpp_std.cpp
  96. rbmap_cpp_lean3.cpp
  97. rbmap_cpp_std.cpp
  98. rbmap_fbip.lean
  99. rbmap_library.lean
  100. README.md
  101. reduceMatch.lean
  102. report.py
  103. run.sh
  104. server_startup.lean
  105. server_startup.log
  106. simp_arith1.lean
  107. speedcenter.exec.velcom.yaml
  108. speedcenter.yaml
  109. states35.lean
  110. test_single.sh
  111. unionfind.lean
  112. unionfind.lean.args
  113. unionfind.lean.expected.out
  114. unionfind_clean.lean
  115. workspaceSymbols.lean
tests/bench/README.md

Lean Benchmark Suites

This folder contains multiple small Lean programs for benchmarking used by two separate benchmark suites based on the temci benchmarking tool:

  • The light-weight “Speedcenter” suite benchmarks the current build of Lean. It can be used for quick comparisons on the cmdline and powers the Lean Speedcenter website.
  • The heavy-weight “Cross” suite benchmarks multiple Lean configurations and other functional compilers against each other and generates CSV and HTML reports from that. It was created for the paper “Counting Immutable Beans - Reference Counting Optimized for Purely Functional Programming” (IFL19).

Speedcenter Suite

Requirements:

  • A local Lean build in ../../build/release. Build at least the bin target.
  • temci. Using Nix, open a nix-shell in the project root directory to add a compatible version to your PATH. Alternatively, try pip3 install git+https://github.com/parttimenerd/temci.git.

To execute the suite and save the results in base.yaml, run (in this folder)

temci exec --config speedcenter.yaml --out base.yaml

Other interesting exec flags:

  • use --runs N to modify the default number of 10 runs per benchmark
  • use --included_blocks fast to excluded slow benchmarks like the stdlib benchmark. You can replace fast with any benchmark name or label in speedcenter.exec.yaml.

If you have multiple saved result files, you can compare them with

temci report --config speedcenter.yaml report1.yaml report2.yaml ...

Cross Suite

We recommend using Nix for building/obtaining all Lean variants and used compilers in a reproducible way. After installing Nix, running the benchmarks is as easy as

nix develop
make

This will record 50 runs for each benchmark configuration (this can be changed with runs in cross.yaml), generate results in report_lean.csv and report_cross.csv, and print them to stdout in a tabulated format. It will also generate HTML reports in report/ comparing the time-based benchmarks.

In order to reduce noise in the benchmarking data, you may instead want to try calling make inside a temci shell:

temci short shell --sudo --preset usable --cpuset_active make

Using root powers, this will temporarily configure your machine similarly to the LLVM benchmarking recommendations and move all your other processes to a single CPU core.