tree: 76a34ea9893c749868004318c412200e086dc4b8 [search]
  1. inundation/
  2. mergeSort/
  3. qsort/
  4. .gitignore
  5. accumulate_profile.py
  6. arith_eval.ml
  7. big_beq.lean
  8. big_beq_rec.lean
  9. big_deceq.lean
  10. big_deceq_rec.lean
  11. big_do.lean
  12. big_match.lean
  13. big_match_partial.lean
  14. big_omega.lean
  15. binarytrees.ghc-6.hs
  16. binarytrees.lean
  17. binarytrees.lean.args
  18. binarytrees.lean.expected.out
  19. binarytrees.ocaml-2.ml
  20. binarytrees.st.hs
  21. binarytrees.st.lean
  22. binarytrees.st.mlton-2.sml
  23. binarytrees.st.sml
  24. binarytrees.st.swift
  25. binarytrees.swift
  26. binarytrees5.ml
  27. binarytrees5_multicore.ml
  28. bv_decide_inequality.lean
  29. bv_decide_large_aig.lean
  30. bv_decide_mod.lean
  31. bv_decide_mul.lean
  32. bv_decide_realworld.lean
  33. bv_decide_rewriter.lean
  34. channel.lean
  35. compile.sh
  36. const_fold.hs
  37. const_fold.lean
  38. const_fold.lean.args
  39. const_fold.lean.expected.out
  40. const_fold.ml
  41. const_fold.sml
  42. const_fold.swift
  43. cross.yaml
  44. dag_hassorry_issue.lean
  45. dag_hassorry_issue.lean.args
  46. dag_hassorry_issue.lean.expected.out
  47. deriv.hs
  48. deriv.lean
  49. deriv.lean.args
  50. deriv.lean.expected.out
  51. deriv.ml
  52. deriv.sml
  53. deriv.swift
  54. ex-50-50-1.leq
  55. flake.lock
  56. flake.nix
  57. full-stdlib.exec.yaml
  58. ghc-gc.py
  59. hashmap.lean
  60. identifier_completion.lean
  61. identifier_completion_didOpen.log
  62. identifier_completion_initialization.log
  63. identifier_completion_runner.lean
  64. ilean_roundtrip.lean
  65. iterators.lean
  66. lean-gc.py
  67. liasolver.lean
  68. liasolver.lean.args
  69. liasolver.lean.expected.out
  70. Makefile
  71. mlkit-gc.py
  72. mut_rec_wf.lean
  73. nat_repr.lean
  74. nat_repr.lean.args
  75. nat_repr.lean.expected.out
  76. ocaml-gc.py
  77. omega_stress.lean
  78. parser.lean
  79. perf.py
  80. phashmap.lean
  81. qsort.hs
  82. qsort.lean
  83. qsort.lean.args
  84. qsort.lean.expected.out
  85. qsort.ml
  86. qsort.sml
  87. qsort.swift
  88. rbmap.hs
  89. rbmap.lean
  90. rbmap.lean.args
  91. rbmap.lean.expected.out
  92. rbmap.ml
  93. rbmap.sml
  94. rbmap.swift
  95. rbmap2.lean
  96. rbmap3.lean
  97. rbmap500k.lean
  98. rbmap_checkpoint.hs
  99. rbmap_checkpoint.lean
  100. rbmap_checkpoint.lean.args
  101. rbmap_checkpoint.lean.expected.out
  102. rbmap_checkpoint.ml
  103. rbmap_checkpoint.sml
  104. rbmap_checkpoint.swift
  105. rbmap_checkpoint2.lean
  106. rbmap_checkpoint2.sml
  107. rbmap_checkpoint_cpp_lean3.cpp
  108. rbmap_checkpoint_cpp_std.cpp
  109. rbmap_cpp_lean3.cpp
  110. rbmap_cpp_std.cpp
  111. rbmap_fbip.lean
  112. rbmap_library.lean
  113. README.md
  114. reduceMatch.lean
  115. report.py
  116. riscv-ast.lean
  117. run.sh
  118. server_startup.lean
  119. server_startup.log
  120. sigmaIterator.lean
  121. simp_arith1.lean
  122. simp_bubblesort_256.lean
  123. simp_congr.lean
  124. simp_local.lean
  125. simp_subexpr.lean
  126. speedcenter.exec.velcom.yaml
  127. speedcenter.yaml
  128. states35.lean
  129. test_single.sh
  130. treemap.lean
  131. unionfind.lean
  132. unionfind.lean.args
  133. unionfind.lean.expected.out
  134. unionfind_clean.lean
  135. workspaceSymbols.lean
  136. workspaceSymbolsNewRanges.lean
tests/bench/README.md

Lean Benchmark Suites

This folder contains multiple small Lean programs for benchmarking used by two separate benchmark suites based on the temci benchmarking tool:

  • The light-weight “Speedcenter” suite benchmarks the current build of Lean. It can be used for quick comparisons on the cmdline and powers the Lean Speedcenter website.
  • The heavy-weight “Cross” suite benchmarks multiple Lean configurations and other functional compilers against each other and generates CSV and HTML reports from that. It was created for the paper “Counting Immutable Beans - Reference Counting Optimized for Purely Functional Programming” (IFL19).

Speedcenter Suite

Requirements:

  • A local Lean build in ../../build/release. Build at least the bin target.
  • temci. Using Nix, open a nix-shell in the project root directory to add a compatible version to your PATH. Alternatively, try pip3 install git+https://github.com/parttimenerd/temci.git.

To execute the suite and save the results in base.yaml, run (in this folder)

temci exec --config speedcenter.yaml --out base.yaml

Other interesting exec flags:

  • use --runs N to modify the default number of 10 runs per benchmark
  • use --included_blocks fast to excluded slow benchmarks like the stdlib benchmark. You can replace fast with any benchmark name or label in speedcenter.exec.yaml.

If you have multiple saved result files, you can compare them with

temci report --config speedcenter.yaml report1.yaml report2.yaml ...

Cross Suite

We recommend using Nix for building/obtaining all Lean variants and used compilers in a reproducible way. After installing Nix, running the benchmarks is as easy as

nix develop
make

This will record 50 runs for each benchmark configuration (this can be changed with runs in cross.yaml), generate results in report_lean.csv and report_cross.csv, and print them to stdout in a tabulated format. It will also generate HTML reports in report/ comparing the time-based benchmarks.

In order to reduce noise in the benchmarking data, you may instead want to try calling make inside a temci shell:

temci short shell --sudo --preset usable --cpuset_active make

Using root powers, this will temporarily configure your machine similarly to the LLVM benchmarking recommendations and move all your other processes to a single CPU core.