tree: 731561ae133fa715a5a1b8e511dece2456dd388c [search]
  1. inundation/
  2. mergeSort/
  3. qsort/
  4. .gitignore
  5. accumulate_profile.py
  6. arith_eval.ml
  7. big_beq.lean
  8. big_beq_rec.lean
  9. big_deceq.lean
  10. big_deceq_rec.lean
  11. big_do.lean
  12. big_match.lean
  13. big_omega.lean
  14. binarytrees.ghc-6.hs
  15. binarytrees.lean
  16. binarytrees.lean.args
  17. binarytrees.lean.expected.out
  18. binarytrees.ocaml-2.ml
  19. binarytrees.st.hs
  20. binarytrees.st.lean
  21. binarytrees.st.mlton-2.sml
  22. binarytrees.st.sml
  23. binarytrees.st.swift
  24. binarytrees.swift
  25. binarytrees5.ml
  26. binarytrees5_multicore.ml
  27. bv_decide_inequality.lean
  28. bv_decide_large_aig.lean
  29. bv_decide_mod.lean
  30. bv_decide_mul.lean
  31. bv_decide_realworld.lean
  32. bv_decide_rewriter.lean
  33. channel.lean
  34. compile.sh
  35. const_fold.hs
  36. const_fold.lean
  37. const_fold.lean.args
  38. const_fold.lean.expected.out
  39. const_fold.ml
  40. const_fold.sml
  41. const_fold.swift
  42. cross.yaml
  43. dag_hassorry_issue.lean
  44. dag_hassorry_issue.lean.args
  45. dag_hassorry_issue.lean.expected.out
  46. deriv.hs
  47. deriv.lean
  48. deriv.lean.args
  49. deriv.lean.expected.out
  50. deriv.ml
  51. deriv.sml
  52. deriv.swift
  53. ex-50-50-1.leq
  54. flake.lock
  55. flake.nix
  56. full-stdlib.exec.yaml
  57. ghc-gc.py
  58. hashmap.lean
  59. identifier_completion.lean
  60. identifier_completion_didOpen.log
  61. identifier_completion_initialization.log
  62. identifier_completion_runner.lean
  63. ilean_roundtrip.lean
  64. iterators.lean
  65. lean-gc.py
  66. liasolver.lean
  67. liasolver.lean.args
  68. liasolver.lean.expected.out
  69. Makefile
  70. mlkit-gc.py
  71. mut_rec_wf.lean
  72. nat_repr.lean
  73. nat_repr.lean.args
  74. nat_repr.lean.expected.out
  75. ocaml-gc.py
  76. omega_stress.lean
  77. parser.lean
  78. perf.py
  79. phashmap.lean
  80. qsort.hs
  81. qsort.lean
  82. qsort.lean.args
  83. qsort.lean.expected.out
  84. qsort.ml
  85. qsort.sml
  86. qsort.swift
  87. rbmap.hs
  88. rbmap.lean
  89. rbmap.lean.args
  90. rbmap.lean.expected.out
  91. rbmap.ml
  92. rbmap.sml
  93. rbmap.swift
  94. rbmap2.lean
  95. rbmap3.lean
  96. rbmap500k.lean
  97. rbmap_checkpoint.hs
  98. rbmap_checkpoint.lean
  99. rbmap_checkpoint.lean.args
  100. rbmap_checkpoint.lean.expected.out
  101. rbmap_checkpoint.ml
  102. rbmap_checkpoint.sml
  103. rbmap_checkpoint.swift
  104. rbmap_checkpoint2.lean
  105. rbmap_checkpoint2.sml
  106. rbmap_checkpoint_cpp_lean3.cpp
  107. rbmap_checkpoint_cpp_std.cpp
  108. rbmap_cpp_lean3.cpp
  109. rbmap_cpp_std.cpp
  110. rbmap_fbip.lean
  111. rbmap_library.lean
  112. README.md
  113. reduceMatch.lean
  114. report.py
  115. riscv-ast.lean
  116. run.sh
  117. server_startup.lean
  118. server_startup.log
  119. simp_arith1.lean
  120. simp_bubblesort_256.lean
  121. simp_congr.lean
  122. simp_local.lean
  123. simp_subexpr.lean
  124. speedcenter.exec.velcom.yaml
  125. speedcenter.yaml
  126. states35.lean
  127. test_single.sh
  128. treemap.lean
  129. unionfind.lean
  130. unionfind.lean.args
  131. unionfind.lean.expected.out
  132. unionfind_clean.lean
  133. workspaceSymbols.lean
  134. workspaceSymbolsNewRanges.lean
tests/bench/README.md

Lean Benchmark Suites

This folder contains multiple small Lean programs for benchmarking used by two separate benchmark suites based on the temci benchmarking tool:

  • The light-weight “Speedcenter” suite benchmarks the current build of Lean. It can be used for quick comparisons on the cmdline and powers the Lean Speedcenter website.
  • The heavy-weight “Cross” suite benchmarks multiple Lean configurations and other functional compilers against each other and generates CSV and HTML reports from that. It was created for the paper “Counting Immutable Beans - Reference Counting Optimized for Purely Functional Programming” (IFL19).

Speedcenter Suite

Requirements:

  • A local Lean build in ../../build/release. Build at least the bin target.
  • temci. Using Nix, open a nix-shell in the project root directory to add a compatible version to your PATH. Alternatively, try pip3 install git+https://github.com/parttimenerd/temci.git.

To execute the suite and save the results in base.yaml, run (in this folder)

temci exec --config speedcenter.yaml --out base.yaml

Other interesting exec flags:

  • use --runs N to modify the default number of 10 runs per benchmark
  • use --included_blocks fast to excluded slow benchmarks like the stdlib benchmark. You can replace fast with any benchmark name or label in speedcenter.exec.yaml.

If you have multiple saved result files, you can compare them with

temci report --config speedcenter.yaml report1.yaml report2.yaml ...

Cross Suite

We recommend using Nix for building/obtaining all Lean variants and used compilers in a reproducible way. After installing Nix, running the benchmarks is as easy as

nix develop
make

This will record 50 runs for each benchmark configuration (this can be changed with runs in cross.yaml), generate results in report_lean.csv and report_cross.csv, and print them to stdout in a tabulated format. It will also generate HTML reports in report/ comparing the time-based benchmarks.

In order to reduce noise in the benchmarking data, you may instead want to try calling make inside a temci shell:

temci short shell --sudo --preset usable --cpuset_active make

Using root powers, this will temporarily configure your machine similarly to the LLVM benchmarking recommendations and move all your other processes to a single CPU core.