tree: c8a282fa249499daadb0c521bb31c7dd65509492 [search]
  1. inundation/
  2. mergeSort/
  3. qsort/
  4. .gitignore
  5. accumulate_profile.py
  6. arith_eval.ml
  7. big_beq.lean
  8. big_beq_rec.lean
  9. big_deceq.lean
  10. big_deceq_rec.lean
  11. big_do.lean
  12. big_match.lean
  13. big_match_nat.lean
  14. big_match_nat_split.lean
  15. big_match_partial.lean
  16. big_omega.lean
  17. binarytrees.ghc-6.hs
  18. binarytrees.lean
  19. binarytrees.lean.args
  20. binarytrees.lean.expected.out
  21. binarytrees.ocaml-2.ml
  22. binarytrees.st.hs
  23. binarytrees.st.lean
  24. binarytrees.st.mlton-2.sml
  25. binarytrees.st.sml
  26. binarytrees.st.swift
  27. binarytrees.swift
  28. binarytrees5.ml
  29. binarytrees5_multicore.ml
  30. bv_decide_inequality.lean
  31. bv_decide_large_aig.lean
  32. bv_decide_mod.lean
  33. bv_decide_mul.lean
  34. bv_decide_realworld.lean
  35. bv_decide_rewriter.lean
  36. channel.lean
  37. compile.sh
  38. const_fold.hs
  39. const_fold.lean
  40. const_fold.lean.args
  41. const_fold.lean.expected.out
  42. const_fold.ml
  43. const_fold.sml
  44. const_fold.swift
  45. cross.yaml
  46. dag_hassorry_issue.lean
  47. dag_hassorry_issue.lean.args
  48. dag_hassorry_issue.lean.expected.out
  49. deriv.hs
  50. deriv.lean
  51. deriv.lean.args
  52. deriv.lean.expected.out
  53. deriv.ml
  54. deriv.sml
  55. deriv.swift
  56. ex-50-50-1.leq
  57. flake.lock
  58. flake.nix
  59. full-stdlib.exec.yaml
  60. ghc-gc.py
  61. hashmap.lean
  62. identifier_completion.lean
  63. identifier_completion_didOpen.log
  64. identifier_completion_initialization.log
  65. identifier_completion_runner.lean
  66. ilean_roundtrip.lean
  67. iterators.lean
  68. lean-gc.py
  69. liasolver.lean
  70. liasolver.lean.args
  71. liasolver.lean.expected.out
  72. Makefile
  73. mlkit-gc.py
  74. mut_rec_wf.lean
  75. nat_repr.lean
  76. nat_repr.lean.args
  77. nat_repr.lean.expected.out
  78. ocaml-gc.py
  79. omega_stress.lean
  80. parser.lean
  81. perf.py
  82. phashmap.lean
  83. qsort.hs
  84. qsort.lean
  85. qsort.lean.args
  86. qsort.lean.expected.out
  87. qsort.ml
  88. qsort.sml
  89. qsort.swift
  90. rbmap.hs
  91. rbmap.lean
  92. rbmap.lean.args
  93. rbmap.lean.expected.out
  94. rbmap.ml
  95. rbmap.sml
  96. rbmap.swift
  97. rbmap2.lean
  98. rbmap3.lean
  99. rbmap500k.lean
  100. rbmap_checkpoint.hs
  101. rbmap_checkpoint.lean
  102. rbmap_checkpoint.lean.args
  103. rbmap_checkpoint.lean.expected.out
  104. rbmap_checkpoint.ml
  105. rbmap_checkpoint.sml
  106. rbmap_checkpoint.swift
  107. rbmap_checkpoint2.lean
  108. rbmap_checkpoint2.sml
  109. rbmap_checkpoint_cpp_lean3.cpp
  110. rbmap_checkpoint_cpp_std.cpp
  111. rbmap_cpp_lean3.cpp
  112. rbmap_cpp_std.cpp
  113. rbmap_fbip.lean
  114. rbmap_library.lean
  115. README.md
  116. reduceMatch.lean
  117. report.py
  118. riscv-ast.lean
  119. run.sh
  120. server_startup.lean
  121. server_startup.log
  122. sigmaIterator.lean
  123. simp_arith1.lean
  124. simp_bubblesort_256.lean
  125. simp_congr.lean
  126. simp_local.lean
  127. simp_subexpr.lean
  128. speedcenter.exec.velcom.yaml
  129. speedcenter.yaml
  130. states35.lean
  131. test_single.sh
  132. treemap.lean
  133. unionfind.lean
  134. unionfind.lean.args
  135. unionfind.lean.expected.out
  136. unionfind_clean.lean
  137. watchdogRss.lean
  138. workspaceSymbols.lean
  139. workspaceSymbolsNewRanges.lean
tests/bench/README.md

Lean Benchmark Suites

This folder contains multiple small Lean programs for benchmarking used by two separate benchmark suites based on the temci benchmarking tool:

  • The light-weight “Speedcenter” suite benchmarks the current build of Lean. It can be used for quick comparisons on the cmdline and powers the Lean Speedcenter website.
  • The heavy-weight “Cross” suite benchmarks multiple Lean configurations and other functional compilers against each other and generates CSV and HTML reports from that. It was created for the paper “Counting Immutable Beans - Reference Counting Optimized for Purely Functional Programming” (IFL19).

Speedcenter Suite

Requirements:

  • A local Lean build in ../../build/release. Build at least the bin target.
  • temci. Using Nix, open a nix-shell in the project root directory to add a compatible version to your PATH. Alternatively, try pip3 install git+https://github.com/parttimenerd/temci.git.

To execute the suite and save the results in base.yaml, run (in this folder)

temci exec --config speedcenter.yaml --out base.yaml

Other interesting exec flags:

  • use --runs N to modify the default number of 10 runs per benchmark
  • use --included_blocks fast to excluded slow benchmarks like the stdlib benchmark. You can replace fast with any benchmark name or label in speedcenter.exec.yaml.

If you have multiple saved result files, you can compare them with

temci report --config speedcenter.yaml report1.yaml report2.yaml ...

Cross Suite

We recommend using Nix for building/obtaining all Lean variants and used compilers in a reproducible way. After installing Nix, running the benchmarks is as easy as

nix develop
make

This will record 50 runs for each benchmark configuration (this can be changed with runs in cross.yaml), generate results in report_lean.csv and report_cross.csv, and print them to stdout in a tabulated format. It will also generate HTML reports in report/ comparing the time-based benchmarks.

In order to reduce noise in the benchmarking data, you may instead want to try calling make inside a temci shell:

temci short shell --sudo --preset usable --cpuset_active make

Using root powers, this will temporarily configure your machine similarly to the LLVM benchmarking recommendations and move all your other processes to a single CPU core.