//
// See README-LCALS_license.txt for access and distribution restrictions
//
================================================================================
================================================================================
LCALS: Livermore Compiler Analysis Loop Suite
 by Rich Hornung (hornung1@llnl.gov), 
    Center for Applied Scientific Computing,
    Lawrence Livermore National Laboratory 
================================================================================
================================================================================

 o This code is under continuing development.  Go to http://codesign.llnl.gov
   to acquire the latest released version.

 o This loop suite is designed to measure performance for a variety of loops
   using different compilers and platforms.  In particular, the suite 
   helps to understand compiler optimization, run-time performance issues,
   and platform capabilities.  The suite is also useful as a source of
   example code snippets for interactions with compiler developers.

 o The loops in the suite are partitioned into three subsets based on their 
   origins (and also to avoid having them all in a single source file).  Each 
   loop is implemented using multiple software constructs (i.e., referred 
   to herein as "variants").  The three loop subsets are:

   -  Subset A: Loops representative of those found in application codes.
      They are implemented in source files named runA<variant>Loops.cxx.  

   -  Subset B: Basic loops that help to illustrate compiler optimization 
      issues. They are implemented in source files named runB<variant>Loops.cxx

   -  Subset C: Loops extracted from "Livermore Loops coded in C" developed by 
      Steve Langer, which were derived from the Fortran version by Frank 
      McMahon.  They are implemented in source files runC<variant>Loops.cxx 

   Please see the contents of the loop source files to understand the 
   differences among the variants.

 o New loops may be added to the suite by inserting them into appropriate 
   loop source files and modifying a few other files that control suite 
   execution and parametrization.  Details are provided below.

 o Various parameters can be adjusted to control how loops are defined and run.
 
   -- Each loop may be run with different loop lengths (currently up to three
      lengths for each loop) and will be sampled some number of times to 
      generate execution timing data.  Loop length and sampling parameters 
      may be modified to evaluate different platform performance 
      characteristics.  Details are provided below.

 o Various run time statistics can be generated for analysis.  Currently,
   these include: min run time, max run time, average run time, 
   standard deviation across run times, and average execution time relative 
   to a reference loop variant.  Here, run time is the time required to 
   execute the loop for one "sampling" pass through the suite.  See below.


--------------------------------------------------------------------------------
Loop kernels and variants:

 o Each loop in the suite is defined by its traditional C/C++ for-loop 
   "kernel".  Then, each loop appears in multiple variants that use different
   programming and execution constructs.

 o Loops that emply traditional C/C++ for-loop syntax are referred to as 
   "Raw" variants.  The "Raw" variant of each loop represents the version 
   obtained from its original source, plus minor modifications necessary 
   to plug into the loop suite framework.  For example, the loops in the 
   runCRawLoops.cxx file are essentially verbatim from the Livermore Loops 
   Coded in C" suite mentioned above.  Typically, the "Raw" loops serve as 
   reference implemenation for runtime comparisons.  

 o Other variants use loop traversal C++ template methods and represent the 
   loop body as a lambda function or functor class.  One of the main goals 
   of the suite is to assess how SIMD vectorization, OpenMP multithreading, 
   etc. work with these different loop implementation choices.

   Note that only a subset of the loops in the suite appear in the OpenMP
   variants since many of the loops do not benefit from thread parallelism
   due to OpenMP overheads.  OpenMP loops are implmented in source files
   named runOMP<variant>Loops.cxx; in particular, they are not broken out
   into separate source files based on the subsets described above.

 o Although all loop bodies contain only C-syntax, the loop framework 
   uses C++ classes and templates. So a C++ compiler is required to compile 
   the code.  All C++ compilers should be able to compile the framework
   code and "Raw" loop variants.  

 o Not all compilers implement the OpenMP standard. Thus, those loop variants
   may not be compiled and run depending on the compiler being used.

 o The intent of the C++ lambda and functor loop variants is to evaluate 
   compilers in the context of C++ abstraction layers using template methods. 
   Not all compilers support standard C++ lambda expressions at this time.  
   Thus, the lambda variants of the loops may not be compiled and run 
   depending on the compiler being used.


******************** Test Suite Note ***********************
*                                                          *
*     Below is the original build instructions, the        *
*     test suite replaces this build system with the       *
*     llvm test-suite CMake system.  The control of        *
*     loop suite and timing has been altered to use        *
*     the google benchmark library included in the         *
*     MicroBenchmarks directory of the llvm test-suite.    *
*                                                          *
************************************************************
--------------------------------------------------------------------------------
Compiling and running the loop suite:

 The loop suite is typically compiled by typing 'make' and then executed as

    ./lcals.exe <optional output directory>

 o The executable generated by the Makefile accepts an optional argument
   which is the name of a directory for placing output files that contain 
   detailed timing, checksum, and FOM (when specified) results.  Some of 
   these files provide a summary of loop suite performance.  Othere
   contain subsets of this information in comma-delimited text files that may
   be imported into Microsoft Excel to generate spreadsheets and plots.  
   When no output directory is given, a summary of the results is printed 
   to standard output.  

 o LCALS is highly parametrized to explore many compilation and execution
   options. Exercising the full range of options can be achieved by making
   straightforward modifications in a few files, as describe below:

   -- Makefile: This file contains a simple build system for the code. 
                It has a variety of configurations for current LLNL 
                computing systems. Building for other platforms or changing
                any compiler options can done by modifying this file.

   -- LCALS_rules.mk: This file contains "-D" compilation options that 
                conrol some aspects of LCALS parametrization. The effect of 
                these options is described in the comments in this file.
                It is also helpful to see how they are used in the 
                LCALSParams.hxx file.

   -- main.cxx: The main program determines many of the LCALS execution
                options, such as which loops are run (kernels and variants).

   -- LCALSSuite.cxx: The routine defineLoopSuiteRunInfo() in this file 
                defines loop lengths and sampling parameters for each loop
                in the suite.  It also defines loop weights used in Figure
                of Merit (FOM) calculations. 

   -- LCALSSuite.hxx: This file contains '#define' preprocessor directives
                that can be used to turn on/off compilation of individual
                loop kernels and loop variants in the suite. This can be 
                helpful for generating assembly code in small doses.

 o Details on many of these items are given in the next section.
                

--------------------------------------------------------------------------------
Controlling loop suite execution and timing output:

 o The execution of the loop suite follows the pattern described here:

   Iterate over specified number of passes through the loop suite {

      Iterate over specified loop variants to run {

         Iterate over loop lengths to run (e.g., long, medium, short) {

            Iterate over each loop specified to run {

               TIMER_START()
               Iterate over specified number of samples (for loop and length) {

                  Execute loop variant and length.
 
               }
               TIMER_STOP()

            }  // end iteration over loops to run

         } // end iteration over loop lengths

      } // end iteration over loop variants

   } // end iteration over suite passes

 o The loop suite is parametrized so that its execution may be controlled 
   by editing various items in a small number of source and header files
   as described below:

   -- Set number of passes through the suite by setting the variable
      'num_suite_passes' in main.cxx.

   -- Set loop variants to run by adding the corresponding enumeration
      constants to the vector 'run_variants' in main.cxx.  To prevent a 
      variant from running, simply comment out the line which adds the 
      corresponding enum value to the vector.

      NOTE: The first entry added this array indicates the reference variant
            for relative execution time statistics.

      NOTE: An additional argument may be given to the exectuable to run
            loops outside of the standard LCASL benchmark.  This requires 
            that "BUILD_MISC" is defined in the Makefile. 

   -- Set which loop lengths to run by setting the appropriate entry in
      the array 'run_loop_length' in main.cxx (true/false for each length).

   -- Set which loop kernels will run be setting entries in the array 
      'run_loop' in main.cxx (true/false for each loop).

   -- The lengths and number of samples per pass for each loop are set 
      in the routine defineLoopSuiteRunInfo() in LCALSSuite.cxx.

      NOTE: The "samples per pass" values for each loop were determined 
      manually to give approximately 1 second of execution time for its
      serial raw variant on an Intel ES-2670 node. To reduce or increase the 
      total suite execution time, or change the loop lengths used, change 
      the 'sample_frac' and/or 'loop_length_factor' variables in 
      main.cxx.  All default loop lengths will be multiplied by the 
      loop_length_factor value.  The sample count for each loop will be 
      multiplied by sample_frac/loop_length_factor.

   -- The "LoopKernelID" and "LoopLength" enumeration types in the file 
      LCALSSuite.hxx are used to identify loops and loop lengths
      in the suite.  Macros are also provided in that file to conditionally
      compile each loop in the suite.

      The way in which the loops are compiled can influence execution times.
      For example, some compilers perform optimizations for loops compiled
      individually that they do not perform when the same loop is compiled as
      part of a larger suite.

 o All loop forms use the same data arrays, which are pre-allocated based 
   on the loop lengths.  To help with SIMD vectorization and ensure corretness
   data arrays are allocated to be aligned width SIMD vector unit boundaries. 
   This can be changed by setting the 'LCALS_DATA_ALIGN' constant in the
   file LCALSParams.hxx.

 o To minimize the effects of execution of each loop on the others, 
   data caches are flushed before each loop is run. 

   -- Data cache size is set for some LLNL platforms based on hostname.
      If unknown, a warning message will appear when loop suite is run.
      Please edit main.cxx to set the largest data cache size for other 
      platforms.

 o A simple checksum mechanism is provided to verify that different variants 
   of each loop, and implementation changes made to individual loops, generate
   the same numerical results.  "-D" compiler options are provided in the
   LCALS_rules.mk file to control this behavior.  Note that certain levels 
   and types of compiler optimization will cause slight differences in 
   checksums due to changes in operation order, for example.  Thus, the 
   checksums may only be a qualitative indicator of correct execution.

   -- Note that the routines loopInit() and loopFinalize() in LCALSSuite.cxx
      initialize data and compute result checksums for each loop.  These
      must remain consistent with the data used in each loop for correctness.


 o There are two mechanisms available to generate execution timing data for
   loops in the suite.  The choice is made by defining/undefining the 
   associated "-D" option in the LCALS_rules.mk file.  See that file for 
   more information.


--------------------------------------------------------------------------------
Figures of Merit:

 o The program output includes a Figure of Merit (FOM) value for each loop 
   variant and loop length that is executed.  The intent of the FOM is to 
   complement execution timing data with another measure of performance and 
   compiler optimization.  Using the FOM values and total loop suite execution 
   time information in the Figure of Merit report, one can compare different 
   compilers' abilities to optimize on a given platform, performance of 
   different optimization levels for a given compiler, or potential performance 
   of different architectures, etc. 

 o In the FOM calculation, execution time for each loop is weighted by a 
   factor defined in the loop setup routines.  The loops are partitioned into 
   six classes depending on their structure; e.g., data-parallel, order-
   dependent, etc.  The weight for each loop class indicates its relative 
   importance based on code constructs we want the suite to emphasize 
   and how easy we think it should be for a compiler to optimize.  Each loop 
   in the suite is given a weight, w_i (i is the loop id), based on which 
   class it exists in.  Loop classes and weights are defined in the file 
   LCALSSuite.cxx.

 o The FOM is calculated as follows.

   - Relative FOM (FOM_rel).  The aim of the FOM_rel value is to measure 
     a compiler's ability to optimize different loop constructs.

     -- When the code is executed, a reference loop execution time, t_ref, is
        computed using a loop that any compiler should be able to optimize 
        well and which should run faster than any loop in the suite.  
        To help insure this, two simple loops are run, an element-wise vector 
        product and a vector dot product. Then, t_ref is the minimum execution 
        time between the two.

     -- After the suite is run, FOM_rel is calulated as:

        FOM_rel = W * t_ref / Sum_i [ w_i * t_i ]

        The denominator is a weighted sum of execution times for the loops
        that were run; t_i is the run time for loop i.  W = Sum_i ( w_i ) is
        the sum of loop weights.

     -- Note that FOM_rel is a dimensionless quantity that satisfies 
        0 <= FOM_rel <= 1, and FOM_rel increases as loop execution times 
        decrease.  In the ideal case, where each loop executes as fast as the 
        reference loop (which should be impossible), t_i = t_ref for each i.
        So FOM_rel = 1.