| |
| SPhot - Monte Carlo Transport Code |
| |
| |
| Benchmark-specific Instructions and Constraints |
| |
| *Privacy & Legal Notice* <http://www.llnl.gov/disclaimer.html> |
| ------------------------------------------------------------------------ |
| |
| |
| Contents |
| |
| * Expected Results |
| * Optimization and Improvement Challenges |
| * Parallelism and Scalability Expectations |
| * Benchmark Specific Instructions |
| * Release and Modification Record |
| |
| ------------------------------------------------------------------------ |
| |
| |
| Expected Results |
| |
| Please see the Summary writeup and the Sphot Sequoia Specific Benchmark |
| tests for information about expected results and the reporting of data. |
| |
| |
| ------------------------------------------------------------------------ |
| |
| |
| Optimization and Improvement Challenges |
| |
| Performance improvement and optimizations for this code fall into the |
| following categories: |
| |
| 1. *MPI Related* |
| |
| Very little data is actually transferred between MPI tasks, since |
| each task has its own copy of the mesh, most likely in memory or |
| even cache. Most of the inter-task data transfer is for the |
| purpose of collecting timing results and involves very little |
| data. Several MPI_Barrier calls are used for synchronization |
| purposes also. |
| |
| Improving MPI related performance would involve minimizing or |
| removing all MPI_Barrier calls and finding a way to minimize the |
| exchange of inter-task timing statistics. In regards to the |
| exchange of timing statistics, task 0 is a definitive bottleneck |
| since it must act as the "master" task and communicate with all |
| other tasks. This bottleneck may pose a challenge as the problem |
| is scaled up to hundreds or thousands of MPI tasks. |
| |
| A very important, platform dependent optimization consideration |
| for this code is determining the optimal number of MPI tasks to |
| use. For example, a cluster of 32-processor SMP machines may |
| demonstrate optimal performance when there are 4 MPI tasks |
| (running 8 OpenMP threads each) on each SMP machine. This |
| optimization will probably best be determined by experimentation |
| on a given platform. |
| |
| 2. *OpenMP Related* |
| |
| The current implementation does not make use of THREADPRIVATE |
| common blocks. Instead, all threads access certain COMMON block |
| variables globally, and that access occurs within the |
| computational core routine (execute.f) of the code. For SMPs with |
| few processors, this does not seem to pose a performance problem. |
| However, it is anticipated that for many-processor SMP machines, |
| it may. Improving OpenMP performance would involve finding a means |
| to implement THREADPRIVATE common blocks. Currently, the code |
| produces "wrong" results if this is attempted. |
| |
| As with MPI, determining the optimal number of OpenMP threads to |
| use is a very important platform dependent optimization |
| consideration. |
| |
| 3. *Computation Kernel Related* |
| |
| Actual execution time for a "run" of SPhot requires very little |
| wallclock or CPU time--certainly much less than a minute for any |
| current platform. Also, due to the small problem size (grid size), |
| and the embarrassingly parallel nature of this code, performance |
| gains in this category would be minimal at best. |
| |
| For those interested in improving the performance of the |
| computationally significant routines, the following might be |
| evaluated: |
| |
| * execute.f |
| |
| This routine is responsible for over 50% of the total |
| execution time. Profiling has shown that most of the cycles |
| are distributed over a fairly large number of lines that |
| perform simple arithmetic operations (multiplies, adds, |
| divides), variable assignments (load/store), and IF |
| condition testing. |
| |
| * pranf.f |
| |
| This routine comprises approximately 10% of the total |
| execution time. It is a very small routine where virtually |
| all of the execution time can be accounted for by a single |
| line of code, shown below: |
| |
| |
| RandNum = float( Seed( 4 ) ) / Divisor( 4 ) + |
| |
| 1 float( Seed( 3 ) ) / Divisor( 3 ) + |
| |
| 2 float( Seed( 2 ) ) / Divisor( 2 ) + |
| |
| 3 float( Seed( 1 ) ) / Divisor( 1 ) |
| |
| |
| |
| * ranfmodmult.f |
| |
| This routine comprises approximately 10% of the total |
| execution time. This is another rather trivial (in size) |
| routine. The execution time is more or less evenly |
| distributed over all lines of executable code (less than 20) |
| which mostly perform simple arithmetic and load/store |
| operations. |
| |
| * sqrt intrinsic function |
| |
| This will vary according to platform. On one platform tested |
| using the native Fortran compiler, approximately 10% of the |
| total execution time was attributed to this intrinsic function. |
| |
| ------------------------------------------------------------------------ |
| |
| |
| Parallelism and Scalability Expectations |
| |
| Because SPhot is trivially parallel, it might be expected that this code |
| should scale "perfectly" to very large processor counts. Additionally, |
| I/O operations have been replaced with MPI communications to eliminate |
| scalability problems common to many I/O systems. |
| |
| |
| When SPhot is executed in hybrid mode (MPI with OpenMP), both shared |
| memory and distributed memory communications hardware and software |
| performance will be demonstrated. CPU time per "run" will remain |
| relatively constant since runs are intended to map to a processor's |
| CPUs. Increasing the number of "runs" which may be done in parallel |
| increases with the number of available CPUs. Perfect scalability is |
| evidenced by an efficiency calculation of 1.00. That is, there is no |
| additional time required to compute N runs on N CPUs. |
| |
| Realistically, the scalability of SPhot may be affected by the MPI and |
| OpenMP related factors discussed under the Optimization and Improvement |
| Challenges Optimization> section. As indicated |
| there, the application's MPI communications include barriers and |
| potential task 0 bottlenecks. Also, the OpenMP sharing of global COMMON |
| block data within the computation loop may also degrade performance for |
| many-processor SMP machines. |
| |
| ------------------------------------------------------------------------ |
| |
| |
| Benchmark Specific Instructions and Constraints |
| |
| 1. *Code Modifications* |
| |
| For the purposes of the ASC benchmarks, modifications to this |
| application's source code are not permitted unless such |
| modifications are: |
| * Needed to modify certain default parameters as discussed in |
| the Building the Code section. |
| * Minor and not intended for optimization purposes. Permitted |
| modifications would include those required to overcome |
| platform specific obstacles to building or running the code. |
| These modifications should be documented and reported to the |
| ASCI benchmark point of contact. |
| |
| 2. *Modifications to Input File Parameters* |
| |
| As discussed in the Running the Code |
| section previously, the *Nruns* parameter will almost invariably |
| need to be modified every time the number of MPI tasks / OpenMP |
| threads used in an execution changes. This input parameter and the |
| output print flag are the only two parameters which can be modified. |
| |
| Note that this parameter must be at least equal to and evenly |
| divisible by the (Number of MPI tasks * Number of OpenMP threads) |
| used. It has a default maximum value of 10001. If this default |
| value is too small, consult the Building the Code |
| section for instructions on how to |
| increase the value of the Nruns parameter. |
| |
| 3. *Compiler Generated Optimizations* |
| |
| Benchmarkers are welcomed and encouraged to employ the use of |
| compiler generated optimizations when building the code. |
| Specifying these optimizations should not require modification to |
| the source code, but instead should take the form of common |
| compiler "flags". These types of optimizations may be specified in |
| the two Makefiles used for that purpose. See Building the Code |
| for additional information. |
| |
| 4. *Maximum Number of MPI Tasks and OpenMP Threads* |
| |
| For the purpose of array dimensioning, particularly for the |
| collection of timing statistics, maximum values for these two |
| parameters are defined in the file includes/times.inc. The |
| distribution software sets these as follows: |
| |
| PARAMETER( maxMPItasks = 16384 ) |
| PARAMETER( maxThreadsPerMPItask = 128 ) |
| |
| If these values need to be increased, see the Building the Code |
| section for instructions. |
| |
| 5. *Required Problems* |
| |
| Please see the Sphot Sequoia Specific Problem set for the problems |
| to be run, the results to be reported, and the description of the |
| calculation of the Figure of Merit (FOM) for the runs. |
| ------------------------------------------------------------------------ |
| |
| |
| Release and Modification Record |
| |
| Version 1.0, this version. No release required. Public domain software. |
| ------------------------------------------------------------------------ |
| |
| For more information about this page, contact: |
| Tom Spelce , spelce1@llnl.gov mailto:spelce1@llnl.gov> |
| |
| <http://www.llnl.gov/> |
| |
| *UCRL-MI-144211* |
| September 19, 2001 |
| |
