Tests/RunCMake/CPack/README.txt - third_party/cmake - Git at Google

 RunCMake.CPack is a test module that is intended for testing of package
 generators that can be validated from command line.

 -------------
 Adding a test
 -------------

 CPack test root directory: 'Tests/RunCMake/CPack/tests'.

 All phases are executed separately for each generator that is bound to a test.
 Tests for each generator are subtests of test 'RunCMake.CPack_<generator_name>'.

 Each test must also be added to 'RunCMakeTest.cmake' script located in CPack
 test root directory.

 Line that adds a test is:
 run_cpack_test(<test_name> "<generator_name_list>" <compile_stage>
                "<packaging_type_list>")

 <generator_name_list> may be one generator e.g. "RPM" or multiple e.g. "RPM;DEB"
 and will be run for all listed generators. In test and verification scripts
 current generator can be accessed through GENERATOR_TYPE variable.

 <compile_stage> is a boolean variable that enables or disables compile stage -
 most tests don't require compilation as a non binary file can be used for
 package content but sometimes an executable or a library has to be created
 before the packaging stage.

 <packaging_type_list> can be a list of one or more packaging types: MONOLITHIC,
 COMPONENT or GROUP - each type sets per generator default variables which can
 be overwritten in the test if needed
 (see <generator_type>/packaging_<packaging_type>_default.cmake for the variables
 that are set by default for each packaging type).
 Alternatively CUSTOM value can be set which means that default values will not
 be set and that values will be set manually in the test itself.

 Alternatively a test with subtests can be added:
 run_cpack_test_subtests(<test_name> "<subtests_list>" "<generator_name_list>"
                         <compile_stage> "<packaging_type_list>")

 <subtests_list> is the only new parameter and it is a list of names that will
 be used for subtests. In test and verification scripts subtest name can be
 accessed through RunCMake_SUBTEST_SUFFIX variable.

 Also source package tests can be added:
 run_cpack_source_test(<test_name> "<generator_name_list>" true)

 Test consists of
 - test prerequirements phase (optional)
 - CMake execution phase
 - build phase (optional and not available for source package tests)
 - CPack execution phase
 - verification of generated files

 The phases are executed once per specified generator, packaging type and subtest
 combinatiion.

 test prerequirements phase (optional):
 --------------------------------------

 In some cases individual tests for certain generator need additional
 prerequirements met.

 In such cases '<test_name>/<generator_name>-Prerequirements.cmake' file
 containing 'function(get_test_prerequirements found_var config_file)' should be
 created. Function should return true in found_var if all prerequirements are
 met. config_file variable contains the location of config file generated by the
 generator so that this function can check if prerequired variable is set in the
 file.
 NOTE: All required programs should be searched for in generator prerequirements
 function and only checked for the variable in configure file in per test
 function.

 If prerequirements are not met test will be skipped outputting
 '<test_name> - SKIPPED' string. Note that this doesn't fail the entire test
 group.

 TODO: skipped tests should provide expected error string so test should fail
       if error string is not found in the output of run test (this would add
       'EXPECTED FAIL' string on success and 'ERROR' on failure).

 CMake execution phase:
 ----------------------

 To add a new CPack test we first create a '<test_name>/test.cmake' script that
 contains CMake commands that should be used as a preparation script for
 generation of different types of packages. This script is placed into CPack
 test root directory.

 If test will be used for multiple generators but some of them require some
 generator specific commands then those commands should be added to 'test.cmake'
 script wrapped with 'if(GENERATOR_TYPE STREQUAL <name_of_the_generator>)'.

 NOTE: In some cases (for example when testing CPackComponent.cmake functions)
 the test has to run some functions after CPack.cmake is included. In such cases
 a function run_after_include_cpack can be declared in test.cmake file and that
 function will run after the inclusion of CPack.cmake.

 NOTE: During CMake configure stage developer warnings may be expected. In such
 cases an expected output regular expression can be provided by creating
 '<test_name>/configure-stdout.txt' and/or '<test_name>/configure-stderr.txt'
 file. There are also more specialized versions of the file available:
 - configure-${PACKAGING_TYPE}-${SUBTEST_SUFFIX}-std${o}.txt
 - configure-${SUBTEST_SUFFIX}-std${o}.txt
 - configure-${PACKAGING_TYPE}-std${o}.txt

 build phase (optional and not available for source package tests)
 -----------------------------------------------------------------

 This phase only runs make command.

 NOTE: By default all tests have CMAKE_BUILD_TYPE variable set to Debug.

 CPack execution phase:
 ----------------------

 Only executes CPack for content that was generated during CMake execution
 phase.

 NOTE: By default CPACK_PACKAGE_NAME variable is set to lower case test name.

 Verification of generated files:
 --------------------------------

 Verification of generated files consists of two phases
 - mandatory verification phase
 - optional verification phase

 Mandatory verification phase checks that expected files were generated and
 contain expected files.
 Mandatory verification phase also checks that no other unexpected package files
 were generated (this is executed even if EXPECTED_FILES_COUNT contains 0 in
 order to verify that no files were generated).
 CMake script '<test_name>/ExpectedFiles.cmake' is required by
 this step and must contain
 - EXPECTED_FILES_COUNT variable that contains the number of expected files that
   will be generated (0 or more)

 - EXPECTED_FILE_<file_number_starting_with_1> that contains globing expression
   that uniquely defines expected file name (will be used to find expected file)
   and should be present once for each expected file.
   NOTE: This variable should be used only as last resort as it sets generator
         specific globbing expression. Each generator can recreate file name from
         parts that are already populated by default but can always be
         overwritten if needed:
   - EXPECTED_FILE_<file_number_starting_with_1>_NAME is the name component of
     the file (by default it is set to package name that is same as test name
     in lowercase)
   - EXPECTED_FILE_<file_number_starting_with_1>_VERSION is the version of the
     package (by default it is set to '0.1.1')
   - EXPECTED_FILE_<file_number_starting_with_1>_REVISION is the revision of the
     package (by default it is set to '1')

 - EXPECTED_FILE_CONTENT_<file_number_starting_with_1> that contains regular
   expression of files that should be present in generated file and should be
   present once for each expected file
   NOTE: This variable should be used only as last resort as it sets generator
         specific regular expression.
         EXPECTED_FILE_CONTENT_<file_number_starting_with_1>_LIST should be
         preferred as it requires a list of expected files and directories that
         is later changed automatically depending on the generator so expected
         package content can be written only once per test for all generators.

 - EXPECTED_FILE_PACKAGING_PREFIX and
   EXPECTED_FILE_<file_number_starting_with_1>_PACKAGING_PREFIX variables can be
   set to explicitly specified CPACK_PACKAGING_PREFIX value. By default this
   variable does not need to be set as it is implicitly set to package generator
   specific prefix.

 Optional verification phase is generator specific and is optionally executed.
 This phase is executed if '<test_name>/VerifyResult.cmake' script exists.

 VerifyResult.cmake script also automatically prints out standard output and
 standard error from CPack execution phase that is compared with
 '<test_name>/<generator_name>-stdout.txt' regular expression and
 '<test_name>/<generator_name>-stderr.txt' regular expresson respectively.
 NOTE: For subtests generator name can also be suffixed with subtest name and/or
       packaging type (MONOLITHIC, COMPONENT, GROUP) and in such cases the
       preferences of which file will be used are as follows:
         - generator name + packaging type + subtest name
         - generator name + packaging type
         - generator name + subtest name
         - generator name
         - subtest name
         - default generator
       File name format: '<generator_name>-<packaging_type>-<subtest_name>-std<type>.txt'
                         where <type> can either be 'out' or 'err'.
       File name format: '<generator_name>-<packaging_type>-std<type>.txt'
                         where <type> can either be 'out' or 'err'.
       File name format: '<generator_name>-<subtest_name>-std<type>.txt' where
                         <type> can either be 'out' or 'err'.
 NOTE: If none of the comparison files are present then the default generator
       file is used if present.

 ----------------------
 Adding a new generator
 ----------------------

 To add a new generator we must
 - add new generator directory (e.g. RPM for RPM generator) to CPack test root
   directory that contains 'Helpers.cmake' script.
   - In this script some functions must exist:
     - getPackageContent: This function should list files that are contained in
                          a package.
       Function parameters:
       + FILE variable that will contain path to file for which the content
         should be listed
       + RESULT_VAR that will tell the function which variable in parent scope
         should contain the result (list of files inside package file)
     - getPackageNameGlobexpr: This function should generate package name
                               globbing expression.
       Function parameters:
       + NAME that will contain the expected package name
       + COMPONENT that will contain the expected package component
       + VERSION that will contain the expected package version
       + REVISION that will contain the expected package revision number
       + FILE_NO that will contain the file number for which the globbing
         expression is generated
       + RESULT_VAR that will tell the function which variable in parent scope
         should contain the result (file name globbing expression)
     - getPackageContentList: This function should return a list of files and
                              directories that are in the provided package.
       Function parameters:
       + FILE that will contain the package file for which the package content
         should be returned.
       + RESULT_VAR that will tell the function which variable in parent scope
         should contain the result (list of package content)
     - toExpectedContentList: This function should convert an expected package
                              content list into one that is expected for the
                              generator (e.g. rpm packages have install/relocate
                              path prefixes which aren't part of the package so
                              those paths have to be removed from the expected
                              content list).
       Function parameters:
       + FILE_NO that will contain the file number for which the conversion
         should be performed
       + CONTENT_VAR that will contain the input list and is also the variable
         in parent scope which should contain the result (converted content list)
 - add 'Prerequirements.cmake' script to generator directory. In this script a
   function named 'get_test_prerequirements' must exist. This function should
   set a variable in parent scope (name of the variable is the first parameter)
   that tells if prerequirements for test execution are met (certain programs,
   OS specifics, ...) and create a config file (name of the variable which
   contains file name and path is provided with the second parameter) that
   should contain 'set' commands for variables that will later be used in tests
   (e.g. location of dpkg program for DEB packages)
 - add tests the same way as described above
 - add generator to 'add_RunCMake_test_group' function call that is located in
   RunCMake CMakeLists.txt file
 - if needed add 'packaging_<packaging_type>_default.cmake' scripts that define
   default variables that will be set for each packaging type (MONOLITHIC,
   COMPONENT and GROUP)
 - if needed add 'default_expected_std<type>.txt' files where <type> is either
   'out' or 'err' which will contain default expected output of package
   generation regular expression.
 - add generator to list of other CPack generators in RunCMake/CMakeLists.txt
	RunCMake.CPack is a test module that is intended for testing of package
	generators that can be validated from command line.

	-------------
	Adding a test
	-------------

	CPack test root directory: 'Tests/RunCMake/CPack/tests'.

	All phases are executed separately for each generator that is bound to a test.
	Tests for each generator are subtests of test 'RunCMake.CPack_<generator_name>'.

	Each test must also be added to 'RunCMakeTest.cmake' script located in CPack
	test root directory.

	Line that adds a test is:
	run_cpack_test(<test_name> "<generator_name_list>" <compile_stage>
	"<packaging_type_list>")

	<generator_name_list> may be one generator e.g. "RPM" or multiple e.g. "RPM;DEB"
	and will be run for all listed generators. In test and verification scripts
	current generator can be accessed through GENERATOR_TYPE variable.

	<compile_stage> is a boolean variable that enables or disables compile stage -
	most tests don't require compilation as a non binary file can be used for
	package content but sometimes an executable or a library has to be created
	before the packaging stage.

	<packaging_type_list> can be a list of one or more packaging types: MONOLITHIC,
	COMPONENT or GROUP - each type sets per generator default variables which can
	be overwritten in the test if needed
	(see <generator_type>/packaging_<packaging_type>_default.cmake for the variables
	that are set by default for each packaging type).
	Alternatively CUSTOM value can be set which means that default values will not
	be set and that values will be set manually in the test itself.

	Alternatively a test with subtests can be added:
	run_cpack_test_subtests(<test_name> "<subtests_list>" "<generator_name_list>"
	<compile_stage> "<packaging_type_list>")

	<subtests_list> is the only new parameter and it is a list of names that will
	be used for subtests. In test and verification scripts subtest name can be
	accessed through RunCMake_SUBTEST_SUFFIX variable.

	Also source package tests can be added:
	run_cpack_source_test(<test_name> "<generator_name_list>" true)

	Test consists of
	- test prerequirements phase (optional)
	- CMake execution phase
	- build phase (optional and not available for source package tests)
	- CPack execution phase
	- verification of generated files

	The phases are executed once per specified generator, packaging type and subtest
	combinatiion.

	test prerequirements phase (optional):
	--------------------------------------

	In some cases individual tests for certain generator need additional
	prerequirements met.

	In such cases '<test_name>/<generator_name>-Prerequirements.cmake' file
	containing 'function(get_test_prerequirements found_var config_file)' should be
	created. Function should return true in found_var if all prerequirements are
	met. config_file variable contains the location of config file generated by the
	generator so that this function can check if prerequired variable is set in the
	file.
	NOTE: All required programs should be searched for in generator prerequirements
	function and only checked for the variable in configure file in per test
	function.

	If prerequirements are not met test will be skipped outputting
	'<test_name> - SKIPPED' string. Note that this doesn't fail the entire test
	group.

	TODO: skipped tests should provide expected error string so test should fail
	if error string is not found in the output of run test (this would add
	'EXPECTED FAIL' string on success and 'ERROR' on failure).

	CMake execution phase:
	----------------------

	To add a new CPack test we first create a '<test_name>/test.cmake' script that
	contains CMake commands that should be used as a preparation script for
	generation of different types of packages. This script is placed into CPack
	test root directory.

	If test will be used for multiple generators but some of them require some
	generator specific commands then those commands should be added to 'test.cmake'
	script wrapped with 'if(GENERATOR_TYPE STREQUAL <name_of_the_generator>)'.

	NOTE: In some cases (for example when testing CPackComponent.cmake functions)
	the test has to run some functions after CPack.cmake is included. In such cases
	a function run_after_include_cpack can be declared in test.cmake file and that
	function will run after the inclusion of CPack.cmake.

	NOTE: During CMake configure stage developer warnings may be expected. In such
	cases an expected output regular expression can be provided by creating
	'<test_name>/configure-stdout.txt' and/or '<test_name>/configure-stderr.txt'
	file. There are also more specialized versions of the file available:
	- configure-${PACKAGING_TYPE}-${SUBTEST_SUFFIX}-std${o}.txt
	- configure-${SUBTEST_SUFFIX}-std${o}.txt
	- configure-${PACKAGING_TYPE}-std${o}.txt

	build phase (optional and not available for source package tests)
	-----------------------------------------------------------------

	This phase only runs make command.

	NOTE: By default all tests have CMAKE_BUILD_TYPE variable set to Debug.

	CPack execution phase:
	----------------------

	Only executes CPack for content that was generated during CMake execution
	phase.

	NOTE: By default CPACK_PACKAGE_NAME variable is set to lower case test name.

	Verification of generated files:
	--------------------------------

	Verification of generated files consists of two phases
	- mandatory verification phase
	- optional verification phase

	Mandatory verification phase checks that expected files were generated and
	contain expected files.
	Mandatory verification phase also checks that no other unexpected package files
	were generated (this is executed even if EXPECTED_FILES_COUNT contains 0 in
	order to verify that no files were generated).
	CMake script '<test_name>/ExpectedFiles.cmake' is required by
	this step and must contain
	- EXPECTED_FILES_COUNT variable that contains the number of expected files that
	will be generated (0 or more)

	- EXPECTED_FILE_<file_number_starting_with_1> that contains globing expression
	that uniquely defines expected file name (will be used to find expected file)
	and should be present once for each expected file.
	NOTE: This variable should be used only as last resort as it sets generator
	specific globbing expression. Each generator can recreate file name from
	parts that are already populated by default but can always be
	overwritten if needed:
	- EXPECTED_FILE_<file_number_starting_with_1>_NAME is the name component of
	the file (by default it is set to package name that is same as test name
	in lowercase)
	- EXPECTED_FILE_<file_number_starting_with_1>_VERSION is the version of the
	package (by default it is set to '0.1.1')
	- EXPECTED_FILE_<file_number_starting_with_1>_REVISION is the revision of the
	package (by default it is set to '1')

	- EXPECTED_FILE_CONTENT_<file_number_starting_with_1> that contains regular
	expression of files that should be present in generated file and should be
	present once for each expected file
	NOTE: This variable should be used only as last resort as it sets generator
	specific regular expression.
	EXPECTED_FILE_CONTENT_<file_number_starting_with_1>_LIST should be
	preferred as it requires a list of expected files and directories that
	is later changed automatically depending on the generator so expected
	package content can be written only once per test for all generators.

	- EXPECTED_FILE_PACKAGING_PREFIX and
	EXPECTED_FILE_<file_number_starting_with_1>_PACKAGING_PREFIX variables can be
	set to explicitly specified CPACK_PACKAGING_PREFIX value. By default this
	variable does not need to be set as it is implicitly set to package generator
	specific prefix.

	Optional verification phase is generator specific and is optionally executed.
	This phase is executed if '<test_name>/VerifyResult.cmake' script exists.

	VerifyResult.cmake script also automatically prints out standard output and
	standard error from CPack execution phase that is compared with
	'<test_name>/<generator_name>-stdout.txt' regular expression and
	'<test_name>/<generator_name>-stderr.txt' regular expresson respectively.
	NOTE: For subtests generator name can also be suffixed with subtest name and/or
	packaging type (MONOLITHIC, COMPONENT, GROUP) and in such cases the
	preferences of which file will be used are as follows:
	- generator name + packaging type + subtest name
	- generator name + packaging type
	- generator name + subtest name
	- generator name
	- subtest name
	- default generator
	File name format: '<generator_name>-<packaging_type>-<subtest_name>-std<type>.txt'
	where <type> can either be 'out' or 'err'.
	File name format: '<generator_name>-<packaging_type>-std<type>.txt'
	where <type> can either be 'out' or 'err'.
	File name format: '<generator_name>-<subtest_name>-std<type>.txt' where
	<type> can either be 'out' or 'err'.
	NOTE: If none of the comparison files are present then the default generator
	file is used if present.

	----------------------
	Adding a new generator
	----------------------

	To add a new generator we must
	- add new generator directory (e.g. RPM for RPM generator) to CPack test root
	directory that contains 'Helpers.cmake' script.
	- In this script some functions must exist:
	- getPackageContent: This function should list files that are contained in
	a package.
	Function parameters:
	+ FILE variable that will contain path to file for which the content
	should be listed
	+ RESULT_VAR that will tell the function which variable in parent scope
	should contain the result (list of files inside package file)
	- getPackageNameGlobexpr: This function should generate package name
	globbing expression.
	Function parameters:
	+ NAME that will contain the expected package name
	+ COMPONENT that will contain the expected package component
	+ VERSION that will contain the expected package version
	+ REVISION that will contain the expected package revision number
	+ FILE_NO that will contain the file number for which the globbing
	expression is generated
	+ RESULT_VAR that will tell the function which variable in parent scope
	should contain the result (file name globbing expression)
	- getPackageContentList: This function should return a list of files and
	directories that are in the provided package.
	Function parameters:
	+ FILE that will contain the package file for which the package content
	should be returned.
	+ RESULT_VAR that will tell the function which variable in parent scope
	should contain the result (list of package content)
	- toExpectedContentList: This function should convert an expected package
	content list into one that is expected for the
	generator (e.g. rpm packages have install/relocate
	path prefixes which aren't part of the package so
	those paths have to be removed from the expected
	content list).
	Function parameters:
	+ FILE_NO that will contain the file number for which the conversion
	should be performed
	+ CONTENT_VAR that will contain the input list and is also the variable
	in parent scope which should contain the result (converted content list)
	- add 'Prerequirements.cmake' script to generator directory. In this script a
	function named 'get_test_prerequirements' must exist. This function should
	set a variable in parent scope (name of the variable is the first parameter)
	that tells if prerequirements for test execution are met (certain programs,
	OS specifics, ...) and create a config file (name of the variable which
	contains file name and path is provided with the second parameter) that
	should contain 'set' commands for variables that will later be used in tests
	(e.g. location of dpkg program for DEB packages)
	- add tests the same way as described above
	- add generator to 'add_RunCMake_test_group' function call that is located in
	RunCMake CMakeLists.txt file
	- if needed add 'packaging_<packaging_type>_default.cmake' scripts that define
	default variables that will be set for each packaging type (MONOLITHIC,
	COMPONENT and GROUP)
	- if needed add 'default_expected_std<type>.txt' files where <type> is either
	'out' or 'err' which will contain default expected output of package
	generation regular expression.
	- add generator to list of other CPack generators in RunCMake/CMakeLists.txt