| RunCMake.CPack is a test module that is intended for testing of package |
| generators that can be validated from command line. |
| |
| ------------- |
| Adding a test |
| ------------- |
| |
| CPack test root directory: 'Tests/RunCMake/CPack/tests'. |
| |
| All phases are executed separately for each generator that is bound to a test. |
| Tests for each generator are subtests of test 'RunCMake.CPack_<generator_name>'. |
| |
| Each test must also be added to 'RunCMakeTest.cmake' script located in CPack |
| test root directory. |
| |
| Line that adds a test is: |
| run_cpack_test(<test_name> "<generator_name_list>" <compile_stage> |
| "<packaging_type_list>") |
| |
| <generator_name_list> may be one generator e.g. "RPM" or multiple e.g. "RPM;DEB" |
| and will be run for all listed generators. In test and verification scripts |
| current generator can be accessed through GENERATOR_TYPE variable. |
| |
| <compile_stage> is a boolean variable that enables or disables compile stage - |
| most tests don't require compilation as a non binary file can be used for |
| package content but sometimes an executable or a library has to be created |
| before the packaging stage. |
| |
| <packaging_type_list> can be a list of one or more packaging types: MONOLITHIC, |
| COMPONENT or GROUP - each type sets per generator default variables which can |
| be overwritten in the test if needed |
| (see <generator_type>/packaging_<packaging_type>_default.cmake for the variables |
| that are set by default for each packaging type). |
| Alternatively CUSTOM value can be set which means that default values will not |
| be set and that values will be set manually in the test itself. |
| |
| Alternatively a test with subtests can be added: |
| run_cpack_test_subtests(<test_name> "<subtests_list>" "<generator_name_list>" |
| <compile_stage> "<packaging_type_list>") |
| |
| <subtests_list> is the only new parameter and it is a list of names that will |
| be used for subtests. In test and verification scripts subtest name can be |
| accessed through RunCMake_SUBTEST_SUFFIX variable. |
| |
| Also source package tests can be added: |
| run_cpack_source_test(<test_name> "<generator_name_list>" true) |
| |
| Test consists of |
| - test prerequirements phase (optional) |
| - CMake execution phase |
| - build phase (optional and not available for source package tests) |
| - CPack execution phase |
| - verification of generated files |
| |
| The phases are executed once per specified generator, packaging type and subtest |
| combinatiion. |
| |
| test prerequirements phase (optional): |
| -------------------------------------- |
| |
| In some cases individual tests for certain generator need additional |
| prerequirements met. |
| |
| In such cases '<test_name>/<generator_name>-Prerequirements.cmake' file |
| containing 'function(get_test_prerequirements found_var config_file)' should be |
| created. Function should return true in found_var if all prerequirements are |
| met. config_file variable contains the location of config file generated by the |
| generator so that this function can check if prerequired variable is set in the |
| file. |
| NOTE: All required programs should be searched for in generator prerequirements |
| function and only checked for the variable in configure file in per test |
| function. |
| |
| If prerequirements are not met test will be skipped outputting |
| '<test_name> - SKIPPED' string. Note that this doesn't fail the entire test |
| group. |
| |
| TODO: skipped tests should provide expected error string so test should fail |
| if error string is not found in the output of run test (this would add |
| 'EXPECTED FAIL' string on success and 'ERROR' on failure). |
| |
| CMake execution phase: |
| ---------------------- |
| |
| To add a new CPack test we first create a '<test_name>/test.cmake' script that |
| contains CMake commands that should be used as a preparation script for |
| generation of different types of packages. This script is placed into CPack |
| test root directory. |
| |
| If test will be used for multiple generators but some of them require some |
| generator specific commands then those commands should be added to 'test.cmake' |
| script wrapped with 'if(GENERATOR_TYPE STREQUAL <name_of_the_generator>)'. |
| |
| build phase (optional and not available for source package tests) |
| ----------------------------------------------------------------- |
| |
| This phase only runs make command. |
| |
| NOTE: By default all tests have CMAKE_BUILD_TYPE variable set to Debug. |
| |
| CPack execution phase: |
| ---------------------- |
| |
| Only executes CPack for content that was generated during CMake execution |
| phase. |
| |
| NOTE: By default CPACK_PACKAGE_NAME variable is set to lower case test name. |
| |
| Verification of generated files: |
| -------------------------------- |
| |
| Verification of generated files consists of two phases |
| - mandatory verification phase |
| - optional verification phase |
| |
| Mandatory verification phase checks that expected files were generated and |
| contain expected files. |
| Mandatory verification phase also checks that no other unexpected package files |
| were generated (this is executed even if EXPECTED_FILES_COUNT contains 0 in |
| order to verify that no files were generated). |
| CMake script '<test_name>/ExpectedFiles.cmake' is required by |
| this step and must contain |
| - EXPECTED_FILES_COUNT variable that contains the number of expected files that |
| will be generated (0 or more) |
| |
| - EXPECTED_FILE_<file_number_starting_with_1> that contains globing expression |
| that uniquely defines expected file name (will be used to find expected file) |
| and should be present once for each expected file. |
| NOTE: This variable should be used only as last resort as it sets generator |
| specific globbing expression. Each generator can recreate file name from |
| parts that are already populated by default but can always be |
| overwritten if needed: |
| - EXPECTED_FILE_<file_number_starting_with_1>_NAME is the name component of |
| the file (by default it is set to package name that is same as test name |
| in lowercase) |
| - EXPECTED_FILE_<file_number_starting_with_1>_VERSION is the version of the |
| package (by default it is set to '0.1.1') |
| - EXPECTED_FILE_<file_number_starting_with_1>_REVISION is the revision of the |
| package (by default it is set to '1') |
| |
| - EXPECTED_FILE_CONTENT_<file_number_starting_with_1> that contains regular |
| expression of files that should be present in generated file and should be |
| present once for each expected file |
| NOTE: This variable should be used only as last resort as it sets generator |
| specific regular expression. |
| EXPECTED_FILE_CONTENT_<file_number_starting_with_1>_LIST should be |
| prefered as it requires a list of expected files and directories that |
| is later changed automatically depending on the generator so expected |
| package content can be written only once per test for all generators. |
| |
| Optional verification phase is generator specific and is optionaly executed. |
| This phase is executed if '<test_name>/VerifyResult.cmake' script exists. |
| |
| VerifyResult.cmake script also automatically prints out standard output and |
| standard error from CPack execution phase that is compared with |
| '<test_name>/<generator_name>-stdout.txt' regular expression and |
| and '<test_name>/<generator_name>-stderr.txt' regular expresson respectively. |
| NOTE: For subtests generator name can also be suffixed with subtest name and/or |
| packaging type (MONOLITHIC, COMPONENT, GROUP) and in such cases the |
| preferences of which file will be used are as follows: |
| - generator name + packaging type + subtest name |
| - generator name + packaging type |
| - generator name + subtest name |
| - generator name |
| - default generator |
| File name format: '<generator_name>-<packaging_type>-<subtest_name>-std<type>.txt' |
| where <type> can either be 'out' or 'err'. |
| File name format: '<generator_name>-<packaging_type>-std<type>.txt' |
| where <type> can either be 'out' or 'err'. |
| File name format: '<generator_name>-<subtest_name>-std<type>.txt' where |
| <type> can either be 'out' or 'err'. |
| NOTE: If none of the comparison files are present then the default generator |
| file is used if present. |
| |
| ---------------------- |
| Adding a new generator |
| ---------------------- |
| |
| To add a new generator we must |
| - add new generator directory (e.g. RPM for RPM generator) to CPack test root |
| directory that contains 'Helpers.cmake' script. |
| - In this script some functions must exist: |
| - getPackageContent: This function should list files that are contained in |
| a package. |
| Function parameters: |
| + FILE variable that will contain path to file for which the content |
| should be listed |
| + RESULT_VAR that will tell the function which variable in parent scope |
| should contain the result (list of files inside package file) |
| - getPackageNameGlobexpr: This function should generate package name |
| globbing expression. |
| Function parameters: |
| + NAME that will contain the expected package name |
| + COMPONENT that will contain the expected package component |
| + VERSION that will contain the expected package version |
| + REVISION that will contain the expected package revision number |
| + FILE_NO that will contain the file number for which the globbing |
| expression is generated |
| + RESULT_VAR that will tell the function which variable in parent scope |
| should contain the result (file name globbing expression) |
| - getPackageContentList: This function should return a list of files and |
| directories that are in the provided package. |
| Function parameters: |
| + FILE that will contain the package file for which the package content |
| should be returned. |
| + RESULT_VAR that will tell the function which variable in parent scope |
| should contain the result (list of pacakge content) |
| - toExpectedContentList: This function should convert an expected package |
| content list into one that is expected for the |
| generator (e.g. rpm packages have install/relocate |
| path prefixes which aren't part of the package so |
| those paths have to be removed from the expected |
| content list). |
| Function parameters: |
| + FILE_NO that will contain the file number for which the conversion |
| should be performed |
| + CONTENT_VAR that will contain the input list and is also the variable |
| in parent scope which should contain the result (converted content list) |
| - add 'Prerequirements.cmake' script to generator directory. In this script a |
| function named 'get_test_prerequirements' must exist. This function should |
| set a variable in parent scope (name of the variable is the first parameter) |
| that tells if prerequirements for test execution are met (certain programs, |
| OS specifics, ...) and create a config file (name of the variable which |
| contains file name and path is provided with the second parameter) that |
| should contain 'set' commands for variables that will later be used in tests |
| (e.g. location of dpkg program for DEB packages) |
| - add tests the same way as described above |
| - add generator to 'add_RunCMake_test_group' function call that is located in |
| RunCMake CMakeLists.txt file |
| - if needed add 'packaging_<packaging_type>_default.cmake' scripts that define |
| default variables that will be set for each packaging type (MONOLITHIC, |
| COMPONENT and GROUP) |
| - if needed add 'default_expected_std<type>.txt' files where <type> is either |
| 'out' or 'err' which will contain default expected output of package |
| generation regular expression. |
| - add generator to list of other CPack generators in RunCMake/CMakeLists.txt |