external/vulkancts/README.md - third_party/vulkan-cts - Git at Google

 Vulkan CTS README
 =================

 This document describes how to build and run Vulkan Conformance Test suite.

 Vulkan CTS is built on dEQP framework. dEQP documentation is available
 at http://source.android.com/devices/graphics/testing.html


 Requirements
 ------------

 ### Common

  * Git (for checking out sources)
  * Python 2.7.x (all recent versions in 2.x should work, 3.x is not supported)
  * CMake 2.8 or newer

 ### Win32

  * Visual Studio 2013 or newer (glslang uses several C++11 features)

 ### Linux

  * Standard toolchain (make, gcc/clang)

 ### Android

  * Android NDK r11
  * Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 22
  * Java Development Kit (JDK)
  * Apache Ant
  * Windows: either NMake or JOM in PATH

 See `android/scripts/common.py` for a list locations where the build system
 expects to find these.


 Building CTS
 ------------

 To build dEQP, you need first to download sources for zlib, libpng, glslang,
 and spirv-tools.

 To download sources, run:

 	python external/fetch_sources.py

 You may need to re-run `fetch_sources.py` to update to the latest glslang and
 spirv-tools revisions occasionally.

 With CMake out-of-source builds are always recommended. Create a build directory
 of your choosing, and in that directory generate Makefiles or IDE project
 using cmake.


 ### Windows x86-32

 	cmake <path to vulkancts> -G"Visual Studio 12"
 	start dEQP-Core-default.sln

 ### Windows x86-64

 	cmake <path to vulkancts> -G"Visual Studio 12 Win64"
 	start dEQP-Core-default.sln

 ### Linux 32-bit Debug

 	cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
 	make -j

 Release build can be done by using -DCMAKE_BUILD_TYPE=Release

 ### Linux 64-bit Debug

 	cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
 	make -j

 ### Android

 Following command will build CTS into android/package/bin/dEQP-debug.apk.

 	python android/scripts/build.py

 The package can be installed by either running:

 	python android/scripts/install.py

 By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
 and x86 ABIs, but that can be changed in android/scripts/common.py script.

 To pick which ABI to use at install time, following commands must be used
 instead:

 	adb install --abi <ABI name> android/package/bin/dEQP-debug.apk /data/local/tmp/dEQP-debug.apk


 Building Mustpass
 -----------------

 Current mustpass is checked into repository and can be found at:

 	external/vulkancts/mustpass/1.0.2/vk-default.txt

 Vulkan CTS mustpass can be re-generated by running:

 	python <vulkancts>/external/vulkancts/build_mustpass.py


 Pre-compiling SPIR-V binaries
 -----------------------------

 For distribution, and platforms that don't support GLSL to SPIR-V compilation,
 SPIR-V binaries can be pre-built with following command:

 	python external/vulkancts/scripts/build_spirv_binaries.py

 Binaries will be written to `external/vulkancts/data/vulkan/prebuilt/`.

 Test modules (or in case of Android, the APK) must be re-built after building
 SPIR-V programs in order for the binaries to be available.


 Running CTS
 -----------

 Following command line options MUST be used when running CTS:

 	--deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/1.0.2/vk-default.txt
 	--deqp-log-images=disable
 	--deqp-log-shader-sources=disable

 In addition on multi-device systems the device for which conformance is claimed
 can be selected with:

 	--deqp-vk-device-id=<value>

 To speed up the conformance run on some platforms the following command line
 option may be used to disable frequent fflush() calls to the output logs:

 	--deqp-log-flush=disable

 By default, the test log will be written into the path "TestResults.qpa". If the
 platform requires a different path, it can be specified with:

 	--deqp-log-filename=<path>

 No other command line options are allowed.

 ### Win32

 	cd <builddir>/external/vulkancts/modules/vulkan
 	Debug/deqp-vk.exe --deqp-caselist-file=...

 Test log will be written into TestResults.qpa

 ### Linux

 	cd <builddir>/external/vulkancts/modules/vulkan
 	./deqp-vk --deqp-vk-caselist-file=...

 ### Android

 	adb push <vulkancts>/external/vulkancts/mustpass/1.0.2/vk-default.txt /sdcard/vk-default.txt
 	adb shell

 In device shell:

 	am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"

 Test progress will be written to device log and can be displayed with:

 	adb logcat -s dEQP

 Test log will be written into `/sdcard/TestResults.qpa`.


 Conformance Submission Package Requirements
 -------------------------------------------

 The conformance submission package must contain the following:

 1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds
 2. Result of `git status` and `git log` from CTS source directory
 3. Any patches used on top of release tag
 4. Conformance statement

 Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>.qpa`,
 for example `TestResults-armeabi-v7a.qpa`. On platforms where multiple different driver
 builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
 for each driver build as part of the submission package.

 The CTS build must always be done from clean git repository that doesn't have any
 uncommitted changes. Thus it is necessary to run and capture output of `git
 status` and `git log` (2) in the source directory:

 	git status > <submission pkg dir>/git-status.txt
 	git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt

 Any changes made to CTS must be committed to the local repository, and provided
 as part of the submission package (3). This can be done by running:

 	git format-patch -o <submission pkg dir> <release tag>..HEAD

 In general, bugfixes and changes to platform-specific code (mostly under
 `framework/platform`) are allowed. The commit message for each change must
 include a clear description of the change and why it is necessary. Non-porting
 related changes must be accompanied by a waiver (see below).

 NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
 to include original commit hash in the commit message.

 Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
 and must contain following:

 	CONFORM_VERSION:         <git tag of CTS release>
 	PRODUCT:                 <string-value>
 	CPU:                     <string-value>
 	OS:                      <string-value>

 Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
 if `vk::Platform::describePlatform()` is implemented.

 If the submission package covers multiple products, you can list them by appending
 additional `PRODUCT:` lines to the conformance statement. For example:

 	CONFORM_VERSION:         vulkan-cts-1.0.2.0
 	PRODUCT:                 Product A
 	PRODUCT:                 Product B
 	...

 The actual submission package consists of the above set of files which must
 be bundled into a gzipped tar file named `VK10_<adopter><_info>.tgz`. `<adopter>`
 is the name of the Adopting member company, or some recognizable abbreviation.
 The `<_info>` field is optional. It may be used to uniquely identify a submission
 by OS, platform, date, or other criteria when making multiple submissions.

 One way to create a suiteable gzipped tar file is to execute the command:

 	tar -cvzf <filename.tgz> -C <submission pkg dir> .

 where `<submission pkg dir>` is the directory containing the files from (1)-(4)
 from above. A submission package must contain all of the files listed above,
 and only those files.

 As an example submission package could contain:

 	STATEMENT-Khronos
 	git-log.txt
 	git-status.txt
 	0001-Remove-Waived-Filtering-Tests.patch
 	0002-Fix-Pipeline-Parameters.patch
 	TestResults-armeabi-v7a.qpa
 	TestResults-arm64-v8a.qpa


 Waivers
 -------

 The process for requesting a waiver is to report the issue by filing a bug
 report in the Gitlab VulkanCTS project (TODO Github?). When creating the
 submission package, include references to the waiver in the commit message of
 the relevant change. Including as much information as possible in your bug
 report (including a unified diff or a merge request of suggested file changes)
 will ensure the issue can be progressed as rapidly as possible. Issues must
 be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
 tests.

 Conformance Criteria
 --------------------

 Conformance run is considered passing if all tests finish with allowed result
 codes. Test results are contained in the TestResults.qpa log. Each
 test case section contains XML tag Result, for example:

 	<Result StatusCode="Pass">Not validated</Result>

 The result code is the value of the StatusCode attribute. Following status
 codes are allowed:

 	Pass
 	NotSupported
 	QualityWarning
 	CompatibilityWarning

 Submission package can be verified using `external/vulkancts/scripts/verify_submission.py`
 script. The script takes two arguments: path to extracted submission package
 and path to current mustpass list. For example:

 	python external/vulkancts/scripts/verify_submission.py VK_10_Khronos_1/ external/vulkancts/mustpass/1.0.2/vk-default.txt


 Vulkan platform port
 --------------------

 Vulkan support from Platform implementation requires providing
 `getVulkanPlatform()` method in `tcu::Platform` class implementation.

 See `framework/common/tcuPlatform.hpp` and examples in
 `framework/platform/win32/tcuWin32Platform.cpp` and
 `framework/platform/android/tcuAndroidPlatform.cpp`.

 If any WSI extensions are supported, platform port must also implement
 methods for creating native display (`vk::Platform::createWsiDisplay`)
 and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
 under `dEQP-VK.wsi` will fail.


 Null (dummy) driver
 -------------------

 For testing and development purposes it might be useful to be able to run
 tests on dummy Vulkan implementation. One such implementation is provided in
 vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
 `vk::createNullDriver()`.


 Validation Layers
 -----------------

 Vulkan CTS framework includes first-party support for validation layers, that
 can be turned on with `--deqp-validation=enable` command line option.

 When validation is turned on, default instance and device will be created with
 validation layers enabled and debug callback is registered to record any
 messages. Debug messages collected during test execution will be included at
 the end of the test case log.

 If any validation errors are found, test result will be set to `InternalError`.

 By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
 are excluded from the log, but that can be customized by modifying
 `vkt::TestCaseExecutor::deinit()` in `vktTestPackage.cpp`.


 Cherry GUI
 ----------

 Vulkan test module can be used with Cherry (GUI for test execution and
 analysis). Cherry is available at
 https://android.googlesource.com/platform/external/cherry. Please follow
 instructions in README to get started.

 Before first launch, and every time test hierarchy has been modified, test
 case list must be refreshed by running:

 	python scripts/build_caselists.py path/to/cherry/data

 Cherry must be restarted for the case list update to take effect.
	Vulkan CTS README
	=================

	This document describes how to build and run Vulkan Conformance Test suite.

	Vulkan CTS is built on dEQP framework. dEQP documentation is available
	at http://source.android.com/devices/graphics/testing.html


	Requirements
	------------

	### Common

	* Git (for checking out sources)
	* Python 2.7.x (all recent versions in 2.x should work, 3.x is not supported)
	* CMake 2.8 or newer

	### Win32

	* Visual Studio 2013 or newer (glslang uses several C++11 features)

	### Linux

	* Standard toolchain (make, gcc/clang)

	### Android

	* Android NDK r11
	* Android SDK with: SDK Tools, SDK Platform-tools, SDK Build-tools, and API 22
	* Java Development Kit (JDK)
	* Apache Ant
	* Windows: either NMake or JOM in PATH

	See `android/scripts/common.py` for a list locations where the build system
	expects to find these.


	Building CTS
	------------

	To build dEQP, you need first to download sources for zlib, libpng, glslang,
	and spirv-tools.

	To download sources, run:

	python external/fetch_sources.py

	You may need to re-run `fetch_sources.py` to update to the latest glslang and
	spirv-tools revisions occasionally.

	With CMake out-of-source builds are always recommended. Create a build directory
	of your choosing, and in that directory generate Makefiles or IDE project
	using cmake.


	### Windows x86-32

	cmake <path to vulkancts> -G"Visual Studio 12"
	start dEQP-Core-default.sln

	### Windows x86-64

	cmake <path to vulkancts> -G"Visual Studio 12 Win64"
	start dEQP-Core-default.sln

	### Linux 32-bit Debug

	cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m32 -DCMAKE_CXX_FLAGS=-m32
	make -j

	Release build can be done by using -DCMAKE_BUILD_TYPE=Release

	### Linux 64-bit Debug

	cmake <path to vulkancts> -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS=-m64 -DCMAKE_CXX_FLAGS=-m64
	make -j

	### Android

	Following command will build CTS into android/package/bin/dEQP-debug.apk.

	python android/scripts/build.py

	The package can be installed by either running:

	python android/scripts/install.py

	By default the CTS package will contain libdeqp.so built for armeabi-v7a, arm64-v8a,
	and x86 ABIs, but that can be changed in android/scripts/common.py script.

	To pick which ABI to use at install time, following commands must be used
	instead:

	adb install --abi <ABI name> android/package/bin/dEQP-debug.apk /data/local/tmp/dEQP-debug.apk


	Building Mustpass
	-----------------

	Current mustpass is checked into repository and can be found at:

	external/vulkancts/mustpass/1.0.2/vk-default.txt

	Vulkan CTS mustpass can be re-generated by running:

	python <vulkancts>/external/vulkancts/build_mustpass.py


	Pre-compiling SPIR-V binaries
	-----------------------------

	For distribution, and platforms that don't support GLSL to SPIR-V compilation,
	SPIR-V binaries can be pre-built with following command:

	python external/vulkancts/scripts/build_spirv_binaries.py

	Binaries will be written to `external/vulkancts/data/vulkan/prebuilt/`.

	Test modules (or in case of Android, the APK) must be re-built after building
	SPIR-V programs in order for the binaries to be available.


	Running CTS
	-----------

	Following command line options MUST be used when running CTS:

	--deqp-caselist-file=<vulkancts>/external/vulkancts/mustpass/1.0.2/vk-default.txt
	--deqp-log-images=disable
	--deqp-log-shader-sources=disable

	In addition on multi-device systems the device for which conformance is claimed
	can be selected with:

	--deqp-vk-device-id=<value>

	To speed up the conformance run on some platforms the following command line
	option may be used to disable frequent fflush() calls to the output logs:

	--deqp-log-flush=disable

	By default, the test log will be written into the path "TestResults.qpa". If the
	platform requires a different path, it can be specified with:

	--deqp-log-filename=<path>

	No other command line options are allowed.

	### Win32

	cd <builddir>/external/vulkancts/modules/vulkan
	Debug/deqp-vk.exe --deqp-caselist-file=...

	Test log will be written into TestResults.qpa

	### Linux

	cd <builddir>/external/vulkancts/modules/vulkan
	./deqp-vk --deqp-vk-caselist-file=...

	### Android

	adb push <vulkancts>/external/vulkancts/mustpass/1.0.2/vk-default.txt /sdcard/vk-default.txt
	adb shell

	In device shell:

	am start -n com.drawelements.deqp/android.app.NativeActivity -e cmdLine "deqp --deqp-caselist-file=/sdcard/vk-default.txt --deqp-log-images=disable --deqp-log-shader-sources=disable --deqp-log-filename=/sdcard/TestResults.qpa"

	Test progress will be written to device log and can be displayed with:

	adb logcat -s dEQP

	Test log will be written into `/sdcard/TestResults.qpa`.


	Conformance Submission Package Requirements
	-------------------------------------------

	The conformance submission package must contain the following:

	1. Full test logs (`TestResults.qpa`) from CTS runs against all driver builds
	2. Result of `git status` and `git log` from CTS source directory
	3. Any patches used on top of release tag
	4. Conformance statement

	Test logs (1) should be named `<submission pkg dir>/TestResults-<driver build type>.qpa`,
	for example `TestResults-armeabi-v7a.qpa`. On platforms where multiple different driver
	builds (for example 64-bit and 32-bit) are present, CTS logs must be provided
	for each driver build as part of the submission package.

	The CTS build must always be done from clean git repository that doesn't have any
	uncommitted changes. Thus it is necessary to run and capture output of `git
	status` and `git log` (2) in the source directory:

	git status > <submission pkg dir>/git-status.txt
	git log --first-parent <release tag>^..HEAD > <submission pkg dir>/git-log.txt

	Any changes made to CTS must be committed to the local repository, and provided
	as part of the submission package (3). This can be done by running:

	git format-patch -o <submission pkg dir> <release tag>..HEAD

	In general, bugfixes and changes to platform-specific code (mostly under
	`framework/platform`) are allowed. The commit message for each change must
	include a clear description of the change and why it is necessary. Non-porting
	related changes must be accompanied by a waiver (see below).

	NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
	to include original commit hash in the commit message.

	Conformance statement (4) must be included in a file called `STATEMENT-<adopter>`
	and must contain following:

	CONFORM_VERSION: <git tag of CTS release>
	PRODUCT: <string-value>
	CPU: <string-value>
	OS: <string-value>

	Note that product/cpu/os information is also captured in `dEQP-VK.info.*` tests
	if `vk::Platform::describePlatform()` is implemented.

	If the submission package covers multiple products, you can list them by appending
	additional `PRODUCT:` lines to the conformance statement. For example:

	CONFORM_VERSION: vulkan-cts-1.0.2.0
	PRODUCT: Product A
	PRODUCT: Product B
	...

	The actual submission package consists of the above set of files which must
	be bundled into a gzipped tar file named `VK10_<adopter><_info>.tgz`. `<adopter>`
	is the name of the Adopting member company, or some recognizable abbreviation.
	The `<_info>` field is optional. It may be used to uniquely identify a submission
	by OS, platform, date, or other criteria when making multiple submissions.

	One way to create a suiteable gzipped tar file is to execute the command:

	tar -cvzf <filename.tgz> -C <submission pkg dir> .

	where `<submission pkg dir>` is the directory containing the files from (1)-(4)
	from above. A submission package must contain all of the files listed above,
	and only those files.

	As an example submission package could contain:

	STATEMENT-Khronos
	git-log.txt
	git-status.txt
	0001-Remove-Waived-Filtering-Tests.patch
	0002-Fix-Pipeline-Parameters.patch
	TestResults-armeabi-v7a.qpa
	TestResults-arm64-v8a.qpa


	Waivers
	-------

	The process for requesting a waiver is to report the issue by filing a bug
	report in the Gitlab VulkanCTS project (TODO Github?). When creating the
	submission package, include references to the waiver in the commit message of
	the relevant change. Including as much information as possible in your bug
	report (including a unified diff or a merge request of suggested file changes)
	will ensure the issue can be progressed as rapidly as possible. Issues must
	be labeled "Waiver" (TODO!) and identify the version of the CTS and affected
	tests.

	Conformance Criteria
	--------------------

	Conformance run is considered passing if all tests finish with allowed result
	codes. Test results are contained in the TestResults.qpa log. Each
	test case section contains XML tag Result, for example:

	<Result StatusCode="Pass">Not validated</Result>

	The result code is the value of the StatusCode attribute. Following status
	codes are allowed:

	Pass
	NotSupported
	QualityWarning
	CompatibilityWarning

	Submission package can be verified using `external/vulkancts/scripts/verify_submission.py`
	script. The script takes two arguments: path to extracted submission package
	and path to current mustpass list. For example:

	python external/vulkancts/scripts/verify_submission.py VK_10_Khronos_1/ external/vulkancts/mustpass/1.0.2/vk-default.txt


	Vulkan platform port
	--------------------

	Vulkan support from Platform implementation requires providing
	`getVulkanPlatform()` method in `tcu::Platform` class implementation.

	See `framework/common/tcuPlatform.hpp` and examples in
	`framework/platform/win32/tcuWin32Platform.cpp` and
	`framework/platform/android/tcuAndroidPlatform.cpp`.

	If any WSI extensions are supported, platform port must also implement
	methods for creating native display (`vk::Platform::createWsiDisplay`)
	and window handles (`vk::wsi::Display::createWindow`). Otherwise tests
	under `dEQP-VK.wsi` will fail.


	Null (dummy) driver
	-------------------

	For testing and development purposes it might be useful to be able to run
	tests on dummy Vulkan implementation. One such implementation is provided in
	vkNullDriver.cpp. To use that, implement `vk::Platform::createLibrary()` with
	`vk::createNullDriver()`.


	Validation Layers
	-----------------

	Vulkan CTS framework includes first-party support for validation layers, that
	can be turned on with `--deqp-validation=enable` command line option.

	When validation is turned on, default instance and device will be created with
	validation layers enabled and debug callback is registered to record any
	messages. Debug messages collected during test execution will be included at
	the end of the test case log.

	If any validation errors are found, test result will be set to `InternalError`.

	By default `VK_DEBUG_REPORT_INFORMATION_BIT_EXT` and `_DEBUG_BIT_EXT` messages
	are excluded from the log, but that can be customized by modifying
	`vkt::TestCaseExecutor::deinit()` in `vktTestPackage.cpp`.


	Cherry GUI
	----------

	Vulkan test module can be used with Cherry (GUI for test execution and
	analysis). Cherry is available at
	https://android.googlesource.com/platform/external/cherry. Please follow
	instructions in README to get started.

	Before first launch, and every time test hierarchy has been modified, test
	case list must be refreshed by running:

	python scripts/build_caselists.py path/to/cherry/data

	Cherry must be restarted for the case list update to take effect.