| \input texinfo @c -*- texinfo -*- |
| @documentencoding UTF-8 |
| |
| @settitle FFmpeg Automated Testing Environment |
| @titlepage |
| @center @titlefont{FFmpeg Automated Testing Environment} |
| @end titlepage |
| |
| @node Top |
| @top |
| |
| @contents |
| |
| @chapter Introduction |
| |
| FATE is an extended regression suite on the client-side and a means |
| for results aggregation and presentation on the server-side. |
| |
| The first part of this document explains how you can use FATE from |
| your FFmpeg source directory to test your ffmpeg binary. The second |
| part describes how you can run FATE to submit the results to FFmpeg's |
| FATE server. |
| |
| In any way you can have a look at the publicly viewable FATE results |
| by visiting this website: |
| |
| @url{http://fate.ffmpeg.org/} |
| |
| This is especially recommended for all people contributing source |
| code to FFmpeg, as it can be seen if some test on some platform broke |
| with their recent contribution. This usually happens on the platforms |
| the developers could not test on. |
| |
| The second part of this document describes how you can run FATE to |
| submit your results to FFmpeg's FATE server. If you want to submit your |
| results be sure to check that your combination of CPU, OS and compiler |
| is not already listed on the above mentioned website. |
| |
| In the third part you can find a comprehensive listing of FATE makefile |
| targets and variables. |
| |
| |
| @chapter Using FATE from your FFmpeg source directory |
| |
| If you want to run FATE on your machine you need to have the samples |
| in place. You can get the samples via the build target fate-rsync. |
| Use this command from the top-level source directory: |
| |
| @example |
| make fate-rsync SAMPLES=fate-suite/ |
| make fate SAMPLES=fate-suite/ |
| @end example |
| |
| The above commands set the samples location by passing a makefile |
| variable via command line. It is also possible to set the samples |
| location at source configuration time by invoking configure with |
| @option{--samples=<path to the samples directory>}. Afterwards you can |
| invoke the makefile targets without setting the @var{SAMPLES} makefile |
| variable. This is illustrated by the following commands: |
| |
| @example |
| ./configure --samples=fate-suite/ |
| make fate-rsync |
| make fate |
| @end example |
| |
| Yet another way to tell FATE about the location of the sample |
| directory is by making sure the environment variable FATE_SAMPLES |
| contains the path to your samples directory. This can be achieved |
| by e.g. putting that variable in your shell profile or by setting |
| it in your interactive session. |
| |
| @example |
| FATE_SAMPLES=fate-suite/ make fate |
| @end example |
| |
| @float NOTE |
| Do not put a '~' character in the samples path to indicate a home |
| directory. Because of shell nuances, this will cause FATE to fail. |
| @end float |
| |
| To use a custom wrapper to run the test, pass @option{--target-exec} to |
| @command{configure} or set the @var{TARGET_EXEC} Make variable. |
| |
| |
| @chapter Submitting the results to the FFmpeg result aggregation server |
| |
| To submit your results to the server you should run fate through the |
| shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs |
| to be invoked with a configuration file as its first argument. |
| |
| @example |
| tests/fate.sh /path/to/fate_config |
| @end example |
| |
| A configuration file template with comments describing the individual |
| configuration variables can be found at @file{doc/fate_config.sh.template}. |
| |
| @ifhtml |
| The mentioned configuration template is also available here: |
| @verbatiminclude fate_config.sh.template |
| @end ifhtml |
| |
| Create a configuration that suits your needs, based on the configuration |
| template. The @env{slot} configuration variable can be any string that is not |
| yet used, but it is suggested that you name it adhering to the following |
| pattern @samp{@var{arch}-@var{os}-@var{compiler}-@var{compiler version}}. The |
| configuration file itself will be sourced in a shell script, therefore all |
| shell features may be used. This enables you to setup the environment as you |
| need it for your build. |
| |
| For your first test runs the @env{fate_recv} variable should be empty or |
| commented out. This will run everything as normal except that it will omit |
| the submission of the results to the server. The following files should be |
| present in $workdir as specified in the configuration file: |
| |
| @itemize |
| @item configure.log |
| @item compile.log |
| @item test.log |
| @item report |
| @item version |
| @end itemize |
| |
| When you have everything working properly you can create an SSH key pair |
| and send the public key to the FATE server administrator who can be contacted |
| at the email address @email{fate-admin@@ffmpeg.org}. |
| |
| Configure your SSH client to use public key authentication with that key |
| when connecting to the FATE server. Also do not forget to check the identity |
| of the server and to accept its host key. This can usually be achieved by |
| running your SSH client manually and killing it after you accepted the key. |
| The FATE server's fingerprint is: |
| |
| @table @samp |
| @item RSA |
| d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 |
| @item ECDSA |
| 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 |
| @end table |
| |
| If you have problems connecting to the FATE server, it may help to try out |
| the @command{ssh} command with one or more @option{-v} options. You should |
| get detailed output concerning your SSH configuration and the authentication |
| process. |
| |
| The only thing left is to automate the execution of the fate.sh script and |
| the synchronisation of the samples directory. |
| |
| @chapter Uploading new samples to the fate suite |
| |
| This is for developers who have an account on the fate suite server. |
| If you upload new samples, please make sure they are as small as possible, |
| space on each client, network bandwidth and so on benefit from smaller test cases. |
| Also keep in mind older checkouts use existing sample files, that means in |
| practice generally do not replace, remove or overwrite files as it likely would |
| break older checkouts or releases. |
| Also all needed samples for a commit should be uploaded, ideally 24 |
| hours, before the push. |
| |
| @example |
| #First update your local samples copy: |
| rsync -vauL --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X fate-suite.ffmpeg.org:/home/samples/fate-suite/ ~/fate-suite |
| |
| #Then do a dry run checking what would be uploaded: |
| rsync -vanL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite |
| |
| #Upload the files: |
| rsync -vaL --no-g --chmod=Dg+s,Duo+x,ug+rw,o+r,o-w,+X ~/fate-suite/ fate-suite.ffmpeg.org:/home/samples/fate-suite |
| @end example |
| |
| |
| @chapter FATE makefile targets and variables |
| |
| @section Makefile targets |
| |
| @table @option |
| @item fate-rsync |
| Download/synchronize sample files to the configured samples directory. |
| |
| @item fate-list |
| Will list all fate/regression test targets. |
| |
| @item fate |
| Run the FATE test suite (requires the fate-suite dataset). |
| @end table |
| |
| @section Makefile variables |
| |
| @table @env |
| @item V |
| Verbosity level, can be set to 0, 1 or 2. |
| @itemize |
| @item 0: show just the test arguments |
| @item 1: show just the command used in the test |
| @item 2: show everything |
| @end itemize |
| |
| @item SAMPLES |
| Specify or override the path to the FATE samples at make time, it has a |
| meaning only while running the regression tests. |
| |
| @item THREADS |
| Specify how many threads to use while running regression tests, it is |
| quite useful to detect thread-related regressions. |
| |
| @item THREAD_TYPE |
| Specify which threading strategy test, either @samp{slice} or @samp{frame}, |
| by default @samp{slice+frame} |
| |
| @item CPUFLAGS |
| Specify CPU flags. |
| |
| @item TARGET_EXEC |
| Specify or override the wrapper used to run the tests. |
| The @env{TARGET_EXEC} option provides a way to run FATE wrapped in |
| @command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets |
| through @command{ssh}. |
| |
| @item GEN |
| Set to @samp{1} to generate the missing or mismatched references. |
| |
| @item HWACCEL |
| Specify which hardware acceleration to use while running regression tests, |
| by default @samp{none} is used. |
| |
| @item KEEP |
| Set to @samp{1} to keep temp files generated by fate test(s) when test is successful. |
| Default is @samp{0}, which removes these files. Files are always kept when a test |
| fails. |
| |
| @end table |
| |
| @section Examples |
| |
| @example |
| make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate |
| @end example |