Google Git
Sign in
fuchsia / third_party / github.com / tensorflow / tensorflow / 8868823c20727507c161b18299731b7790a1b3ca / . / tensorflow / tools / ci_build
tree: 67d6b3e23f087e7a754b1b479259e7c13c9d6f1c [path history] [tgz]
  1. a100/
  2. build_scripts/
  3. builds/
  4. ctpu/
  5. devtoolset/
  6. gpu_build/
  7. horovod/
  8. install/
  9. linux/
  10. nightly_release/
  11. osx/
  12. pi/
  13. presubmit/
  14. protobuf/
  15. rel/
  16. release/
  17. windows/
  18. xla/
  19. build_rbe.sh
  20. ci_build.sh
  21. cloudbuild.yaml
  22. code_link_check.sh
  23. copy_binary.py
  24. cuda-clang.patch
  25. Dockerfile.android
  26. Dockerfile.cmake
  27. Dockerfile.cpu
  28. Dockerfile.cpu-py36
  29. Dockerfile.cpu.arm64
  30. Dockerfile.cpu.ppc64le
  31. Dockerfile.cuda-clang
  32. Dockerfile.custom_op_gpu
  33. Dockerfile.custom_op_ubuntu_16
  34. Dockerfile.custom_op_ubuntu_16_cuda10.0
  35. Dockerfile.custom_op_ubuntu_16_cuda10.1
  36. Dockerfile.custom_op_ubuntu_16_cuda11.2
  37. Dockerfile.debian.bullseye.cpu
  38. Dockerfile.gpu
  39. Dockerfile.gpu.ppc64le
  40. Dockerfile.horovod.gpu
  41. Dockerfile.local-toolchain-ubuntu18.04-manylinux2010
  42. Dockerfile.local-toolchain-ubuntu20.04-manylinux2014
  43. Dockerfile.micro
  44. Dockerfile.pi
  45. Dockerfile.rbe.cpu
  46. Dockerfile.rbe.cuda10.1-cudnn7-ubuntu16.04-manylinux2010
  47. Dockerfile.rbe.cuda10.1-cudnn7-ubuntu16.04-manylinux2010-multipython
  48. Dockerfile.rbe.cuda10.1-cudnn7-ubuntu18.04-manylinux2010-multipython
  49. Dockerfile.rbe.cuda10.2-cudnn7-ubuntu18.04-manylinux2010-multipython
  50. Dockerfile.rbe.cuda11.0-cudnn8-ubuntu18.04-manylinux2010-multipython
  51. Dockerfile.rbe.cuda11.1-cudnn8-ubuntu20.04-manylinux2014-multipython
  52. Dockerfile.rbe.cuda11.2-cudnn8.1-ubuntu20.04-manylinux2014-multipython
  53. Dockerfile.rbe.cuda11.4-cudnn8.2-ubuntu20.04-manylinux2014-multipython
  54. Dockerfile.rbe.cuda11.8-cudnn8.6-ubuntu20.04-manylinux2014-multipython
  55. Dockerfile.rbe.cuda12.0.1-cudnn8.8-ubuntu20.04-manylinux2014-multipython
  56. Dockerfile.rbe.gpu
  57. Dockerfile.rbe.rocm-ubuntu18.04-manylinux2010-multipython
  58. Dockerfile.rbe.rocm-ubuntu20.04-manylinux2014-multipython
  59. Dockerfile.rbe.ubuntu16.04-manylinux2010
  60. Dockerfile.rocm
  61. pep8
  62. pylintrc
  63. README.md
  64. sizetrack_helper.py
  65. update_version.py
tensorflow/tools/ci_build/README.md

TensorFlow Builds

This directory contains all the files and setup instructions to run all the important builds and tests. You can run it yourself!

Run It Yourself

You have two options when running TensorFlow tests locally on your machine. First, using docker, you can run our Continuous Integration (CI) scripts on tensorflow devel images. The other option is to install all TensorFlow dependencies on your machine and run the scripts natively on your system.

Run TensorFlow CI Scripts using Docker

  1. Install Docker following the instructions on the docker website.

  2. Start a container with one of the devel images here: https://hub.docker.com/r/tensorflow/tensorflow/tags/.

  3. Based on your choice of the image, pick one of the scripts under https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/ci_build/linux and run them from the TensorFlow repository root.

Run TensorFlow CI Scripts Natively on your Machine

  1. Follow the instructions at https://www.tensorflow.org/install/source, but stop when you get to the section “Configure the installation”. You do not need to configure the installation to run the CI scripts.

  2. Pick the appropriate OS and python version you have installed, and run the script under tensorflow/tools/ci_build/.

TensorFlow Continuous Integration

To verify that new changes don’t break TensorFlow, we run builds and tests on either Jenkins or a CI system internal to Google.

We can trigger builds and tests on updates to master or on each pull request. Contact one of the repository maintainers to trigger builds on your pull request.

View CI Results

The Pull Request will show if the change passed or failed the checks.

From the pull request, click Show all checks to see the list of builds and tests. Click on Details to see the results from Jenkins or the internal CI system.

Results from Jenkins are displayed in the Jenkins UI. For more information, see the Jenkins documentation.

Results from the internal CI system are displayed in the Build Status UI. In this UI, to see the logs for a failed build:

  • Click on the INVOCATION LOG tab to see the invocation log.

  • Click on the ARTIFACTS tab to see a list of all artifacts, including logs.

  • Individual test logs may be available. To see these logs, from the TARGETS tab, click on the failed target. Then, click on the TARGET LOG tab to see its test log.

    If you’re looking at target that is sharded or a test that is flaky, then the build tool divided the target into multiple shards or ran the test multiple times. Each test log is specific to the shard, run, and attempt. To see a specific log:

    1. Click on the log icon that is on the right next to the shard, run, and attempt number.

    2. In the grid that appears on the right, click on the specific shard, run, and attempt to view its log. You can also type the desired shard, run, or attempt number in the field above its grid.

Third party TensorFlow CI

Mellanox TensorFlow CI

How to start CI
  • Submit special pull request (PR) comment to trigger CI: bot:mlx:test
  • Test session is run automatically.
  • Test results and artifacts (log files) are reported via PR comments
CI Steps

CI includes the following steps: * Build TensorFlow (GPU version) * Run TensorFlow tests: * TF CNN benchmarks (TensorFlow 1.13 and less) * TF models (TensorFlow 2.0): ResNet, synthetic data, NCCL, multi_worker_mirrored distributed strategy

Test Environment

CI is run in the Mellanox lab on a 2-node cluster with the following parameters:

  • Hardware * IB: 1x ConnectX-6 HCA (connected to Mellanox Quantum™ HDR switch) * GPU: 1x Nvidia Tesla K40m * Software * Ubuntu 16.04.6 * Internal stable MLNX_OFED, HPC-X™ and SHARP™ versions
Support (Mellanox)

With any questions/suggestions or in case of issues contact Artem Ryabov.