proto/core.rst

Go Protocol buffers
===================

.. _proto_library: https://docs.bazel.build/versions/master/be/protocol-buffer.html#proto_library
.. _default Go plugin: https://github.com/golang/protobuf
.. _common plugins: #predefined-plugins
.. _Go providers: /go/providers.rst
.. _GoLibrary: /go/providers.rst#golibrary
.. _GoSource: /go/providers.rst#gosource
.. _GoArchive: /go/providers.rst#goarchive
.. _Gazelle: https://github.com/bazelbuild/bazel-gazelle
.. _Make variable substitution: https://docs.bazel.build/versions/master/be/make-variables.html#make-var-substitution
.. _Bourne shell tokenization: https://docs.bazel.build/versions/master/be/common-definitions.html#sh-tokenization
.. _gogoprotobuf: https://github.com/gogo/protobuf
.. _compiler.bzl: compiler.bzl
.. _bazelbuild/bazel#3867: https://github.com/bazelbuild/bazel/issues/3867

.. role:: param(kbd)
.. role:: type(emphasis)
.. role:: value(code)
.. |mandatory| replace:: **mandatory value**

rules_go provides rules that generate Go packages from .proto files. These
packages can be imported like regular Go libraries.

.. contents:: :depth: 2

-----

Overview
--------

Protocol buffers are built with the three rules below. ``go_proto_library`` and
``go_proto_compiler`` may be loaded from ``@io_bazel_rules_go//proto:def.bzl``.

* `proto_library`_: This is a Bazel built-in rule. It lists a set of .proto
  files in its ``srcs`` attribute and lists other ``proto_library`` dependencies
  in its ``deps`` attribute. ``proto_library`` rules may be referenced by
  language-specific code generation rules like ``java_proto_library`` and
  ``go_proto_library``.
* `go_proto_library`_: Generates Go code from .proto files using one or more
  proto plugins, then builds that code into a Go library. ``go_proto_library``
  references ``proto_library`` sources via the ``proto`` attribute. They may
  reference other ``go_proto_library`` and ``go_library`` dependencies via the
  ``deps`` attributes.  ``go_proto_library`` rules can be depended on or
  embedded directly by ``go_library`` and ``go_binary``.
* `go_proto_compiler`_: Defines a protoc plugin. By default,
  ``go_proto_library`` generates Go code with the `default Go plugin`_, but
  other plugins can be used by setting the ``compilers`` attribute. A few
  `common plugins`_ are provided in ``@io_bazel_rules_go//proto``.

The ``go_proto_compiler`` rule produces a `GoProtoCompiler`_ provider. If you
need a greater degree of customization (for example, if you don't want to use
protoc), you can implement a compatible rule that returns one of these.

The ``go_proto_library`` rule produces the normal set of `Go providers`_. This
makes it compatible with other Go rules for use in ``deps`` and ``embed``
attributes.

Avoiding conflicts
------------------

When linking programs that depend on protos, care must be taken to ensure that
the same proto isn't registered by more than one package. This may happen if
you depend on a ``go_proto_library`` and a vendored ``go_library`` generated
from the same .proto files. You may see compile-time, link-time, or run-time
errors as a result of this.

There are two main ways to avoid conflicts.

Option 1: Use go_proto_library exclusively
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can avoid proto conflicts by using ``go_proto_library`` to generate code
at build time and avoiding ``go_library`` rules based on pre-generated .pb.go
files.

Gazelle generates rules in this mode by default. When .proto files are present,
it will generate ``go_proto_library`` rules and ``go_library`` rules that embed
them (which are safe to use). Gazelle will automatically exclude .pb.go files
that correspond to .proto files. If you have .proto files belonging to multiple
packages in the same directory, add the following directives to your
root build file:

.. code:: bzl

    # gazelle:proto package
    # gazelle:proto_group go_package

rules_go provides ``go_proto_library`` rules for commonly used proto libraries.
The Well Known Types can be found in the ``@io_bazel_rules_go//proto/wkt``
package. There are implicit dependencies of ``go_proto_library`` rules
that use the default compiler, so they don't need to be written
explicitly in ``deps``. You can also find rules for Google APIs and gRPC in
``@go_googleapis//``. You can list these rules with the commands:

.. code:: bash

    $ bazel query 'kind(go_proto_library, @io_bazel_rules_go//proto/wkt:all)'
    $ bazel query 'kind(go_proto_library, @go_googleapis//...)'

Some commonly used Go libraries, such as ``github.com/golang/protobuf/ptypes``,
depend on the Well Known Types. In order to avoid conflicts when using these
libraries, separate versions of these libraries are provided with
``go_proto_library`` dependencies. Gazelle resolves imports of these libraries
automatically. For example, it will resolve ``ptypes`` as
``@com_github_golang_protobuf//ptypes:go_default_library_gen``.

Option 2: Use pre-generated .pb.go files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

You can also avoid conflicts by generating .pb.go files ahead of time and using
those exclusively instead of using ``go_proto_library``. This may be a better
option for established Go projects that also need to build with ``go build``.

Gazelle can generate rules for projects built in this mode. Add the following
comment to your root build file:

.. code:: bzl

    # gazelle:proto disable_global

This prevents Gazelle from generating ``go_proto_library`` rules. .pb.go files
won't be excluded, and all special cases for imports (such as ``ptypes``) are
disabled.

If you have ``go_repository`` rules in your ``WORKSPACE`` file that may
have protos, you'll also need to add
``build_file_proto_mode = "disable_global"`` to those as well.

.. code:: bzl

    go_repository(
        name = "com_example_some_project",
        importpath = "example.com/some/project",
        tag = "v0.1.2",
        build_file_proto_mode = "disable_global",
    )

A note on vendored .proto files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Bazel can only handle imports in .proto files that are relative to a repository
root directory. This means, for example, if you import ``"foo/bar/baz.proto"``,
that file must be in the directory ``foo/bar``, not
``vendor/example.com/repo/foo/bar``.

If you have proto files that don't conform to this convention, follow the
instructions for using pre-generated .pb.go files above.

The Bazel tracking issue for supporting this is `bazelbuild/bazel#3867`_.

API
---

go_proto_library
~~~~~~~~~~~~~~~~

``go_proto_library`` generates a set of .go files from a set of .proto files
(specified in a ``proto_library`` rule), then builds a Go library from those
files. ``go_proto_library`` can be imported like any ``go_library`` rule.

Providers
^^^^^^^^^

* GoLibrary_
* GoSource_
* GoArchive_

Attributes
^^^^^^^^^^

+---------------------+----------------------+-------------------------------------------------+
| **Name**            | **Type**             | **Default value**                               |
+---------------------+----------------------+-------------------------------------------------+
| :param:`name`       | :type:`string`       | |mandatory|                                     |
+---------------------+----------------------+-------------------------------------------------+
| A unique name for this rule.                                                                 |
|                                                                                              |
| By convention, and in order to interoperate cleanly with Gazelle_, this                      |
| should be a name like ``foo_go_proto``, where ``foo`` is the Go package name                 |
| or the last component of the proto package name (hopefully the same). The                    |
| ``proto_library`` referenced by ``proto`` should be named ``foo_proto``.                     |
+---------------------+----------------------+-------------------------------------------------+
| :param:`proto`      | :type:`label`        | |mandatory|                                     |
+---------------------+----------------------+-------------------------------------------------+
| Points to the ``proto_library`` containing the .proto sources this rule                      |
| should generate code from. Avoid using this argument, use ``protos`` instead.                |
+---------------------+----------------------+-------------------------------------------------+
| :param:`protos`     | :type:`label`        | |mandatory|                                     |
+---------------------+----------------------+-------------------------------------------------+
| List of ``proto_library`` targets containing the .proto sources this rule should generate    |
| code from. This argument should be used instead of ``proto`` argument.                       |
+---------------------+----------------------+-------------------------------------------------+
| :param:`deps`       | :type:`label_list`   | :value:`[]`                                     |
+---------------------+----------------------+-------------------------------------------------+
| List of Go libraries this library depends on directly. Usually, this will be                 |
| a list of ``go_proto_library`` rules that correspond to the ``deps`` of the                  |
| ``proto_library`` rule referenced by ``proto``.                                              |
|                                                                                              |
| Additional dependencies may be added by the proto compiler. For example, the                 |
| default compiler implicitly adds dependencies on the ``go_proto_library``                    |
| rules for the Well Known Types.                                                              |
+---------------------+----------------------+-------------------------------------------------+
| :param:`importpath` | :type:`string`       | |mandatory|                                     |
+---------------------+----------------------+-------------------------------------------------+
| The source import path of this library. Other libraries can import this                      |
| library using this path. This must be specified in ``go_proto_library`` or                   |
| inherited from one of the targets in ``embed``.                                              |
|                                                                                              |
| ``importpath`` must match the import path specified in ``.proto`` files using                |
| ``option go_package``. The option determines how ``.pb.go`` files generated                  |
| for protos importing this proto will import this package.                                    |
+---------------------+----------------------+-------------------------------------------------+
| :param:`importmap`  | :type:`string`       | :value:`""`                                     |
+---------------------+----------------------+-------------------------------------------------+
| The Go package path of this library. This is mostly only visible to the                      |
| compiler and linker, but it may also be seen in stack traces. This may be                    |
| set to prevent a binary from linking multiple packages with the same import                  |
| path, e.g., from different vendor directories.                                               |
+---------------------+----------------------+-------------------------------------------------+
| :param:`embed`      | :type:`label_list`   | :value:`[]`                                     |
+---------------------+----------------------+-------------------------------------------------+
| List of Go libraries that should be combined with this library. The ``srcs``                 |
| and ``deps`` from these libraries will be incorporated into this library when it             |
| is compiled. Embedded libraries must have the same ``importpath`` and                        |
| Go package name.                                                                             |
+---------------------+----------------------+-------------------------------------------------+
| :param:`gc_goopts`  | :type:`string_list`  | :value:`[]`                                     |
+---------------------+----------------------+-------------------------------------------------+
| List of flags to add to the Go compilation command when using the gc                         |
| compiler. Subject to `Make variable substitution`_ and `Bourne shell tokenization`_.         |
+---------------------+----------------------+-------------------------------------------------+
| :param:`compiler`   | :type:`label`        | :value:`None`                                   |
+---------------------+----------------------+-------------------------------------------------+
| Equivalent to ``compilers`` with a single label.                                             |
+---------------------+----------------------+-------------------------------------------------+
| :param:`compilers`  | :type:`label_list`   | :value:`["@io_bazel_rules_go//proto:go_proto"]` |
+---------------------+----------------------+-------------------------------------------------+
| List of rules producing `GoProtoCompiler`_ providers (normally                               |
| `go_proto_compiler`_ rules). This is usually understood to be a list of                      |
| protoc plugins used to generate Go code. See `Predefined plugins`_ for                       |
| some options.                                                                                |
+---------------------+----------------------+-------------------------------------------------+

Example: Basic proto
^^^^^^^^^^^^^^^^^^^^

Suppose you have two .proto files in separate packages: foo/foo.proto and
bar/bar.proto. foo/foo.proto looks like this:

.. code:: proto

  syntax = "proto3";

  option go_package = "example.com/repo/foo";

  import "google/protobuf/any.proto";
  import "bar/bar.proto";

  message Foo {
    bar.Bar x = 1;
    google.protobuf.Any y = 2;
  };

In foo/BUILD.bazel, we need to declare a ``proto_library`` rule that lists
foo.proto in its ``srcs`` attribute. Since we import some other protos, we
also need a label in ``deps`` for each imported package. We will need to
create another ``proto_library`` in bar/BUILD.bazel, but we can use an
existing library for any.proto, since it's one of the Well Known Types.

.. code:: bzl

  proto_library(
      name = "foo_proto",
      srcs = ["foo.proto"],
      deps = [
          "//bar:bar_proto",
          "@com_google_protobuf//:any_proto",
      ],
      visibility = ["//visibility:public"],
  )

In order to these this proto in Go, we need to declare a ``go_proto_library``
that references to ``proto_library`` to be built via the ``proto`` attribute.
Like ``go_library``, an ``importpath`` attribute needs to be declared.
Ideally, this should match the ``option go_package`` declaration in the .proto
file, but this is not required. We also need to list Go packages that the
generated Go code imports in the ``deps`` attributes. Generally, ``deps``
in ``go_proto_library`` will correspond with ``deps`` in ``proto_library``,
but the Well Known Types don't need to be listed (they are added automatically
by the compiler in use).

.. code:: bzl

  load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

  go_proto_library(
      name = "foo_go_proto",
      importpath = "example.com/repo/foo",
      proto = ":foo_proto",
      visibility = ["//visibility:public"],
      deps = ["//bar:bar_go_proto"],
  )

This library can be imported like a regular Go library by other rules.

.. code:: bzl

  load("@io_bazel_rules_go//go:def.bzl", "go_binary")

  go_binary(
      name = "main",
      srcs = ["main.go"],
      deps = ["//foo:foo_go_proto"],
  )

If you need to add additional source files to a package built from protos,
you can do so with a separate ``go_library`` that embeds the
``go_proto_library``.

.. code:: bzl

  load("@io_bazel_rules_go//go:def.bzl", "go_library")

  go_library(
      name = "foo",
      srcs = ["extra.go"],
      embed = [":foo_go_proto"],
      importpath = "example.com/repo/foo",
      visibility = ["//visibility:public"],
  )

For convenience, ``proto_library``, ``go_proto_library``, and ``go_binary``
can all be generated by Gazelle_.

Example: gRPC
^^^^^^^^^^^^^

To compile protos that contain service definitions, just use the ``go_grpc``
plugin.

.. code:: bzl

  load("@io_bazel_rules_go//proto:def.bzl", "go_proto_library")

  proto_library(
      name = "foo_proto",
      srcs = ["foo.proto"],
      visibility = ["//visibility:public"],
  )

  go_proto_library(
      name = "foo_go_proto",
      compilers = ["@io_bazel_rules_go//proto:go_grpc"],
      importpath = "example.com/repo/foo",
      proto = ":foo_proto",
      visibility = ["//visibility:public"],
      deps = ["//bar:bar_go_proto"],
  )

go_proto_compiler
~~~~~~~~~~~~~~~~~

``go_proto_compiler`` describes a plugin for protoc, the proto compiler.
Different plugins will generate different Go code from the same protos.
Compilers may be chosen through the ``compilers`` attribute of
``go_proto_library``. 

Several instances of this rule are listed in `Predefined plugins`_. You will
only need to use this rule directly if you need a plugin which is not there.

Providers
^^^^^^^^^

* GoProtoCompiler_
* GoLibrary_
* GoSource_

Attributes
^^^^^^^^^^

+-----------------------------+----------------------+-----------------------------------------------------+
| **Name**                    | **Type**             | **Default value**                                   |
+-----------------------------+----------------------+-----------------------------------------------------+
| :param:`name`               | :type:`string`       | |mandatory|                                         |
+-----------------------------+----------------------+-----------------------------------------------------+
| A unique name for this rule.                                                                             |
+-----------------------------+----------------------+-----------------------------------------------------+
| :param:`deps`               | :type:`label_list`   | :value:`[]`                                         |
+-----------------------------+----------------------+-----------------------------------------------------+
| List of Go libraries that Go code *generated by* this compiler depends on                                |
| implicitly. Rules in this list must produce the `GoLibrary`_ provider. This                              |
| should contain libraries for the Well Known Types at least.                                              |
+-----------------------------+----------------------+-----------------------------------------------------+
| :param:`options`            | :type:`string_list`  | :value:`[]`                                         |
+-----------------------------+----------------------+-----------------------------------------------------+
| List of command line options to be passed to the compiler. Each option will                              |
| be preceded by ``--option``.                                                                             |
+-----------------------------+----------------------+-----------------------------------------------------+
| :param:`suffix`             | :type:`string`       | :value:`.pb.go`                                     |
+-----------------------------+----------------------+-----------------------------------------------------+
| File name suffix of generated Go files. ``go_proto_compiler`` assumes that                               |
| one Go file will be generated for each input .proto file. Output file names                              |
| will have the .proto suffix removed and this suffix appended. For example,                               |
| ``foo.proto`` will become ``foo.pb.go``.                                                                 |
+-----------------------------+----------------------+-----------------------------------------------------+
| :param:`valid_archive`      | :type:`bool`         | :value:`True`                                       |
+-----------------------------+----------------------+-----------------------------------------------------+
| Whether code generated by this compiler can be compiled into a standalone                                |
| archive file without additional sources.                                                                 |
+-----------------------------+----------------------+-----------------------------------------------------+
| :param:`import_path_option` | :type:`bool`         | :value:`True`                                       |
+-----------------------------+----------------------+-----------------------------------------------------+
| When true, the ``importpath`` attribute from ``go_proto_library`` rules                                  |
| using this compiler will be passed to the compiler on the command line as                                |
| ``--option import_path={}``.                                                                             |
+-----------------------------+----------------------+-----------------------------------------------------+
| :param:`plugin`             | :type:`label`        | :value:`@com_github_golang_protobuf//protoc-gen-go` |
+-----------------------------+----------------------+-----------------------------------------------------+
| The plugin to use with protoc via the ``--plugin`` option. This rule must                                |
| produce an executable file.                                                                              |
+-----------------------------+----------------------+-----------------------------------------------------+

Predefined plugins
------------------

Several ``go_proto_compiler`` rules are predefined in
``@io_bazel_rules_go//proto``.

* ``go_proto``: default plugin from github.com/golang/protobuf.
* ``go_grpc``: default gRPC plugin.
* ``go_proto_validate``: validator plugin from
  github.com/mwitkow/go-proto-validators. Generates ``Validate`` methods.
* gogoprotobuf_ plugins for the variants ``combo``, ``gofast``, ``gogo``,
  ``gogofast``, ``gogofaster``, ``gogoslick``, ``gogotypes``, ``gostring``.
  For each variant, there is a regular version (e.g., ``gogo_proto``) and a
  gRPC version (e.g., ``gogo_grpc``).

Providers
---------

Providers are objects produced by Bazel rules and consumed by other rules that
depend on them. See `Go providers`_ for information about Go providers,
specifically GoLibrary_, GoSource_, and GoArchive_.

GoProtoCompiler
~~~~~~~~~~~~~~~

GoProtoCompiler is the provider returned by the ``go_proto_compiler`` rule and
anything compatible with it. The ``go_proto_library`` rule expects any rule
listed in its ``compilers`` attribute to provide ``GoProtoCompiler``. If the
``go_proto_compiler`` rule doesn't do what you need (e.g., you don't want to
use protoc), you can write a new rule that produces this.

``GoProtoCompiler`` is loaded from ``@io_bazel_rules_go//proto:def.bzl``.

``GoProtoCompiler`` has the fields described below. Additional fields may be
added to pass information to the ``compile`` function. This interface is 
*not final* and may change in the future.

+-----------------------------+-------------------------------------------------+
| **Name**                    | **Type**                                        |
+-----------------------------+-------------------------------------------------+
| :param:`deps`               | :type:`Target list`                             |
+-----------------------------+-------------------------------------------------+
| A list of Go libraries to be added as dependencies to any                     |
| ``go_proto_library`` compiled with this compiler. Each target must provide    |
| GoLibrary_, GoSource_, and GoArchive_. This list should include libraries     |
| for the Well Known Types and anything else considered "standard".             |
+-----------------------------+-------------------------------------------------+
| :param:`compile`            | :type:`Function`                                |
+-----------------------------+-------------------------------------------------+
| A function which declares output files and actions when called. See           |
| `compiler.bzl`_ for details.                                                  |
+-----------------------------+-------------------------------------------------+
| :param:`valid_archive`      | :type:`bool`                                    |
+-----------------------------+-------------------------------------------------+
| Whether the compiler produces a complete Go library. Compilers that just add  |
| methods to structs produced by other compilers will set this to false.        |
+-----------------------------+-------------------------------------------------+

Dependencies
------------

In order to support protocol buffers, rules_go declares the external
repositories listed below in ``go_rules_dependencies()``. These repositories
will only be downloaded if proto rules are used.

* ``@com_google_protobuf (github.com/google/protobuf)``: Well Known Types and
  general proto support.
* ``@com_github_golang_protobuf (github.com/golang/protobuf)``: standard
  Go proto plugin.
* ``@com_github_mwitkow_go_proto_validators
  (github.com/mwitkow/go-proto-validators)``: validator plugin.
* ``@com_github_gogo_protobuf (github.com/gogo/protobuf)``: gogoprotobuf
  plugins.
* ``@org_golang_google_grpc (github.com/grpc/grpc-go``: gRPC support.
* gRPC dependencies

  * ``@org_golang_x_net (golang.org/x/net)``
  * ``@org_golang_x_text (golang.org/x/text)``
  * ``@org_golang_google_genproto (google.golang.org/genproto)``