Skip to main content

Protocol Buffers

It is known that Cosmos SDK uses protocol buffers extensively, this document is meant to provide a guide on how it is used in the cosmos-sdk.

To generate the proto file, the Cosmos SDK uses a docker image, this image is provided to all to use as well. The latest version is ghcr.io/cosmos/proto-builder:0.12.x

Below is the example of the Cosmos SDK's commands for generating, linting, and formatting protobuf files that can be reused in any applications makefile.

https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/Makefile#L411-L432

The script used to generate the protobuf files can be found in the scripts/ directory.

https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/scripts/protocgen.sh

Buf​

Buf is a protobuf tool that abstracts the needs to use the complicated protoc toolchain on top of various other things that ensure you are using protobuf in accordance with the majority of the ecosystem. Within the cosmos-sdk repository there are a few files that have a buf prefix. Lets start with the top level and then dive into the various directories.

Workspace​

At the root level directory a workspace is defined using buf workspaces. This helps if there are one or more protobuf containing directories in your project.

Cosmos SDK example:

https://github.com/cosmos/cosmos-sdk/blob/main/buf.work.yaml#L6-L9

Proto Directory​

Next is the proto/ directory where all of our protobuf files live. In here there are many different buf files defined each serving a different purpose.

├── README.md
├── buf.gen.gogo.yaml
├── buf.gen.pulsar.yaml
├── buf.gen.swagger.yaml
├── buf.lock
├── buf.md
├── buf.yaml
├── cosmos
└── tendermint

The above diagram all the files and directories within the Cosmos SDK proto/ directory.

buf.gen.gogo.yaml​

buf.gen.gogo.yaml defines how the protobuf files should be generated for use with in the module. This file uses gogoproto, a separate generator from the google go-proto generator that makes working with various objects more ergonomic, and it has more performant encode and decode steps

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.gen.gogo.yaml#L1-L9
tip

Example of how to define gen files can be found here

buf.gen.pulsar.yaml​

buf.gen.pulsar.yaml defines how protobuf files should be generated using the new golang apiv2 of protobuf. This generator is used instead of the google go-proto generator because it has some extra helpers for Cosmos SDK applications and will have more performant encode and decode than the google go-proto generator. You can follow the development of this generator here.

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.gen.pulsar.yaml#L1-L18
tip

Example of how to define gen files can be found here

buf.gen.swagger.yaml​

buf.gen.swagger.yaml generates the swagger documentation for the query and messages of the chain. This will only define the REST API end points that were defined in the query and msg servers. You can find examples of this here

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.gen.swagger.yaml#L1-L6
tip

Example of how to define gen files can be found here

buf.lock​

This is an autogenerated file based off the dependencies required by the .gen files. There is no need to copy the current one. If you depend on cosmos-sdk proto definitions a new entry for the Cosmos SDK will need to be provided. The dependency you will need to use is buf.build/cosmos/cosmos-sdk.

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.lock#L1-L16

buf.yaml​

buf.yaml defines the name of your package, which breakage checker to use and how to lint your protobuf files.

https://github.com/cosmos/cosmos-sdk/blob/main/proto/buf.yaml#L1-L24

We use a variety of linters for the Cosmos SDK protobuf files. The repo also checks this in ci.

A reference to the github actions can be found here

https://github.com/cosmos/cosmos-sdk/blob/main/.github/workflows/proto.yml#L1-L32