Using Protobuf Message in Redis with Golang

Data marshaling, migration and backward compatibility

In this article, I want to describe how to use protobuf message for Redis storage, generate protocol buffers messages from code and migrate data to protobuf with backward-compatibility. Golang and Redis are just taken as an example, you can do the same with almost any language and database.

What is Protocol Buffers?

Protobuf is a language-neutral mechanism for serializing structured data. Protobuf uses dynamic message passing to allow for multiple languages to share the same code base and interact with each other. We can use it as a data exchange format, like JSON.

Protobuf most commonly used for gRPC protocol, but you can define your data structure in protobuf and it will be encoded into the binary representation and might be stored anywhere or transferred on top of any other protocols.

JSON vs Protobuf

Protocol Buffers give you built-in data types, fast serialization/deserialization, versioning and backward compatibility. An advantage of JSON might be just higher popularity and lack of implementation Protocol Buffers in some languages.

How to generate the first proto file?

Let’s create a sample project and initialize our main file.

go mod init protobuftest
touch main.go

Then install required dependencies and protobuf code generators.

go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2

As an example, I will create the structure of the Review message, which might be a review for a product in a real-life project.

touch review.proto

Define a latest protocol version and a package name where you plan to use a message struct.

syntax = "proto3";
option go_package=".;storage";
message Review {
  string comment = 1;
}

And generate a protofile with protoc command line tool.

protoc --go_out=. --go-grpc_out=require_unimplemented_servers=false:. --proto_path=. -I=. review.proto

It will create the next review.pb.go file with the Review message struct itself and some other helper functions.

// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
//  protoc-gen-go v1.28.0
//  protoc        v3.19.4
// source: review.proto
package storage
import (
    protoreflect "google.golang.org/protobuf/reflect/protoreflect"
    protoimpl "google.golang.org/protobuf/runtime/protoimpl"
    reflect "reflect"
    sync "sync"
)
const (
    // Verify that this generated code is sufficiently up-to-date.
    _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
    // Verify that runtime/protoimpl is sufficiently up-to-date.
    _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)
type Review struct {
    state         protoimpl.MessageState
    sizeCache     protoimpl.SizeCache
    unknownFields protoimpl.UnknownFields
    Comment string `protobuf:"bytes,1,opt,name=comment,proto3"json:"comment,omitempty"`
}
# ..

You can check the complete file in gist.

How to put and retrieve protobuf data from Redis?

To establish connection with Redis from Go app I’ll use https://github.com/go-redis/redis library.

To run and test the code below, you also needed Redis server installed and running.

package main 
import (
  "context"
  "fmt"
  "github.com/go-redis/redis/v8"
) 
func main() {
  var ctx = context.Background()

  rdb := redis.NewClient(&redis.Options{})
  err := rdb.Set(ctx, "key", "value", 0).Err()
  if err != nil {
    panic(err)
  }
  val, err := rdb.Get(ctx, "key").Result()
  
  if err == redis.Nil {
    fmt.Println("key does not exist")
  } else if err != nil {
    panic(err)
  } else {
    fmt.Println(val) 
  }
}

Now let’s put protobuf Review message into Redis. Before that we need to install proto package.

go get google.golang.org/protobuf/proto

And then marshal Review struct and write into Redis just as a string value.

review := storage.Review{Comment: "Great Product!"}
data, err := proto.Marshal(&review)
if err != nil {
  panic(err)
}
rdb.Set(ctx, "key", data, 0)

To retrieve data, we need to get a value from Redis and unmarshal data to the Review struct.

val, err := rdb.Get(ctx, "key").Result()
if err == redis.Nil {
  fmt.Println("key does not exist")
} else if err != nil {
  panic(err)
}
rev := storage.Review{}
err = proto.Unmarshal([]byte(val), &rev)
if err != nil {
  panic(err)
}
fmt.Println("key", rev.Comment)

Now rerun your app go run main.go and it will print our value from Redis.

Great Product!

And here is all the code above together.

How to extend Protobuf data structure?

It’s pretty easy! Let’s consider a case when you want to extend your Review with rating.

We just need to open our review.proto file and add the rating field.

syntax = "proto3";
option go_package=".;storage";
message Review {
  string comment = 1;
  int32 rating = 2;
}

And then regenerate Review structure.

protoc --go_out=. --go-grpc_out=require_unimplemented_servers=false:. --proto_path=. -I=. review.proto

That’s it. Now you can create Review with a rating field and retrieve existing data with no backward compatibility issues.

storage.Review{Comment: "Great Product!", Rating: 5}

How to migrate from other data types to Protobuf in production?

If you already have a horizontally scaled application with some data in production and want to switch data structure to protobuf, migration will require a few steps depending on your application. Here is acouple of approaches I can recommend:

Dublicate data

Create a new key. Once you finish deployment and old data are gone by TTL, just remove the old code and store data only with a new key in protobuf.

Pros: Easy to implement and finish migration.
Cons: Double the data, might be a significant cost increase depending on app traffic and TTL.

Split the data

Save data partially: old field value in previous key, the protobuf in the new one and then merge the data into protobuf from both keys. After deploying the application, switch feature flag to write data only as protocol buffer in the new key.

Pros: No duplicated data.
Cons: Harder to implement because it requires handling a lot of corner cases.