Using Protobuf Message in Redis with Golang
Data marshaling, migration and backward compatibility
In this article, I want to describe how to use protobuf message for Redis storage, generate protocol buffers messages from code and migrate data to protobuf with backward-compatibility. Golang and Redis are just taken as an example, you can do the same with almost any language and database.
What is Protocol Buffers?
Protobuf is a language-neutral mechanism for serializing structured data. Protobuf uses dynamic message passing to allow for multiple languages to share the same code base and interact with each other. We can use it as a data exchange format, like JSON.
Protobuf most commonly used for gRPC protocol, but you can define your data structure in protobuf and it will be encoded into the binary representation and might be stored anywhere or transferred on top of any other protocols.
JSON vs Protobuf
Protocol Buffers give you built-in data types, fast serialization/deserialization, versioning and backward compatibility. An advantage of JSON might be just higher popularity and lack of implementation Protocol Buffers in some languages.
How to generate the first proto file?
Let’s create a sample project and initialize our main file.
go mod init protobuftest
touch main.go
Then install required dependencies and protobuf code generators.
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2
As an example, I will create the structure of the Review
message, which might be a review for a product in a real-life project.
touch review.proto
Define a latest protocol version and a package name where you plan to use a message struct.
syntax = "proto3";option go_package=".;storage";message Review {
string comment = 1;
}
And generate a protofile with protoc
command line tool.
protoc --go_out=. --go-grpc_out=require_unimplemented_servers=false:. --proto_path=. -I=. review.proto
It will create the next review.pb.go
file with the Review
message struct itself and some other helper functions.
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.28.0
// protoc v3.19.4
// source: review.protopackage storageimport (
protoreflect "google.golang.org/protobuf/reflect/protoreflect"
protoimpl "google.golang.org/protobuf/runtime/protoimpl"
reflect "reflect"
sync "sync"
)const (
// Verify that this generated code is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
// Verify that runtime/protoimpl is sufficiently up-to-date.
_ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
)type Review struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields Comment string `protobuf:"bytes,1,opt,name=comment,proto3"json:"comment,omitempty"`
}# ..
You can check the complete file in gist.
How to put and retrieve protobuf data from Redis?
To establish connection with Redis from Go app I’ll use https://github.com/go-redis/redis library.
To run and test the code below, you also needed Redis server installed and running.
package main import (
"context"
"fmt"
"github.com/go-redis/redis/v8"
) func main() {
var ctx = context.Background()
rdb := redis.NewClient(&redis.Options{})
err := rdb.Set(ctx, "key", "value", 0).Err() if err != nil {
panic(err)
} val, err := rdb.Get(ctx, "key").Result()
if err == redis.Nil {
fmt.Println("key does not exist")
} else if err != nil {
panic(err)
} else {
fmt.Println(val)
}
}
Now let’s put protobuf Review
message into Redis. Before that we need to install proto package.
go get google.golang.org/protobuf/proto
And then marshal Review
struct and write into Redis just as a string value.
review := storage.Review{Comment: "Great Product!"}data, err := proto.Marshal(&review)if err != nil {
panic(err)
}rdb.Set(ctx, "key", data, 0)
To retrieve data, we need to get a value from Redis and unmarshal data to the Review
struct.
val, err := rdb.Get(ctx, "key").Result()if err == redis.Nil {
fmt.Println("key does not exist")
} else if err != nil {
panic(err)
}rev := storage.Review{}
err = proto.Unmarshal([]byte(val), &rev)if err != nil {
panic(err)
}fmt.Println("key", rev.Comment)
Now rerun your app go run main.go
and it will print our value from Redis.
Great Product!
And here is all the code above together.
How to extend Protobuf data structure?
It’s pretty easy! Let’s consider a case when you want to extend your Review
with rating.
We just need to open our review.proto
file and add the rating field.
syntax = "proto3";option go_package=".;storage";message Review {
string comment = 1;
int32 rating = 2;
}
And then regenerate Review
structure.
protoc --go_out=. --go-grpc_out=require_unimplemented_servers=false:. --proto_path=. -I=. review.proto
That’s it. Now you can create Review
with a rating field and retrieve existing data with no backward compatibility issues.
storage.Review{Comment: "Great Product!", Rating: 5}
How to migrate from other data types to Protobuf in production?
If you already have a horizontally scaled application with some data in production and want to switch data structure to protobuf, migration will require a few steps depending on your application. Here is acouple of approaches I can recommend:
Dublicate data
Create a new key. Once you finish deployment and old data are gone by TTL, just remove the old code and store data only with a new key in protobuf.
Pros: Easy to implement and finish migration.
Cons: Double the data, might be a significant cost increase depending on app traffic and TTL.
Split the data
Save data partially: old field value in previous key, the protobuf in the new one and then merge the data into protobuf from both keys. After deploying the application, switch feature flag to write data only as protocol buffer in the new key.
Pros: No duplicated data.
Cons: Harder to implement because it requires handling a lot of corner cases.