剑客
关注科技互联网

Go使用gRPC与Protocol Buffers构建高性能API 服务

由于近年来软件的规模日益庞大,API成为了应用程序的的骨干。API作为后台服务为web与移动端提供了强有力的支撑,也用于各平台直接的数据交互,当我们想构建一个基于API的web服务时,我们通常会选择RESTful Api 和 JSON 作为之间的数据交换的方式。RESTful Api 和 JSON确实可以很方便的为移动客户端提供服务,由于Docker为代表的容器技术的火热,微服务架构也随之被很多公司推崇。其中微服务能否实现大规模和高性能非常关键。然而微服务肯定需要一个高性能通信机制与各种服务之间通信。那么最大的问题就是Json是否能够满足基于API提供高性能和扩展性的现代应用程序?然而Json是应用程序直接交换数据最快的数据格式吗?RESTful架构是否能够构建复杂的API?我们能够轻松的通过RESTful架构实现双向的API吗?

现在HTTP/2协议比之前的版本提供了更多的功能,当我们需要满足未来需求构建下一代的API服务时,我们可以考虑使用gRPC 与 protocol buffers。

什么是 Protocol Buffers

Protocol Buffers,是Google公司开发的一种数据描述语言,类似于XML能够将结构化数据序列化,可用于数据存储、通信协议等方面。

它不依赖于语言和平台并且可扩展性极强。现阶段官方支持C++、JAVA、Python等三种编程语言,但可以找到大量的几乎涵盖所有语言的第三方拓展包。通过它,你可以定义你的数据的结构,并生成基于各种语言的代码。这些你定义的数据流可以轻松地在传递并不破坏你已有的程序。并且你也可以更新这些数据而现有的程序也不会受到任何的影响。Protocol Buffers经常被简称为protobuf。

根据以下操作生成protocol buffers文件:

1.从这里 https://github.com/google/protobuf 下载并安装protoc编译器。将protoc二进制文件的位置添加到PATH环境变量中,以便可以从任何位置调用protoc编译器。

2.为go言安装安装protoc插件,运行go get命令为Go安装protoc插件

goget -u github.com/golang/protobuf/protoc-gen-go
 

什么是gRPC

gRPC是一个高性能、通用的开源RPC框架,其由Google主要面向移动应用开发并基于HTTP/2协议标准而设计,基于ProtoBuf(Protocol Buffers)序列化协议开发,且支持众多开发语言。gRPC提供了一种简单的方法来精确地定义服务和为iOS、Android和后台支持服务自动生成可靠性很强的客户端功能库。客户端充分利用高级流和链接功能,从而有助于节省带宽、降低的TCP链接次数、节省CPU使用、和电池寿命。

我们可以从这里 http://www.grpc.io/blog/principles gRPC的动机和设计原则。

Go使用gRPC与Protocol Buffers构建高性能API 服务

使用gRPC和Protocol Buffers的Go语言示例:

首先我们看下Go语言使用gRPC和Protocol Buffers构建示例API项目的文件结构:

Go使用gRPC与Protocol Buffers构建高性能API 服务

为了能够再Go语言中使用gRPC,我们需要使用go get 命令安装gRPC:

gogetgoogle.golang.org/grpc
 

定义Protocol Buffer的消息类型和服务:

关于Protocol Buffer使用我们可以查看 https://developers.google.com/protocol-buffers/docs/proto#generating 这里也推荐一篇译文 http://colobu.com/2015/01/07/Protobuf-language-guide/

我们首先定义一个服务叫做 “ Customer”,他有2个RPC的方法。

syntax = "proto3";
package customer;
 
 
// The Customer service definition.
service Customer {  
  // Get all Customers with filter - A server-to-client streaming RPC.
  rpcGetCustomers(CustomerFilter) returns (streamCustomerRequest) {}
  // Create a new Customer - A simple RPC
  rpcCreateCustomer (CustomerRequest) returns (CustomerResponse) {}
}
 
// Request message for creating a new customer
message CustomerRequest {
  int32id = 1;  // Unique ID number for a Customer.
  string name = 2;
  string email = 3;
  string phone= 4;
  
  message Address {
    string street = 1;
    string city = 2;
    string state = 3;
    string zip = 4;
    bool isShippingAddress = 5; 
  }
 
  repeatedAddressaddresses = 5;
}
 
message CustomerResponse {
  int32id = 1;
  bool success = 2;
}
message CustomerFilter {    
  string keyword = 1;
}

这里有多种gRPC定义RPC方法的方式:

1.简单的的RPC方式,与典型的Request/Response模型在客户端向使用服务端的RPC服务器发送一个请求并等待响应。

2.服务器端数据流RPC方式, 客户端发送请求到服务器,拿到一个流去读取返回的消息序列。 客户端读取返回的流,直到里面没有任何消息。从例子中可以看出,通过在 响应 类型前插 stream关键字,可以指定一个服务器端的流方法。

3.客户端数据流式RPC方式,客户端写入一个消息序列并将其发送到服务器,同样也是使用流。一旦客户端完成写入消息,它等待服务器完成读取返回它的响应。通过在 请求 类型前指定 stream 关键字来指定一个客户端的流方法。

4.双向数据流式RPC方式,是双方使用读写流去发送一个消息序列。两个流独立操作,因此客户端和服务器可以以任意喜欢的顺序读写:比如, 服务器可以在写入响应前等待接收所有的客户端消息,或者可以交替的读取和写入消息,或者其他读写的组合。 每个流中的消息顺序被预留。你可以通过在请求和响应前加 stream 关键字去制定方法的类型

我们上面的Customer 提供了2种RPC方法,一个RPC方法叫做GetCustomers,他是简单的RPC方式,GetCustomers方法是服务器端数据流RPC方式。CreateCustomer创建了一个custom,作为作为Request/Response范型执行,GetCustomers提供了一个custom的列表,服务器已数据流的方式提供数据。

为客户端和服务器生成Go代码:

定义proto文件后,下一步是为gRPC客户端和服务器接口生成源代码,以便编写服务器实现,并根据proto文件中定义的消息类型和服务接口进行客户端调用。 protocol buffer编译器协议与gRPC Go插件一起使用以生成客户端和服务器代码。 从应用程序的根目录,使用Go的gRPC插件运行protoc编译器

protoc -I customer/ customer/customer.proto --go_out=plugins=grpc:customer
 

这将在客户目录中生成一个名为customer.pb.go的Go源文件。 生成的源文件提供了创建服务器和通过RPC进行客户端调用的所有基本代码。

// Code generated by protoc-gen-go.
// source: customer.proto
// DO NOT EDIT!
 
/*
Package customer is a generated protocol buffer package.
It is generated from these files:
customer.proto
It has these top-level messages:
CustomerRequest
CustomerResponse
CustomerFilter
*/
package customer
 
importproto "github.com/golang/protobuf/proto"
importfmt "fmt"
importmath "math"
 
import (
 context "golang.org/x/net/context"
 grpc "google.golang.org/grpc"
)
 
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = fmt.Errorf
var _ = math.Inf
 
// This is a compile-time assertion to ensure that this generated file
// is compatible with the proto package it is being compiled against.
// A compilation error at this line likely means your copy of the
// proto package needs to be updated.
const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
 
// Request message for creating a new customer
type CustomerRequest struct {
 Id        int32                      `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
 Name      string                    `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
 Email    string                    `protobuf:"bytes,3,opt,name=email" json:"email,omitempty"`
 Phone    string                    `protobuf:"bytes,4,opt,name=phone" json:"phone,omitempty"`
 Addresses []*CustomerRequest_Address `protobuf:"bytes,5,rep,name=addresses" json:"addresses,omitempty"`
}
 
func (m *CustomerRequest) Reset()                    { *m = CustomerRequest{} }
func (m *CustomerRequest) String() string            { return proto.CompactTextString(m) }
func (*CustomerRequest) ProtoMessage()              {}
func (*CustomerRequest) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{0} }
 
func (m *CustomerRequest) GetAddresses() []*CustomerRequest_Address {
 if m != nil {
 return m.Addresses
 }
 return nil
}
 
type CustomerRequest_Address struct {
 Street            string `protobuf:"bytes,1,opt,name=street" json:"street,omitempty"`
 City              string `protobuf:"bytes,2,opt,name=city" json:"city,omitempty"`
 State            string `protobuf:"bytes,3,opt,name=state" json:"state,omitempty"`
 Zip              string `protobuf:"bytes,4,opt,name=zip" json:"zip,omitempty"`
 IsShippingAddressbool  `protobuf:"varint,5,opt,name=isShippingAddress" json:"isShippingAddress,omitempty"`
}
 
func (m *CustomerRequest_Address) Reset()                    { *m = CustomerRequest_Address{} }
func (m *CustomerRequest_Address) String() string            { return proto.CompactTextString(m) }
func (*CustomerRequest_Address) ProtoMessage()              {}
func (*CustomerRequest_Address) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{0, 0} }
 
type CustomerResponse struct {
 Id      int32 `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
 Successbool  `protobuf:"varint,2,opt,name=success" json:"success,omitempty"`
}
 
func (m *CustomerResponse) Reset()                    { *m = CustomerResponse{} }
func (m *CustomerResponse) String() string            { return proto.CompactTextString(m) }
func (*CustomerResponse) ProtoMessage()              {}
func (*CustomerResponse) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{1} }
 
type CustomerFilter struct {
 Keywordstring `protobuf:"bytes,1,opt,name=keyword" json:"keyword,omitempty"`
}
 
func (m *CustomerFilter) Reset()                    { *m = CustomerFilter{} }
func (m *CustomerFilter) String() string            { return proto.CompactTextString(m) }
func (*CustomerFilter) ProtoMessage()              {}
func (*CustomerFilter) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{2} }
 
funcinit() {
 proto.RegisterType((*CustomerRequest)(nil), "customer.CustomerRequest")
 proto.RegisterType((*CustomerRequest_Address)(nil), "customer.CustomerRequest.Address")
 proto.RegisterType((*CustomerResponse)(nil), "customer.CustomerResponse")
 proto.RegisterType((*CustomerFilter)(nil), "customer.CustomerFilter")
}
 
// Reference imports to suppress errors if they are not otherwise used.
var _ context.Context
var _ grpc.ClientConn
 
// This is a compile-time assertion to ensure that this generated file
// is compatible with the grpc package it is being compiled against.
const _ = grpc.SupportPackageIsVersion3
 
// Client API for Customer service
 
type CustomerClient interface {
 // Get all Customers with filter - A server-to-client streaming RPC.
 GetCustomers(ctxcontext.Context, in *CustomerFilter, opts ...grpc.CallOption) (Customer_GetCustomersClient, error)
 // Create a new Customer - A simple RPC
 CreateCustomer(ctxcontext.Context, in *CustomerRequest, opts ...grpc.CallOption) (*CustomerResponse, error)
}
 
type customerClient struct {
 cc *grpc.ClientConn
}
 
funcNewCustomerClient(cc *grpc.ClientConn) CustomerClient {
 return &customerClient{cc}
}
 
func (c *customerClient) GetCustomers(ctxcontext.Context, in *CustomerFilter, opts ...grpc.CallOption) (Customer_GetCustomersClient, error) {
 stream, err := grpc.NewClientStream(ctx, &_Customer_serviceDesc.Streams[0], c.cc, "/customer.Customer/GetCustomers", opts...)
 if err != nil {
 return nil, err
 }
 x := &customerGetCustomersClient{stream}
 if err := x.ClientStream.SendMsg(in); err != nil {
 return nil, err
 }
 if err := x.ClientStream.CloseSend(); err != nil {
 return nil, err
 }
 return x, nil
}
 
type Customer_GetCustomersClient interface {
 Recv() (*CustomerRequest, error)
 grpc.ClientStream
}
 
type customerGetCustomersClient struct {
 grpc.ClientStream
}
 
func (x *customerGetCustomersClient) Recv() (*CustomerRequest, error) {
 m := new(CustomerRequest)
 if err := x.ClientStream.RecvMsg(m); err != nil {
 return nil, err
 }
 return m, nil
}
 
func (c *customerClient) CreateCustomer(ctxcontext.Context, in *CustomerRequest, opts ...grpc.CallOption) (*CustomerResponse, error) {
 out := new(CustomerResponse)
 err := grpc.Invoke(ctx, "/customer.Customer/CreateCustomer", in, out, c.cc, opts...)
 if err != nil {
 return nil, err
 }
 return out, nil
}
 
// Server API for Customer service
 
type CustomerServer interface {
 // Get all Customers with filter - A server-to-client streaming RPC.
 GetCustomers(*CustomerFilter, Customer_GetCustomersServer) error
 // Create a new Customer - A simple RPC
 CreateCustomer(context.Context, *CustomerRequest) (*CustomerResponse, error)
}
 
funcRegisterCustomerServer(s *grpc.Server, srvCustomerServer) {
 s.RegisterService(&_Customer_serviceDesc, srv)
}
 
func _Customer_GetCustomers_Handler(srv interface{}, streamgrpc.ServerStream) error {
 m := new(CustomerFilter)
 if err := stream.RecvMsg(m); err != nil {
 return err
 }
 return srv.(CustomerServer).GetCustomers(m, &customerGetCustomersServer{stream})
}
 
type Customer_GetCustomersServer interface {
 Send(*CustomerRequest) error
 grpc.ServerStream
}
 
type customerGetCustomersServer struct {
 grpc.ServerStream
}
 
func (x *customerGetCustomersServer) Send(m *CustomerRequest) error {
 return x.ServerStream.SendMsg(m)
}
 
func _Customer_CreateCustomer_Handler(srv interface{}, ctxcontext.Context, dec func(interface{}) error, interceptorgrpc.UnaryServerInterceptor) (interface{}, error) {
 in := new(CustomerRequest)
 if err := dec(in); err != nil {
 return nil, err
 }
 if interceptor == nil {
 return srv.(CustomerServer).CreateCustomer(ctx, in)
 }
 info := &grpc.UnaryServerInfo{
 Server:    srv,
 FullMethod: "/customer.Customer/CreateCustomer",
 }
 handler := func(ctxcontext.Context, req interface{}) (interface{}, error) {
 return srv.(CustomerServer).CreateCustomer(ctx, req.(*CustomerRequest))
 }
 return interceptor(ctx, in, info, handler)
}
 
var _Customer_serviceDesc = grpc.ServiceDesc{
 ServiceName: "customer.Customer",
 HandlerType: (*CustomerServer)(nil),
 Methods: []grpc.MethodDesc{
 {
 MethodName: "CreateCustomer",
 Handler:    _Customer_CreateCustomer_Handler,
 },
 },
 Streams: []grpc.StreamDesc{
 {
 StreamName:    "GetCustomers",
 Handler:      _Customer_GetCustomers_Handler,
 ServerStreams: true,
 },
 },
 Metadata: fileDescriptor0,
}
 
funcinit() { proto.RegisterFile("customer.proto", fileDescriptor0) }
 
var fileDescriptor0 = []byte{
 // 326 bytes of a gzipped FileDescriptorProto
 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0x74, 0x92, 0xef, 0x4a, 0xc3, 0x30,
 0x10, 0xc0, 0x97, 0x6e, 0xdd, 0x9f, 0x53, 0xea, 0x0c, 0x22, 0xb1, 0x9f, 0x6a, 0x3f, 0x15, 0x91,
 0x21, 0xf3, 0xab, 0x20, 0x32, 0x70, 0xf8, 0xb5, 0x3e, 0x41, 0x6d, 0x0f, 0x17, 0xdc, 0xda, 0x9a,
 0xcb, 0x90, 0xf9, 0x0a, 0xbe, 0x83, 0xcf, 0xe0, 0x23, 0x4a, 0xd2, 0x66, 0x03, 0xe7, 0xbe, 0xdd,
 0xef, 0x72, 0x77, 0xf9, 0xe5, 0x08, 0x04, 0xf9, 0x9a, 0x74, 0xb5, 0x42, 0x35, 0xa9, 0x55, 0xa5,
 0x2b, 0x3e, 0x74, 0x1c, 0xff, 0x78, 0x70, 0x32, 0x6b, 0x21, 0xc5, 0xf7, 0x35, 0x92, 0xe6, 0x01,
 0x78, 0xb2, 0x10, 0x2c, 0x62, 0x89, 0x9f, 0x7a, 0xb2, 0xe0, 0x1c, 0x7a, 0x65, 0xb6, 0x42, 0xe1,
 0x45, 0x2c, 0x19, 0xa5, 0x36, 0xe6, 0x67, 0xe0, 0xe3, 0x2a, 0x93, 0x4b, 0xd1, 0xb5, 0xc9, 0x06,
 0x4c, 0xb6, 0x5e, 0x54, 0x25, 0x8a, 0x5e, 0x93, 0xb5, 0xc0, 0xef, 0x61, 0x94, 0x15, 0x85, 0x42,
 0x22, 0x24, 0xe1, 0x47, 0xdd, 0xe4, 0x68, 0x7a, 0x39, 0xd9, 0x1a, 0xfd, 0xb9, 0x7d, 0xf2, 0xd0,
 0x94, 0xa6, 0xbb, 0x9e, 0xf0, 0x8b, 0xc1, 0xa0, 0x4d, 0xf3, 0x73, 0xe8, 0x93, 0x56, 0x88, 0xda,
 0x0a, 0x8e, 0xd2, 0x96, 0x8c, 0x64, 0x2e, 0xf5, 0xc6, 0x49, 0x9a, 0xd8, 0xe8, 0x90, 0xce, 0x34,
 0x3a, 0x49, 0x0b, 0x7c, 0x0c, 0xdd, 0x4f, 0x59, 0xb7, 0x8a, 0x26, 0xe4, 0xd7, 0x70, 0x2a, 0xe9,
 0x79, 0x21, 0xeb, 0x5a, 0x96, 0xaf, 0xed, 0x45, 0xc2, 0x8f, 0x58, 0x32, 0x4c, 0xf7, 0x0f, 0xe2,
 0x3b, 0x18, 0xef, 0x9c, 0xa9, 0xae, 0x4a, 0xc2, 0xbd, 0x95, 0x09, 0x18, 0xd0, 0x3a, 0xcf, 0xcd,
 0x1c, 0xcf, 0xce, 0x71, 0x18, 0x5f, 0x41, 0xe0, 0xba, 0x1f, 0xe5, 0x52, 0xa3, 0x32, 0xb5, 0x6f,
 0xb8, 0xf9, 0xa8, 0x54, 0xd1, 0x3e, 0xc9, 0xe1, 0xf4, 0x9b, 0xc1, 0xd0, 0x15, 0xf3, 0x39, 0x1c,
 0xcf, 0x51, 0x3b, 0x24, 0x2e, 0xf6, 0x57, 0xd8, 0x0c, 0x0c, 0x2f, 0x0e, 0x2e, 0x37, 0xee, 0xdc,
 0x30, 0xfe, 0x04, 0xc1, 0x4c, 0x61, 0xa6, 0x71, 0x3b, 0xfa, 0x70, 0x43, 0x18, 0xfe, 0x77, 0xd4,
 0x3c, 0x3a, 0xee, 0xbc, 0xf4, 0xed, 0x77, 0xba, 0xfd, 0x0d, 0x00, 0x00, 0xff, 0xff, 0xde, 0x91,
 0xd3, 0x62, 0x60, 0x02, 0x00, 0x00,
}

创建gRPC Server :

package main
 
import (
 "log"
 "net"
 "strings"
 
 "golang.org/x/net/context"
 "google.golang.org/grpc"
 
 pb "github.com/shijuvar/go-recipes/grpc/customer"
)
 
const (
 port = ":50051"
)
 
// server is used to implement customer.CustomerServer.
type server struct {
 savedCustomers []*pb.CustomerRequest
}
 
// CreateCustomer creates a new Customer
func (s *server) CreateCustomer(ctxcontext.Context, in *pb.CustomerRequest) (*pb.CustomerResponse, error) {
 s.savedCustomers = append(s.savedCustomers, in)
 return &pb.CustomerResponse{Id: in.Id, Success: true}, nil
}
 
// GetCustomers returns all customers by given filter
func (s *server) GetCustomers(filter *pb.CustomerFilter, streampb.Customer_GetCustomersServer) error {
 for _, customer := range s.savedCustomers {
 if filter.Keyword != "" {
 if !strings.Contains(customer.Name, filter.Keyword) {
 continue
 }
 }
 if err := stream.Send(customer); err != nil {
 return err
 }
 }
 return nil
}
 
func main() {
 lis, err := net.Listen("tcp", port)
 if err != nil {
 log.Fatalf("failed to listen: %v", err)
 }
 // Creates a new gRPC server
 s := grpc.NewServer()
 pb.RegisterCustomerServer(s, &server{})
 s.Serve(lis)
}

创建gRPC客户端:

package main
 
import (
 "io"
 "log"
 
 "golang.org/x/net/context"
 "google.golang.org/grpc"
 
 pb "github.com/shijuvar/go-recipes/grpc/customer"
)
 
const (
 address = "localhost:50051"
)
 
// createCustomer calls the RPC method CreateCustomer of CustomerServer
func createCustomer(clientpb.CustomerClient, customer *pb.CustomerRequest) {
 resp, err := client.CreateCustomer(context.Background(), customer)
 if err != nil {
 log.Fatalf("Could not create Customer: %v", err)
 }
 if resp.Success {
 log.Printf("A new Customer has been added with id: %d", resp.Id)
 }
}
 
// getCustomers calls the RPC method GetCustomers of CustomerServer
func getCustomers(clientpb.CustomerClient, filter *pb.CustomerFilter) {
 // calling the streaming API
 stream, err := client.GetCustomers(context.Background(), filter)
 if err != nil {
 log.Fatalf("Error on get customers: %v", err)
 }
 for {
 // Receiving the stream of data
 customer, err := stream.Recv()
 if err == io.EOF {
 break
 }
 if err != nil {
 log.Fatalf("%v.GetCustomers(_) = _, %v", client, err)
 }
 log.Printf("Customer: %v", customer)
 }
}
func main() {
 // Set up a connection to the gRPC server.
 conn, err := grpc.Dial(address, grpc.WithInsecure())
 if err != nil {
 log.Fatalf("did not connect: %v", err)
 }
 defer conn.Close()
 // Creates a new CustomerClient
 client := pb.NewCustomerClient(conn)
 
 customer := &pb.CustomerRequest{
 Id:    101,
 Name:  "Shiju Varghese",
 Email: "shiju@xyz.com",
 Phone: "732-757-2923",
 Addresses: []*pb.CustomerRequest_Address{
 &pb.CustomerRequest_Address{
 Street:            "1 Mission Street",
 City:              "San Francisco",
 State:            "CA",
 Zip:              "94105",
 IsShippingAddress: false,
 },
 &pb.CustomerRequest_Address{
 Street:            "Greenfield",
 City:              "Kochi",
 State:            "KL",
 Zip:              "68356",
 IsShippingAddress: true,
 },
 },
 }
 
 // Create a new customer
 createCustomer(client, customer)
 
 customer = &pb.CustomerRequest{
 Id:    102,
 Name:  "Irene Rose",
 Email: "irene@xyz.com",
 Phone: "732-757-2924",
 Addresses: []*pb.CustomerRequest_Address{
 &pb.CustomerRequest_Address{
 Street:            "1 Mission Street",
 City:              "San Francisco",
 State:            "CA",
 Zip:              "94105",
 IsShippingAddress: true,
 },
 },
 }
 
 // Create a new customer
 createCustomer(client, customer)
 // Filter with an empty Keyword
 filter := &pb.CustomerFilter{Keyword: ""}
 getCustomers(client, filter)
}

Source Code:

https://github.com/shijuvar/go-recipes/tree/master/grpc

参考文献:

https://medium.com/@shijuvar/building-high-performance-apis-in-go-using-grpc-and-protocol-buffers-2eda5b80771b#.qmqhzdhom

Go使用gRPC与Protocol Buffers构建高性能API 服务

分享到:更多 ()

评论 抢沙发

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址