Using Java Protocol Buffers in JRuby

June 27, 2013

At Square, we use Protocol Buffers for communication between services. Ruby has good native Protocol Buffer support, but by using the compiled Java classes, we can leverage all the benefits of any extensions and infrastructure relying on those extensions. We'll demonstrate how we utilize Java-backed Protocol Buffers while still enjoying idiomatic Ruby for Protocol Buffer construction.

Protocol Buffers

We have added support to our JRuby-Java integration framework (Minecart) to seamlessly speak Square's internal RPC protocol. If you are unfamiliar with the Protocol Buffer syntax, and you'd like to know more, please have a quick read. Protocol Buffers are a strongly typed message format which serialize to a binary representation. This results in an efficient wire transfer and documents the API. Furthermore, extensions can be built which add functionality such as idempotence, request chaining, and even behavioral requirements such as authentication for service endpoints.

To show off Protocol Buffers and how they work in Minecart, let's implement part of a pizza ordering service. To start we'll look at the service definition.

service OrderingService {
  rpc PlaceOrder ( OrderRequest ) returns ( OrderResponse )
}

Great. This tells us that we have a service which can handle the action 'PlaceOrder'. It requires an 'OrderRequest' message and returns an 'OrderResponse'.

message OrderRequest {
  optional Person person = 1;
  optional Address address = 2;
  repeated LineItem line_items = 3;
}

message OrderResponse {
  optional string order_id = 1;
  optional bool success = 2;
  optional string error_message = 3;
}

Here we have the filled in OrderRequest and OrderResponse messages. These contain several nested messages which are defined similarly. You can find the filled out service definition here.

To get this service running in JRuby we're going to install protoc, download the pizza service, download the Java dependencies, and then compile, install, and run.

brew install protobuf-c
mkdir protos
curl https://gist.github.com/hassox/d4b3d65c9e00c116f208/raw > protos/pizza.proto
protoc --java_out=. --proto_path=protos/ protos/pizza.proto
curl -O http://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.4.1/protobuf-java-2.4.1.jar
mkdir target
javac -classpath protobuf-java-2.4.1.jar com/pizza/Pizza.java -d target
# For the future...
curl https://gist.github.com/hassox/e5f27688a783c1b43e7e/raw > hash_to_proto_converter.rb

BOOM! Now you we have access to the raw Java Protocol Buffers classes, give it a shot!

#JRuby console
require 'protobuf-java-2.4.1'
$CLASSPATH << File.join(Dir.pwd, "target")

order_builder = com.pizza.Pizza::OrderRequest.newBuilder

person_builder = order_builder.getPersonBuilder
person_builder.setName   "Homer J. Simpson"
person_builder.setEmail  "homer@example.com"
person_builder.setPhone  "555 1234"

address_builder = order_builder.getAddressBuilder
address_builder.setStreet1      "742 Evergreen Terrace"
address_builder.setCity         "Springfield"
address_builder.setState        "Unknown"
address_builder.setPostalCode   "12345"
address_builder.setCountry      "USA"

line_item_builder = com.pizza.Pizza::LineItem.newBuilder
[
  {name: "Peperroni Pizza", quantity: 1},
  {name: "Cheese Pizza", quantity: 8}
].each_with_index do |item, idx|
  line_item_builder.setItem  item[:name]
  line_item_builder.setQuantity  item[:quantity]
  order_builder.addLineItems(line_item_builder.build)
end

order = order_builder.build

You'll notice that we have to set each field individually, and use a builder for nested types and enum values. Though we have gained access to any Protocol Buffer extensions, the cost is noticeable when compared to native Ruby Protocol Buffer support.

  # Alternative Ruby example using Ruby support for protos
  Pizza::OrderRequest.new(
    person: {name: "Homer J. Simpson", email: "homer@example.com"}
    ...
  )

Enter the Java proto to Ruby Hash converter

To bring back delight and ease of use, we built a helper to interoperate Java Protocol Buffers and Ruby hashes. Note: we downloaded the converter previously during the setup so you should have access to it.

# JRuby
require 'protobuf-java-2.4.1'
$CLASSPATH << File.join(Dir.pwd, "target")

require 'hash_to_proto_converter'

proto = Minecart::HashProtoBuilder.hash_to_proto(
    com.pizza.Pizza::Person,
    name: 'Fred',
    email: 'fred@example.com',
    phone: '555 1234'
  )

Minecart::HashProtoBuilder.hash_from_proto(proto)
# => { name: "Fred", email: "fred@example.com", phone: "555 1234"}

As a developer you get to specify the contract, have a consistent view of the APIs and keep terse hash syntax.

A peek behind the curtain: this utility has allowed us to write Ruby handlers which tie seamlessly into our Java request lifecycle and infrastructure.

class OrderingService < Minecart::Service(com.pizza.Pizza::OrderingService)

  # Dispatch is mapped by snake cased name defined in the service proto
  def place_order
    params # the order Protocol Buffer Message, deserialized as a nested hash
    # snip… process the order
    { order_id: order.id, success: true } # return a hash that maps to the response message
  end
end

Beautiful!

This post is part of a series, which highlights discoveries and insights found while integrating Ruby with our robust Java stack.

Engineer, Father, Husband, Australian!
Co-Inventor of Square Wallet, Engineer

Comments

Get support help at squareup.com/support. We'll delete off-topic comments.