Oracle announced today that it has open sourced GraphPipe, a tool that simplifies machine learning model deployment. GraphPipe offers a standard, high-performance protocol for transmitting tensor data over a network, along with simple implementations of clients and servers. Using the tool, teams can deploy and query models built in TensorFlow, PyTorch, mxnet, CNTK, or Caffe2.
“In the enterprise, machine-learning models are often trained individually and deployed using bespoke techniques,” wrote Cloud Development Architect Vish Abrams in a post on the Oracle Developers Blog. “This impacts an organization's ability to derive value from its machine learning efforts. If marketing wants to use a model produced by the finance group, they will have to write custom clients to interact with the model. If the model becomes popular sales wants to use it as well, the custom deployment may crack under the load.”
GraphPipe is designed to solve three main issues that prevent teams from deploying models built with popular frameworks:
- There is no standard for model serving APIs, so teams must use what is provided by the framework used to build the model, such as protocol buffers or custom JSON. Consequently, a business application will generally need a bespoke client just to talk to the deployed model. If a team uses multiple frameworks for a project, they will have to write custom code to combine them.
- Building a model server can be incredibly complicated. Deployment gets much less attention than training, so out-of-the-box solutions are few and far between.
- Many of the existing solutions don't focus on performance, so for certain use cases, they fall short. Serving tensor data from a complex model via a Python-JSON API is not going to cut it for performance-critical applications.
On the decision to make the tool publicly available, Abrams, who led the GraphPipe project, told TechCrunch: “We care more about the standard becoming quite broadly adopted than we do about our particular implementation of it because that makes it easier for everyone. It’s really up to the community decide that this is valuable and interesting.”
GraphPipe includes a set of flatbuffer definitions, guidelines for serving models consistently according to those definitions, examples for serving models from TensorFlow, ONNX, and Caffe2, and client libraries for querying models served via GraphPipe.
To get started, visit the project on Oracle's GitHub.