A matrix for FIX messages
NYFIX Marketplace is sometimes referred to as a hub-and-spoke system. From the outside, it can look like that, with NYFIX as the 'hub' and counterparties as the 'spokes' (diagram 1). Internally, however, NYFIX Marketplace is, in fact, a matrix, with each counterparty connected directly to its counterparties. FIX messages flow directly between counterparties, with some messages also sent to database servers for safe storage or 'drop-copied' to multiple destinations (diagram 2).
Keeping track of all message senders and receivers can get complicated, especially as the size of the network increases. With NYFIX Marketplace, which is the world's largest FIX network, there are many thousands of interconnections to manage. This is where middleware comes in. It allows applications to 'publish' messages from customers or counterparties onto the network without needing to know their actual destinations. Instead, the destinations – applications servicing customers or counterparties – 'subscribe' to and receive messages relevant to them.
This publish-subscribe architecture greatly simplifies the problem of delivering millions of messages between thousands of counterparties. It also makes the NYFIX network more scalable and reliable by avoiding bottlenecks and single points of failure.
Publish-subscribe middleware is also commonly referred to as a 'service bus'.
Building our own
Initially, we relied on middleware from other software vendors, but with the middleware being such an integral part of our operation for the NYFIX marketplace and our need for more control, we decided to develop our own and started work on a project a few years ago. After assessing our requirements and the tools available, we decided to combine two leading open-source software projects and add further functionality where we needed it.
For the messaging interface, we chose OpenMAMA, an application programming interface (API) that works with any middleware, and for several years has been incorporated into commercial products and in-house systems. Middleware also requires a network transport layer – technology that deals with things like connection communication, reliability, and flow control. We turned to ZeroMQ, an open-source messaging library that is widely used in Linux-based operating systems and by major technology and R&D organizations and included in many Linux distributions. It offered the features, reliability, and performance we required for a network transport layer. ZeroMQ is essentially a toolkit, which has the advantage of not imposing architecture on what you build. On the flip side, its flexibility carries with it a steep learning curve, and developing applications with it can involve a lot of effort.
By combining OpenMAMA and ZeroMQ with the functionality we developed ourselves, we created a production- quality, high-performance message-oriented middleware that we call OZ. It provides many of the most popular features of this type of middleware, such as publish-subscribe and request/reply (inbox) messaging, dynamic service discovery ('yellow pages' for finding services) and broker-less architecture (no intermediary between message sender and receiver) for reduced latency and optimum throughput.
Sharing OZ
We rolled out the new middleware earlier this year across our Europe, Asia and North America data centers. From March 2020, OZ was live across all three regions, processing over 50 million FIX messages a day with no issues. This includes periods of significant market volatility during the coronavirus pandemic.
Itiviti believes in sharing ideas and experiences – within our teams, as well as with clients, partners and our wider community. With OZ, we have gone even further. We are sharing the actual code that we use to run our business.
Our OZ innovation would not have been possible without the contributions of many others, including the OpenMAMA and ZeroMQ communities. In the process of developing our solution as part of our evolving technology stack, we contributed back to those communities. And by open-sourcing OZ, we hope it will establish its own community that will improve the software further.
