This blog describes how New York Stock Exchange Group expanded its cloud-based market data offering by launching NYSE Cloud Streaming. NYSE Cloud Streaming is a real-time market data streaming solution that delivers high-quality market data to AWS with sub-100ms latency in Kafka-compatible streams. The solution makes it easy for customers to take advantage of real-time market data and integrate it with cloud-native applications.
About the New York Stock Exchange
New York Stock Exchange Group (NYSE Group) operates five stock exchanges and two options exchanges. These US-based markets are major global liquidity centers built to meet the needs of companies and exchange traded fund (ETF) issuers, giving investors greater choice in how they trade. We provide. The NYSE Group represents the best of capitalism and operates the largest stock trading group in the United States by market share, including the New York Stock Exchange, which believes that free markets provide all individuals with the opportunity to benefit from success. Owned and operated. NYSE Group trades an average of more than 2.5 billion shares across all markets and processes more than 500 billion messages.
In addition to owning and operating seven exchanges, NYSE Group provides customers with a wide range of data solutions across a variety of data types and granularity levels, including real-time and historical tick data, corporate actions, and reference data. . These data offerings play a key role in how customers develop trading strategies, make investment decisions, manage risk and streamline post-trade processes.
Building NYSE Cloud Streaming to Solve Customer Challenges
NYSE Group has continued its commitment to the cloud for several years to expand access to market data offerings and launch new services to meet new customer demands. In 2021, NYSE Group Markets migrated all non-real-time data offerings to AWS and shared key business drivers and migration results on the AWS Blog the following year.
Driven by the early success of its data access and distributed cloud efforts, NYSE Group has worked with customers to identify and build new cloud-based solutions that provide additional value and access. An area of strong interest from both existing customers and potential new customers is an easy and efficient way to access high-quality, real-time streaming market data natively in the cloud.
Previously, access to high-performance, real-time streaming data directly from NYSE Group required customers to build, manage, and maintain an on-premises environment or work with a third-party provider to facilitate data delivery. did. This process can be time consuming and requires upfront capital expenditure (CAPEX) investments. It also includes ongoing maintenance and tuning, capacity expansion, and hardware and software upgrades.
Customers often cite time-to-deployment, up-front investment, and the complexity of managing these environments as challenges and, in some cases, barriers to adoption. Another challenge cited by customers was the inability to receive streaming data where and in the format they needed. Customers running cloud-native applications said they have difficulty retrieving on-premises multicast feeds and integrating them into their products and applications. cloud.
NYSE Group has addressed these challenges by building NYSE Cloud Streaming, a data distribution platform that enables customers to quickly access and utilize real-time streaming exchange data in a developer-friendly format. . NYSE Cloud Streaming allows customers to access market data in the cloud with latency of less than a few hundred milliseconds. You can also stand up an environment on AWS within minutes without the upfront costs or logistical hurdles of building a physical infrastructure footprint. In addition to reducing infrastructure-related challenges, data is exposed in Kafka-compatible streams via NYSE Cloud Streaming, allowing developers to more easily integrate data in widely used formats.
NYSE Group launches NYSE Cloud Streaming on AWS, leveraging widespread adoption in capital markets around the world, industry-leading technology platforms and expertise, and data providers building and expanding real-time data delivery platforms in the cloud. This is because of my experience supporting. AWS was also a natural choice for NYSE Group. Deploying this solution on AWS provides customers with a streamlined way to access both real-time market data and historical products such as NYSE Daily TAQ in a common infrastructure environment. Anthony Zawadzki, head of NYSE Proprietary Data, said seamless connectivity to NYSE Group feeds and between different data products is important to customers. Zawadzki said, “Our customers’ key expectations are that NYSE Group continues to be at the forefront of innovation and provide solutions that make it easier for our customers to run their businesses. NYSE Cloud Streaming accelerates time to value. Today, our customers’ cloud-native applications and workflows connect to NYSE Group data while simplifying and streamlining operational overhead. It has an easy way to get started.
One of NYSE Group’s early decisions was which feeds to make available on NYSE Cloud Streaming. NYSE Group has selected NYSE Best Quotes and Trades (BQT) as the first feed for NYSE Cloud Streaming. This is because BQT provides a comprehensive view of Level 1 data, providing a unified view of the best bids & offers and trades for all five stock exchanges in the NYSE Group. NYSE Trade Reporting Facility (TRF) and various important summary information such as open price, high price, low price, close price, and consolidated volume. BQT provides subscribers with a cost-effective, integrated market data feed with lightweight network access requirements and is NYSE Group’s most popular market data feed among fintech and cloud-native subscribers.
Designing and building NYSE cloud streaming
To build NYSE Cloud Streaming, NYSE Group needed to design a platform that would deliver high throughput and low latency while facilitating direct integration into customer applications. The first decision NYSE Group had to make was whether to implement a multicast feed or build a publish/subscribe solution. With the growing demand for multicast-based cloud delivery solutions, NYSE Group believes that customers are increasingly interested in unicast streams delivered in a publish/subscribe delivery model that is most easily integrated with display applications and tools. Ta.
As a result, NYSE Group built a streaming data platform that is compatible with Kafka APIs using Amazon Elastic Compute Cloud (Amazon EC2) and Redpanda’s platforms. The platform offers the benefits of Apache Kafka, including high throughput, low latency, and most importantly, numerous client SDKs in virtually any programming language, making it easy for developers to get started in the language of their choice. I’ll make it.
Figure 1: NYSE Cloud Streaming Architecture
NYSE Cloud Streaming takes data from the original BQT multicast feed located in the NYSE datacenter, converts it to protocol buffer (Protobuf) format, and exposes it to the streaming data platform over a dedicated AWS Direct Connect link. NYSE Group chose Protobuf for its broad programming language support, performance for serializing structured data (think JSON, but smaller, faster, and simpler), and its encoding for network transport. This is because it generates optimized messages and reduces data transfer costs.
Once your data reaches AWS, it’s stored in a streaming data platform that spans three Availability Zones (AZs) to ensure data integrity and high availability. To minimize end-to-end delays for customers, NYSE Group is publishing the AZ ID where the platform is hosted. This allows customers to host applications in these AZs and reduce total latency and data transfer costs.
NYSE Group built its own Kafka cluster for ease of management and cost efficiency. Vinil Bhandari, head of cloud and full-stack engineering at NYSE, said: “The NYSE’s cloud streaming real-time offering will be a best-in-class Kafka implementation that has been thoroughly tested for high throughput and low latency. We are very excited to usher in a new era of data delivery. We look forward to pushing the boundaries with even more cutting-edge technology.”
To achieve cross-account connectivity, NYSE Group used AWS PrivateLink to make market data privately available to customers in their AWS accounts. To connect to NYSE Cloud Streaming, customers provide their AWS account information, a list of bootstrap brokers, authentication details, and the Amazon Virtual Private Cloud (Amazon VPC) they use to request a connection to NYSE Cloud Streaming. Endpoint service information must be received. .
After a customer establishes an AWS PrivateLink connection to NYSE Cloud Streaming, the next step is to configure the DNS infrastructure for name resolution for the bootstrap broker. NYSE Cloud Streaming uses TLS to encrypt communications from the streaming platform to receiving clients. By default, most, if not all, clients validate certificates during the connection handshake, checking the hostname of the server they are connecting to against the provided broker certificate.
To ensure that the bootstrap broker name resolves to the Amazon VPC endpoint IP, customers can use Amazon Route 53 private hosted zones (Route53 PHZ) to connect the broker’s name to the appropriate IP address in their subnet. needs to be resolved. Next, the customer must create an “A record” in Route53 PHZ and alias the entry to the DNS name of the VPC interface endpoint created earlier. Your customers can now start consuming feeds through Kafka-compliant clients.
How NYSE Cloud Streaming Benefits Customers
NYSE Cloud Streaming allows customers to access real-time market data feeds in minutes from anywhere in the world without needing to procure or manage on-premises infrastructure. NYSE Cloud Streaming not only reduces time to market, but also simplifies and streamlines the data consumption process. With an AWS account, customers can access NYSE Group’s real-time market data and instantly leverage a full range of AWS services, reducing time to insight and complexity.
NYSE Group also listened to customer feedback and decided to initially host NYSE Cloud Streaming in the AWS US-EAST-1 region. This region hosts historical, corporate actions, and reference data products. This not only accommodates the many customers currently deployed, but also simplifies the process of leveraging both NYSE Group’s historical and real-time data solutions.
NYSE Group continues to work closely with customers to improve this experience, including enabling cloud-based access to additional real-time data feeds such as NYSE Pillar Depth, expanding into more AWS Regions, and providing value-added services. The first step is to evaluate how to build. Additional access methods, analysis, and more.
NYSE Cloud Streaming customers benefit from:
There is no infrastructure procurement or reliance on third-party infrastructure providers, resulting in faster time to market and faster connection to NYSE Group feeds. A scalable platform that provides data to customers around the world and provides access to all NYSE Group feeds (depending on customer demand). Cloud-based real-time data feed for market data distribution powered by Kafka using industry standard security protocols. Easily integrate with your current applications and choose your preferred programming language.
All these benefits lead to a better customer experience and lower total cost of data ownership. Customers who want to start using NYSE Cloud Streaming can use a github repository that includes sample code. This code deploys all the necessary infrastructure and a Kafka client that can use the BQT feed.
