By Ivy Schmerken
October 22, 2008
Wall Street & Technology 

Any market data professional working during the week in which Lehman Brothers filed for bankruptcy and Merrill Lynch arranged its own sale to Bank of America would have noticed that market data message rates soared to new highs. The turmoil in the credit markets in September and October caused equity and option trading volumes to spike, leading to record-breaking trading volumes and quotation update rates.

Coincidentally, the same week that the credit crisis reached new heights, Exegy, a St. Louis-based consolidator of arket data feeds, launched a new Web site to help market data professionals at brokers, exchanges, ECNs and vendors cope with capacity planning. Exegy built the site, MarketDataPeaks.com, and is sponsoring it in cooperation with Xasax, a network provider of colocation services, and the Financial Information Forum (FIF), an industry association that tracks market data capacity as well as other issues that impact financial technology operations.

No Time Like Real Time

“The real-time aspect of it is the most powerful component because we can compare relative quote traffic observed on the MarketDataPeaks.com Web site with real-time issues we’re dealing with in our own environment,” comments Rob Wallos, global head of market data architecture at Citi and cochair of the FIF’s Market Data Capacity Planning (MDCP) working group.

Though FIF has historically tracked the market data message rates from the exchanges on a monthly basis, Tom Jordan, chairman of the FIF Advisory Committee, explains that the availability of real-time data is new. “This is really moving [the information from the exchanges] toward real-time presentation of the information,” he says.

For the first time, members of FIF — including the exchanges; utilities such as NYSE/SIAC; the big market data vendors Bloomberg, Interactive Data and Thomson Reuters; as well as the financial recipients of the data, such as UBS, Citi and Credit Suisse — “who are concerned with the potential stresses and strains on their systems” created by exploding market data volumes will have access to this real-time data, says Jeff Wells, Exegy’s VP of product management. “They are interested in knowing what the peaks have been and listening to the exchanges on what may be coming next to make sure they are provisioning their systems and bandwidth,” he adds.

Take the case of Monday, Sept. 15, when Lehman’s bankruptcy filing and Merrill’s sale to BofA were announced, causing the Dow to drop 500 points. That day, U.S. equity and option markets’ update rates peaked at 990,800 messages per second, according to MarketDataPeaks.com. The following day, Sept. 16, the North American equities and options trading venues set a historic high, reaching 1.83 million message updates per second at 12:17 p.m. “We saw the market peak very quickly following the various news items through the 16th,” recalls Wells. Previously, the record rate had been set on May 19, when the peak was just more than 1 million (1,058,227) messages per second.

FIF’s Jordan notes that he had always assumed daily peaks occurred at 9:31 a.m., after the market open. But he now realizes this isn’t always the case.

According to Exegy’s Wells, “On normal days, the peaks are at the beginning of the day, as the markets open up, and at the close of the day with all the closing messages. But these peaks are happening any time the news hits, and the big news is very unpredictable.” But even under normal conditions, Wells adds, the market currently experiences about 200,000 messages per second. “A few years ago, that would have blown everyone out of the water,” he says.

Since the majority of those message updates originate from option and equity order books, many FIF members are interested in a breakdown of which exchanges are generating the peak volumes, Wells continues. Explaining MarketDataPeaks.com’s functionality, he says, “You would click on the high, show how much was options, how much was stock, how much was Level II or Level I, BATS or Arca.” According to the site, it tracks all the market data messages that occur simultaneously in any given second across all live data feeds, including NYSE/SIAC, OPRA, Nasdaq, Arca, BATS and Direct Edge, including both Level I and Level II.

“It is clear that a lot of the volume comes from the order book feeds,” adds Wells. MarketDataPeaks.com reports that BATS hit 113,312 messages per second and Nasdaq TotalView hit 125,619 messages per second at the same time on Tuesday, Sept. 16. And Wells points out that the site actually underreported NYSE Arca volumes on the big day because the software was measuring data packets, not messages.

“NYSE Arca told us on Thursday [Sept. 18] that their system peaked at around 300,000 messages per second,” Wells relates. “The system has been fixed now, but that means we’ll never actually know for sure how busy the market really was.”

Exegy is constantly keeping track of the message rates coming from the North American trading venues. “All the market data peak numbers that we see [from Exegy] are provided on a per-second basis, and they include all the North American equity feeds, option feeds from OPRA as well as international feeds,” explains Arsalon Shahid, program office manager for FIF in New York.

To measure the peak market data rates and aggregate them in real time Exegy relies on its hardware acceleration appliance within the Exegy Ticker Plant. All the data is processed through the Exegy Single Ticker Plant and is updated in Xasax’s colocation facility in New York. Xasax provides on-demand virtual servers and runs a financial network called the Xasax Financial Backbone, pulling in the majority of the liquidity available from the U.S. customers of Xasax, including independent traders, smaller hedge funds and brokers that want access to the fast data to do their own high-frequency trading without contracting for the data center space themselves.

Xasax has contracted with the various exchanges to bring in those direct data feeds and is leasing machines from Exegy to handle the feeds, says Exegy’s Wells. “So that means we’ve got access to the boxes to do these measurements,” he explains.

Normalizing Data Rates

What the FIF and Exegy are trying to do is normalize the data rates observed across various trading venues and present these statistics in a common format, says Citi’s Wallos. Tracking message rate statistics across trading venues is often quite difficult since there is no requirement that exchanges publish their metrics in any standard format, he adds.

For example, some exchanges might publish messages per second versus updates per second. “If Nasdaq is publishing a statistic in megabits per second [Mbps] and OPRA is publishing something in packets per second, what does that mean for a brokerage house trying to rationalize a spend on more servers or bandwidth?” Wallos asks.

Even though most of the investment banks that take in direct exchange feeds calculate these statistics on their own systems, if it normalizes the data across all the exchanges and ECNs, MarketDataPeaks.com could become a benchmark for the investment banks to use as a reference point for their capacity planning. Under such a scenario, if a firm were breaking through capacity thresholds on its systems that process raw OPRA data, for example, it could use the site as a reference point to figure out whether it was having capacity issues with internal systems or if the problem were Streetwide.

“The whole point is to provide a relationship between the peaks and the trading-related information,” says FIF’s Shahid. “If someone is on the [NYSE] floor and they hear news related to market data capacity, they can quickly go to the Web site and see if it’s not just them being affected — the spike or the drop can tell them if it’s something bigger.”

Going forward, FIF’s Jordan says, FIF is going to build this information into its capacity-planning meetings. “FIF will provide all the peak information, and we’ll provide what time of day it happened. We’ll do it by entity, by OPRA or NYSE, for instance,” he says, adding, “The peaks are not necessarily [occurring] at the same time.”

Now that penny pricing and algo trading are expanding to options, Jordan notes, market data and infrastructure professionals are eager to delve into the underlying reasons for data spikes. “This will tell you what the real peak is,” he asserts. “It’s a tool to help people evaluate what their overall capacity needs will be.”