Kung Fu Data 09 Sep 2016
In a meeting room off ElectroRoute's trading floor last month an analyst presents output statistics to his colleagues. As we are led through the insights from tens of thousands of market orders I quip. "Really nice analysis but unfortunately the results are not that compelling, could you redo for the French market instead? Surely it's as simple as changing one line of code from UK to FR"
My sarcastic jibe did not find a landing in most of the young room but did appear to chime with one colleague of similar age to myself. His wide-eyed glance confirmed a distant memory of the toil and diligence required to redo this sort of analysis from scratch on a completely new data set.
As the famous management consultant, Peter Drucker never said: "If you can’t measure it, you can’t manage it.”
Our performance is numerically driven so the measurement of each trade is key to managing our customer's assets and our business. Our trading platform processes one hundred thousand trades each year.
Our production systems have been designed and have evolved to capture and process transactional trade data. Trade capture, risk, position monitoring, credit, reporting and settlement systems all rely on this data. On the other hand, analytics plays a big part in the success of trading companies today. Some believe that analytics is the single most important way to achieve a competitive advantage. As our analytical requirements grew from the early days of ElectroRoute we found that our transactional based production system was not ideally suited to support our pre-trade analytics.
A focus on transactions, tight change control, central ownership and management and low amounts of non-trade related ad-hoc data were not the core characteristics sought by those asking analytical questions. Without an analytics infrastructure, data analysis was a manual heavy process fraught with toil and error.
In designing an analytics infrastructure we were faced with the question of how to balance the requirement for centralised transactional data for our customers and ourselves with the more ad-hoc and often sparse analytical data required to feed the curiosity of the organisation. Aware that legacy design choices would colour our vision of the road ahead we did not want to impose an enterprise-wide solution at too early a stage of the company's growth.
We instead opted for diversity in our data infrastructure. Front office analysts were provided full autonomy to design and build an analytical infrastructure to suit their emerging needs. What has emerged is an analytics infrastructure that exists in parallel to our core production systems. It is designed by our analysts to suit their exact needs. Agility and ad-hoc data processing of a wide array of trade and non-trade data are key features of the infrastructure. Yes data duplication does exist in places between both systems but this downside is far outweighed by the autonomy, flexibility and agility this infrastructure delivers to the organisation.
So what did we learn as we forked our data infrastructure in this manner ?
- do your production on stable infrastructure designed and operated for transactions
- do development on infrastructure designed and operated for flexibility
- when it comes to analytics data never trade quality for flexibility. Bad data yields bad insights but distributing results from contaminated data is like having horse meat in your beef burgers. You have many insights but only one reputation, bad data can easily damage it
- pay attention to data entry and data cleansing
- industrialise and standardise the fetching of data to cut out recurring errors.
Cutting back to the meeting room off ElectroRoute’s trading floor last month the analyst has little effort in producing the new results "Yes give me a few minutes here and I'll rerun it for France". New insights delivered quickly from our analytics infrastructure is the new norm while next door our trades flow independently and seamlessly through our production systems. Magic, data agility at pre-trade stage backed by data stability post trade. At ElectroRoute people like us do things like this.
– Alan Mullane, Head of proprietary trading
Photograph: British Wildlife Photography Awards 2016, Young people’s award, 12-18 years: Rebecca Bunce (age 18)