How to Create Flexibility in a Structured World
When we built the first version of our Documentation product 4 years ago, we had a single goal: create structured data in the primary bond market.
At the time, the industry had a major problem that blocked any hopes of automation: unstructured data. At source, new issuances were being defined in termsheets and final terms in unstructured Word documents. Every part of the financial system had data locked away in this way, and a manual process was required to extract it.
Against that backdrop, our Documentation product was born.
The power of structure
The idea behind our product was to define the set of Word documents that needed to be produced and insert a number of variables and logic into each of them. Then, for any given trade, our system would parse the set of documents needed for the issuance and dynamically create an intelligent and user-friendly input form.
As an example, here’s a snippet of a termsheet:
Our system reads this template and creates a form for the user to fill in. In this case, it asks for 3 inputs:
Currency
Issue Size
Minimum Denomination
The system can then output a document with the appropriate values in the right place. The user gets the document they need, and the system gets the important data points in a structured electronic format. For example:
This core idea reversed the existing status quo where documents were created first and data was pulled out second. Our product allowed users to input the data in one place and get all the documents that depend on that data as a byproduct. And, at the end of the process, every relevant data point for that new issuance was stored and validated, ready for distribution throughout the industry.
This original idea fueled the first round of industry adoption – and even created one or two copycats along the way! But we soon found that it wasn’t enough. In the process of structuring data, we’d killed something really important: flexibility.
Enforcing a strict template had, undoubtedly, improved the process for documents that had a strict pro forma (like final terms), but it had made it harder to work with others that were more loosely defined (like most termsheets).
To return to our example, users would come to us and say they wanted the termsheet fields to have different labels:
They were happy that the system captured the 3 important data points (currency, size, and denom), but for whatever reason, they wanted the template to look slightly different to the one we had created. As much as customers tell us they want standardisation, the messiness of the real financial world will always get in the way of that ideal.
A balance worth fighting for
Over the years, we’ve been faced with this paradox: how to offer flexibility to document creators without sacrificing the capture of the data for the purposes of automation.
Our first insight into how we could do this came from a simple question: what if we only structured the parts of the document where we needed the data? There are many parts of the document (e.g. disclaimers, use of proceeds) that didn’t include any data that we needed to capture, and we realised that we could make these fields free text without sacrificing the data extraction process. We decided to do this by creating a new “rich-text” input type that allowed users to define both the content and the styling of a particular field’s value.
The process of creating a rich-text field that worked in a browser and maintained formatting when placed into a Word doc was difficult. We tried (at least!) 5 different pre-built rich-text editors before deciding on the one to be released to our customers. It got the job done, but it wasn’t user friendly and caused a lot of issues in our system due to its inefficient design.
To overcome a few of these problems, this year, we took the plunge and decided to build our own rich-text editor. We always try to use out-of-the-box components, as it, generally, results in a better user experience – if an organisation’s whole mission is to make an excellent rich-text editor, it should be fantastic! But, unfortunately, this wasn’t possible here and we spent half a year developing a new rich-text editor that was fit for the needs of the platform.
While it was a lot of work, the results seemed to be impressive, but whilst our rich-text fields managed to get us a little closer to our goal of flexibility, it came at a cost. Any data contained inside the content of a rich-text field could not be structured for downstream use, meaning that only a few cases actually could use this field.
We realised that we needed something with flexibility that didn’t sacrifice structure. We had to find the sweet spot between the two.
A balancing act
This problem of creating flexibility in a document without losing structure has been one of the hardest problems that Origin has tried to solve in its product development history. Over the years, we’ve discussed thousands of ways to solve it, but never came up with a solution.
But on a warm summer day at last year’s team offsite in Tuscany, we made a breakthrough. Many of our really difficult technical challenges have been solved with ideas that originated from one of our hackathons (our CTO, Rob, wrote a great post about them last week) and this problem was no different. During our offsite, we ran a hackathon with two teams looking at this problem. At the end of 48 hours, after the two teams presented their findings, it was hard to believe that a seemingly unsolvable problem had 2 potential solutions!
At the beginning of this year, we took one of the proposed solutions and began to build a version we could release to our customers. The solution enabled us to build on top of the work we had already done with our document templating engine and the idea behind it was simple: let users edit everything in the document that wasn’t the data we were extracting. Despite its simplicity, it was tough to get right, and our design and engineering team spent nearly a year building a user-friendly solution that worked across all the document variations.
As before, users define their data in one place and get their documents as a byproduct. But today when you get those documents, you can edit, comment or leave suggestions on any part of the document that is not part of the data. The data is still protected to ensure that it can work for you at the end of the trade. Returning to our example:
Users can now edit all the text that they see. If they edit one of the structured data inputs, that will alter the variables in green, as well as the electronic structured data we hold against the trade. But they can also edit all the grey text surrounding those green variables in case, for whatever reason, cosmetic or otherwise, they want the text to say something different.
A never ending quest
As Origin’s lead product manager, I’ve learned over the years that products are never complete. They continue to evolve and improve based on feedback we receive from our customers. I really hope that the latest addition to our flexibility toolbox has improved your workflow, but I would love to hear what your experience has been.
Who knows, maybe there’s still more to do on this part of the product? Given what I know about the complexity of the problem we’re trying to solve, I wouldn’t be surprised!