It has never been quicker or easier for organisations to share data: individuals can share terabytes of data at the click of a button via email or a cloud storage service. But that doesn't mean building a business-to-business (B2B) data-sharing capability is technologically straightforward.

For the most part, business IT systems have not been designed with either data sharing or 'big data' in mind. So businesses that want to share their data, or use data shared by others, will need to ensure that the systems they use to capture, store, manage and process data are up to the task.

The impetus to share data also impacts the IT infrastructure of all the parties involved, not least because systems might not have been designed to operate across corporate boundaries, or at least not outside of established supply chains.

One of the first challenges is to ensure that data is kept in a format that can be used and understood by all parties. Often this relies on industry standards. These are well-established in some sectors but still a work in progress in many others.

Another challenge is creating an IT architecture that supports multiple data sources and allows for a high degree of automation, such as through application programming interfaces (APIs). The trend currently is towards using independent platforms to gather shared data; while this has advantages for security and privacy, it brings its own design challenges.

Lastly, data sharing platforms need to be able to scale up as they gain more users and more data sources. These data spaces are now more likely than not to be created in the public cloud. However, this requires accompanying investment in standards, automation and information technology to ensure adequate performance.

Setting standards for B2B data sharing

Both the UK Government, in its National Data Strategy, and the EU, through the European Data Act, want to encourage data sharing. "There is clear indication from governments that they want to see data sharing, and the next piece is in the EU [European] Data Act, to proactively encourage the sharing of data within European public sector organisations," says Jocelyn Paulley, Gowling WLG partner and co-head of the firm's Data Protection and Cyber Security practice.

But, according to trade group the European Round Table for Industry (ERT), a business advocacy organisation, the absence of common data standards, as well as Europe's lack of cloud computing infrastructure, stands in the way of moves to expand data sharing within Europe. The ERT calls for common European "data spaces" – common infrastructure for sharing data – and industry-driven standards. Vitally, the group says, both corporations and SMEs must be involved in drafting standards. The hope is that this will lead to "sharing-ready, high-quality and interoperable data". This point is echoed by techUK, the trade body that represents the IT industry in the UK, which is also calling for industry-driven standards.

Poor-quality data, on the other hand, makes sharing more difficult and costly, and limits how businesses can use analytics or algorithms to extract value from datasets. Part of the challenge is that, as with privacy and security standards, businesses are moving into uncharted territory when they start to share data with third parties.

Much of the investment in sharing data has so far been within existing supply chains and commercial relationships. Systems such as electronic data interchange (EDI) are an established way for organisations to share business messaging and transactional information. Bodies such as data standardisation group GS1 have developed this further with services, such as GS1 XML, that allow businesses to share information using the internet.

However, industries still need to deliver their own terminology, or Core Business Vocabulary (CBV), and data standards. This becomes all the more complex when firms start to bring in data from multiple sources and combine it for analysis or to create a data product. Even something as simple as an address or postal code can be recorded in several ways, causing cost and friction, or even errors.

Fixing this needs investment up front, before businesses can start to benefit from data sharing. "You've probably heard the adage 'rubbish in rubbish out'," says Seftton Samuels, managing director at SMMT Data Intelligence, the data arm of the Society of Motor Manufacturers and Traders. "We receive data in various quality formats. When we gather it, whether it's from participants in the sector [or] government departments, we use various processes to cleanse, improve and enhance that data so our data is more accurate. Indeed, the quality of the data is our USP."

Quality data is the foundation of SMMT's growing data sharing service. But the investment in time and skills is significant. Although there are standardised technical terms across the industry, there is still ambiguity, Samuels cautions. This requires SMMT to liaise closely with the UK government and the UK's Driver and Vehicle Licensing Agency, as well as international partners. And the SMMT's data scientists need to remove misunderstandings and human error before information is shared.

Chris Connelly, chief strategy officer at Heywood Pension Technologies and one of the architects of the UK's Pension Dashboard Project, has similar experiences. Establishing the right framework for data standards might not be glamorous, but it is vital.

"There are lots of ISO codes for how you can format data, for example," he says. These provide vital shortcuts for creating data products, but you also need a standard to explain what the data means. "I know what language I'm speaking in when I send you my data. But when you consider 12 different sectors of our industry and their varying data, each can look at the standard and say, okay, that's basically our common language."

Architecture and integration

Besides standards, sharing data also requires up-front investment in both technology and integration. The trend, at least where "open" data is concerned, is to create data spaces, or architectures where all those holding data can collaborate. This is safer from a cyber security point of view and poses fewer compliance risks than transferring records to a centralised database.

This is the architecture adopted by the UK Pensions Dashboards project, which will allow savers to access their records using one or more websites approved by the UK's Financial Conduct Authority. These dashboards themselves, though, will not hold pensions data. This data mastery will remain with the pension funds, allowing them to keep their own data-matching controls. A saver might use their chosen dashboard website to check their retirement income projections, and in the process find a pension they had forgotten about.

"What the website will do is call the industry infrastructure that we're all building, go through some consents with the saver, and then search everywhere where there are pensions," explains Connelly. "So instead of you searching one bucket, you're potentially searching 40,000 different places where pensions are held. Then, the architecture makes sure that is secure and will contact every single endpoint where the pensions are stored. And you'll see them all listed together on the website. The feet that are paddling away, whilst the swan serenely goes over the top."

For the most part, data sharing services are being built using APIs, which allow high degrees of automation and almost instant transfers of data. However, there is, as yet, no standard reference architecture for data sharing. The ERT, for example, ranks such an architecture, alongside data quality, as one of the pillars of data sharing.

The lack of established technical data sharing standards for data transfer between energy systems is holding back much-needed data exchange, says Pete Davies, director of data, technology and transformation at the Retail Energy Code Company. "There is no standardised machine-to-machine basis for communication," he says. "It is one of the reasons why data sharing across energy is currently lacking and needs to improve."

Data sharing at scale

Standards and architectures will provide the building blocks for data sharing, but questions remain about whether, and how, data sharing can scale.

The growth of cloud computing allows businesses to stand up IT resources quickly and without up-front capital investment. And the cloud lends itself to collaboration between businesses, using shared infrastructure, in a way that would be far harder to do with proprietary systems in private data centres. There are concerns, however, about whether the European IT industry has sufficient capacity to allow for widespread B2B data sharing, even though promoting such data sharing is a priority for the UK Government and the EU.

Efforts are underway to address this. The EU's data strategy includes the ambition to establish industry-specific data spaces, which combine data standards and integration protocols, governance controls, and dedicated infrastructure to facilitate B2B data sharing. But even if these spaces live up to the vision, effective data sharing will also require investment in the technology by the businesses that seek to profit from it.

The technical barriers of architecture and data standards can, Paulley suggests, be overcome, as both individual firms and the wider economy stand to benefit. "Someone having that clarity of vision, or being able to articulate what they are going to achieve and why everyone should contribute to the benefit that comes out of data sharing is absolutely key," she says.

Key takeaways

  • Investigate your industry's standards – Data standards may already be well established in your sector, or efforts may be underway to create them.
  • Invest in quality – Data quality will be essential to the success of any B2B data-sharing initiative but requires skills, process and technology.
  • Consider your architecture – Your company's current IT infrastructure is unlikely to have been built to support data sharing at scale, and you may need to build, buy or rent new systems.

Data Unlocked

Want to learn more about B2B data sharing? Read the other articles in our Data Unlocked series:

If you have any questions relating to this article, or data sharing in general, please contact Jocelyn Paulley.

You can also sign up to our mailing list to receive future data-related articles.