Learn key steps to streamline the process of building a data platform based on our experience working with our customers over the past 6 years
A good data platform is a non-negotiable part of any data strategy. Still, building a powerful, secure, and flexible data platform with good user experience, strong governance, and effective automation is not a trivial undertaking.
Here are some key steps (with examples) that we’ve identified while working with our customers over 5 years, which can help you get through them with less effort:
Defining a platform roadmap
Setting the right expectations for your platform
Building a well-rounded platform team
Bringing platform users in
Each aspect needs careful consideration, budgeting, planning, convincing stakeholders, and possibly fighting against resistance from other departments. All these difficulties are amplified at the start, when the platform still hasn’t shown its full potential, so people might be skeptical about it.
Let's discuss each of them.
Defining a roadmap is a must. Not only does it allow the Platform Team to appropriately organize its implementation, but it is a key tool to communicate what the platform does now and in the future for stakeholders and users.
Customers with a roadmap in place had a much easier time communicating with the rest of the company and onboarding teams to the platform. Those without a clear roadmap had a very hard time getting others to believe in the platform.
Especially at the start, it can be difficult to define a full roadmap, but you may not necessarily have to. Even a draft will go a long way. First, it is a way to think about priorities and what tradeoffs may be needed. Second, it provides something to discuss with users and potentially integrate their feedback.
Roadmaps can and do change; while users may not be thrilled to know that the Fabric integration they wanted has been postponed because something else that is more urgent took its place, having a roadmap will allow you to show them why you made that choice and plan for it ahead of time to minimize issues.
With that said, do not treat your roadmap as something that you can change on a whim. Putting something on your roadmap is a commitment to do it. An unreliable roadmap is of no use to your stakeholders and users.
If you don’t have one at all, it will be difficult for them to see how they can leverage the platform in their day-to-day work: they won’t know when they can use the platform to solve their problems, such as start implementing their use cases, when the technologies they are interested in exploring will come online.
Let's take an example. We had a customer who couldn’t effectively define a roadmap for a variety of reasons, the main one being uncertainty in the technological direction of the broader organization. This meant they did not have an obvious choice on what to support first.
Compounding this was budget limitations, which meant that the option of starting with multiple technologies, and then eventually phasing some out once the technological direction was clarified, was out of the question. Standing still and waiting it out was also out of the question.
This uncertainty had the effect of wasting the limited development resources available:
The main mistake was following the potential use cases, which had timelines that were incompatible with the platform, and sometimes were just ideas that were quickly abandoned.
After evaluating what was going on and searching for a way out of this situation, a simple strategy was used to define a shortlist of technologies to integrate that formed the initial steps of the roadmap.
The strategy was extremely simple: integrate the technologies the teams want to use the most now, based on past, current, and future use cases. This meant some use cases and teams that may have been perfectly good candidates to be onboarded had to be left aside for the time being, as they were targeting more niche technologies.
Completing these first steps meant the platform was now able to support at least some use cases.
The remainder of the roadmap was then defined based on the needs of the users who were finally starting to use the platform, which also enabled more use cases to come in, and once the most used technologies were supported, the more niche ones could be taken care of as well.
The key was to make a decision, as imperfect as it was, to unblock the situation. As the saying goes, perfect is the enemy of good.
Overpromising upfront is the fastest way to lose trust. Set expectations early and clearly: a data platform is a long-term product, not a quick project.
Roadmaps set expectations, but they are not the only way in which the Platform Team does this. The important thing is not setting the wrong expectations, as it can quickly snowball into a lack of trust and dysfunctional processes. We'll focus mainly on the expectations that the Platform Team sets when working with the users of the platform, thus mainly Data Product teams.
The first rule is the most obvious. There is enough uncertainty in most organizations that even promises you think you can maintain sometimes have to be amended; adding unrealistic ones on top is just not a good idea. The temptation is there at the start: you may find yourself wanting to attract users to justify the investment, and making big promises to them may work to get them on board, at least in the short term.
However, not delivering on those promises will most of the time result in larger problems. Missing the delivery of a technology integration, for example, will result in use cases that fail to materialize as they relied on those technologies to be completed on time, which leads to the use case teams not trusting the platform and escalating the matters to their stakeholders, which will deem the platform unreliable.
If supporting that use case is vital, instead of making unrealistic promises, investigate alternatives that can reasonably be delivered more quickly:
The second situation in which it is important to set the right expectations is when onboarding a team or supporting them in their efforts to develop Data Products. The Platform Team (or a dedicated Onboarding Team) must help new users when they first approach the platform.
How much or how little this is needed depends on the complexity of the technologies, of the platform itself, and the skill level of the users.
Users expecting support when getting onboarded or developing their first Data Product is perfectly reasonable, and the Platform Team should deliver on this expectation.
However, the Platform Team should not be developing Data Products or acting as a pure user support role: it will quickly drain resources from platform development and maintenance.
One key fact to keep in mind is that this onboarding/support effort must scale together with the platform. If the onboarding is done purely by members of the Platform/Onboarding team, the effort needed can quickly grow out of control. Thus, the Platform/Onboarding team must find sustainable ways to onboard and support new users and reduce their reliance on direct interaction with them. The easiest ways to support your data product teams are:
These resources and processes should be built iteratively, as with each team that gets onboarded, the Platform/Onboarding team gains valuable experience that they can then solidify and use to reduce the effort for the next onboarding session.
One advantage of platforms built with Witboost is that adoption can be driven through a self-service experience: marketplaces, standardized Data Product scaffolding, and clear governance guardrails make it easier for teams to get started without hand-holding, reducing the effort needed for the Platform/Onboarding team.
Last but not least, another situation is dealing with exceptions. Use cases will often push the boundaries of the platform rules and processes, especially if it allows them to save effort when faced with a challenging deadline.
In situations like that, teams may ask for exceptions, shortcuts, or other custom workarounds. These are a part of the game, so they should not be denied by default.
However, they should be treated with the utmost care, and the pros and cons must be evaluated. Temporary exceptions (a more lenient governance policy, for example) can be made, but they should always be treated as what they are:
Temporary
and exceptions
When evaluating them, always clarify that the use case team is expected to resolve the situation that led to the exception in a reasonable time frame, and get back inside the guardrails that the platform puts in place.
Failing to do so and allowing use case teams to bend the rules of the platform whenever they wish will lead to the platform becoming unmaintainable and can impact the downstream consumers in unexpected ways.
When left unchecked, the platform devolves into anarchy, failing one of its main goals, which is to set shared rules and standards for all projects.
For a Platform Team, you want the following capabilities:
As we said, building platforms is not an easy task. They involve a lot of abstractions, generalization on specific use cases, and development of integrations between different technologies.
Building a well-rounded Platform Team will go a long way in producing a quality result. The ideal Platform Team contains experts in multiple areas. It is cross-functional. This is not that different from the approach to team composition that DevOps methodologies suggest, though the ingredients do change.
Witboost reduces the undifferentiated heavy lifting, allowing teams to focus on integration, governance, and user experience. This includes adapting existing integrations to work with your particular infrastructure setup, tuning the governance to meet your organization’s needs, and abstracting common patterns in templates.
Without a product like Witboost, you'd also need to build and maintain the control plane yourself:
The governance engine
The catalog
The templating system
The access management layer.
That's a significant additional surface area requiring dedicated software engineering effort. With it already handled, the same team can be leaner or reach maturity faster.
You can have specialized or T-shaped team members, but you should strive to check the boxes above; otherwise, your Platform Team will be less effective, and the platform will take longer to set up and bring to maturity than it could have been.
Most teams don't start with all these skills in-house, and that's fine. These gaps should be treated as delivery risks to address early, not as blockers. Plan accordingly for upskilling, new roles, or trusted external partners.
Regardless of the current role of each member, previous data engineering experience is a big plus, as it means the team members have practical knowledge of the development workflow and data pipeline lifecycle concepts that they must model and implement in the platform.
While it's not strictly necessary, if it's not available inside the platform team, this experience should be sought after elsewhere, for example, by involving teams of potential users early in the development of new features to make sure what's being planned and implemented meshes well with their way of working.
Failure to include such experience in the team or look for it outside can lead to unnecessary friction when data engineers try to use the platform, leading to rework after the users complain that the platform can't do what they need it to do.
Beyond the technical skills, you want the Platform Team members to understand and respect the needs of the users. Previous data engineering experience is beneficial, but it's not enough.
What you need to do is have the Platform Team use the platform they built. To be clear, they should not be the ones to implement the use cases. Try putting them in the shoes of their users in activities like testing a new release, preparing demos for stakeholders, onboarding user teams, etc.
This "dogfooding" aspect can really help consolidate the Platform Team's understanding of their user base and helps them focus on the problems the platform needs to solve, rather than aimlessly developing features that look good to them.
A platform that is not being actively engaged with by users is just a cost that brings no value; it will be quickly shut down, and rightly so.
The initial motivation to build a platform comes from different needs:
A platform that satisfies these needs will have no trouble justifying the initial investment.
This honeymoon does not last long at all. In the medium and long term, a platform’s sustainability and value is judged by its adoption, which means the number of users using it for their day-to-day work. Your platform may have all the solutions the organization needs, but if nobody uses it, the problems stay unsolved. “No users, no value”.
Finding users and getting them to use the platform is crucial for its continued future; failure to do so will lead to questions about the platform’s usefulness, and rightfully so. Scaling down or even canceling the project outright are all very real possibilities for a platform with few users.
The first step to getting users is making sure that people know it exists. Depending on your organization and how the platform project came to life, the management in the various areas may be aware of it and may be interested in adopting it, but there often is no such guarantee, so the effort of ensuring that other areas know about it and are on board falls onto the platform itself.
In general, don’t expect users to come to you. You have to go and get them. Having a champion helps a lot, as does using internal channels to spread awareness about the platform. Depending on your organization, these internal channels take many forms: communities of practice, guilds, internal showcases, newsletters, project portals.
It falls on you to use them. Other informal channels, like personal or business connections to other organization areas come into play as well. Your management should also be able to help you reach out and search for users.
After reaching the users, the first impression they get of the platform is of course important. They are typically looking for something that solves some pain points or problems fast.
Difficulty adopting a technology, project startup effort, development time are common problems they seek to alleviate. Focus on asking them what they need, understand their situation, and then try to develop a plan to onboard them. As discussed above, avoid making promises or excessively twisting your roadmap to suit their needs.
Users can act as a powerful way to evangelize the platform to others, help you with showcasing its functionality and advantages, and get you connected to other potential users, becoming a flywheel of your platform adoption efforts.
The best salesman is a happy customer: potential users will naturally trust opinions from other users more than those coming from the platform or management itself.
Building a successful platform is often more an organization- and process-driven challenge than a technological one. The truth is, all the technical prowess in the world will lead to cool features, but unrealistic expectations, bad communication, and no adoption will make even the best platform fail.
A platform that solves problems, is known across the company, and is widely used and liked by users will prosper even if it’s initially lacking on the technological side. After all, it is much easier to improve the technological side of things than the organizational one.
Whether you build everything yourself or leverage a product like Witboost, the success of a data platform ultimately depends on trust, adoption, and sustained alignment with user needs.
Transform data from liability to asset with proactive governance, ensuring high quality and trust from the start for better business outcomes.
Companies commit errors by staking their Data Mesh implementation on technology when the entire paradigm shift is based on a technology-agnostic...
Why conversational AI with your data often fails and how proper data governance can transform the outcome. Learn how context is key to unlock AI.