Insights · 12 min read
A data platform centralizes the data scattered across your CRM, payment processor, and production systems. Here's what it costs to build, what to buy instead, and where projects stall.
By GGP Editorial
Most companies do not have a data problem. They have a data location problem. The data is already there: in the CRM, the payment processor, the production database, the marketing tools, a folder of spreadsheets. It is just scattered, and any question that needs two of those sources costs a day of manual export and a nervous spreadsheet. A data platform is the fix. It is the layer that pulls everything into one place, cleans it, and hands it to whoever needs it: a dashboard, a forecasting model, a finance team, a customer-facing feature.
I want to be clear about scope before we go further, because this term gets used loosely. This article is not about building an analytics product with dashboards for customers. That is a reporting application, and I cover it separately in our analytics platform guide. A data platform is the plumbing underneath it: the ingestion, storage, transformation, and serving layers that make data reliable enough to trust. It is the difference between a warehouse full of boxes and a system that knows where every box is.
If you are a founder, CTO, or data lead whose teams keep asking for "one source of truth" and not getting it, this is for you. I will cover what a data platform is, which parts you should buy instead of build, what it costs, the build sequence that avoids the usual stalls, and where most projects go wrong.
A data platform has five layers, and almost every serious implementation has all five whether or not anyone drew a diagram of them.
Ingestion. Getting data out of source systems and into one place. This is connectors to your CRM, database, payment processor, product analytics, and whatever else the business runs on, plus the scheduling that keeps them current.
Storage. Where the data lives once it arrives. Usually a cloud data warehouse or lakehouse: Snowflake, BigQuery, Databricks, Redshift. This layer holds raw and transformed data and charges you for compute and storage.
Transformation. Cleaning, joining, and reshaping raw data into tables that are actually usable: a `customers` table, a `revenue` table, a `churn` table. This is where "one source of truth" is either built or faked.
Orchestration. The scheduler that runs ingestion and transformation jobs in the right order, retries them when they fail, and tells you when something broke.
Serving. Handing finished data to the people and systems that use it: BI dashboards, spreadsheets, an ML model, an API that a product feature calls.
Notice what is not on that list: a dashboard. Dashboards are consumers of a data platform, not part of it. A company can have beautiful dashboards sitting on top of a mess, and the dashboards will disagree with each other and with finance. The platform is what makes them agree.
The single most useful decision in a data platform project is what to buy versus what to build, and the answer is more lopsided than most people expect.
| Layer | Buy or build | Why |
|---|---|---|
| Storage (warehouse/lakehouse) | Buy | Snowflake, BigQuery, and Databricks have spent years on query engines. You will not beat them, and you do not need to. |
| Ingestion | Buy first, build the gaps | Fivetran, Airbyte, and built-in connectors cover most sources. Build custom connectors only for your own internal systems. |
| Transformation | Build (in SQL) | dbt gives you the framework; the SQL that encodes your business rules is yours and has to be written by someone who knows the business. |
| Orchestration | Buy or use open source | Airflow, Dagster, Prefect. This is configuration, not invention. |
| Serving | Build | Dashboards and APIs are specific to your business. This is where your team adds value. |
The pattern: the expensive, hard, commodity layers are bought. The layers that encode your business rules and serve your customers are built. I wrote a fuller treatment of this logic in build vs buy software, and it applies here with unusual force. A custom query engine is a vanity project. Custom metrics are the whole point.
One clarification on the storage layer, because "lake" and "warehouse" get thrown around. A data warehouse stores structured data optimized for fast queries, which is what BI needs. A data lake stores raw files cheaply, which is where you keep logs, images, and unprocessed events. A lakehouse (Databricks, and Snowflake with its Iceberg tables) tries to be both. For most businesses a warehouse is the right first purchase, with a lake added later if raw data volume or ML training demands it.
There is no honest single price for a data platform, because scope ranges from "one warehouse and five connectors" to "a regulated, real-time, multi-tenant platform." What I can tell you is where the money goes, because it is not where most buyers expect.
| Scope | What you get | Typical range |
|---|---|---|
| Starter platform | One warehouse, 5-10 connectors, a handful of core models, one BI tool connected | $25,000-$60,000 |
| Operational platform | Many sources, governance and access controls, data quality checks, CI for models | $60,000-$150,000 |
| Product-grade platform | Real-time or near-real-time pipelines, data APIs serving product features, ML training data, strict compliance | $150,000-$400,000+ |
These are ranges, not quotes, and they move with the number of sources, the latency you need, and how regulated the data is. The number that matters comes from a scoped proposal, which estimating a custom software project walks through.
The surprise for most buyers is that the largest cost is people, not software. The warehouse bill is real and grows with use, but the dominant line is data engineering time: designing the models, writing and maintaining the pipelines, fixing the jobs that break when a source changes its schema. A data engineer is one of the more expensive engineering roles, and a platform that "only costs $2,000 a month in the warehouse" still needs someone paid to run it. Our custom software development cost breakdown covers the same dynamic, that engineering effort dominates custom builds.
The second surprise is that governance costs more than anyone budgets. Access controls, PII handling, retention rules, and audit logging are unglamorous, but they are the difference between a platform finance will trust and one it will not. In regulated industries this is not optional; it is the first requirement.
Most data platform projects stall because they try to boil the ocean: every source, every historical backfill, a governance framework, and a real-time pipeline, all before the first dashboard. The sequence below is the version that ships.
Start with one warehouse and three tables. Pick the three questions your leadership actually asks: revenue, churn, order volume. Build the models for those three and nothing else. Get finance to sign off that the numbers are right. That agreement is the foundation of everything after it, and it is where most teams should spend their first month.
Add sources one at a time. Each new connector is a small project with its own surprises: a field that means different things in different regions, a timestamp in the wrong time zone. Do them one by one and you learn the shape of your data. Do ten at once and you drown in edge cases.
Add governance when a second team shows up. Governance before anyone uses the platform is paperwork. Governance after three teams are silently exporting different numbers is firefighting. The trigger is the second team: when marketing and finance both depend on the platform, access controls and a data dictionary stop being optional.
Add real-time only where the business needs it. Batch jobs that run every hour cover almost everything. Real-time is for fraud checks, live pricing, and operational alerts, and it costs multiples more to build and operate. Our fraud detection guide shows where the real-time line actually sits, and our recommendation engine guide shows the pipeline work even a modest ML feature demands.
A few failure patterns show up so often they are worth naming.
Hiring a data scientist before a data engineer. A data scientist with no clean data builds models on garbage and produces beautiful slides. A data engineer makes the data trustworthy first. When a company has one data hire to make, it should usually be the engineer.
Modeling after the dashboards are built. If each dashboard queries raw source data with its own joins and filters, the numbers will disagree. The shared model layer has to come first, or you are building ten silos instead of one platform.
Buying a platform before knowing the questions. A data platform is infrastructure for answering business questions. If you cannot name the five questions the business needs answered, you are not ready to build; you are ready to think. No software fixes an undefined problem.
Treating it as a one-off project. A data platform is never finished. Sources change, the business changes, the questions change. If you budget for the build and nothing for the run, you get a very expensive shelf.
Given how much is bought, it is fair to ask when building a data platform from scratch makes sense at all. It makes sense in a few specific situations.
When the data platform is the product. If your product is reporting, analytics, or data APIs sold to customers, the data layer is your competitive edge and you will build a lot of it in-house. When latency is a hard requirement. Real-time operational systems, like a multi-market trading system that has to keep order state consistent across exchanges, need a data layer you control end to end. When the data is too sensitive or too unusual for standard tooling, or when the integrations are all internal systems no vendor has a connector for.
In every other case the right move is a bought foundation with a built layer on top. Buy the warehouse, wire the connectors, write the models, and keep the parts that are specific to your business.
GlobeSoft is a software development company founded in 2018, with 40-plus engineers and more than 300 delivered projects for over 100 clients across the US, Brazil, South Africa, and Singapore. We build the data layers that sit under a lot of our other work: the real-time market data in a trading system, the financial reporting in an accounting platform, the pipelines that feed recommendation and fraud models. Data platform work sits inside our enterprise software and AI development practices, and our stack for it is Java, Spring Boot, Go, Python, and the usual warehouse and orchestration tooling.
What we tell clients at the start is usually this: define the three questions first, buy the storage, and let us build the models and the serving layer. A financial management system with real-time reporting is the kind of result you get when the data layer is built right instead of bolted on after.
If your teams keep asking for one source of truth and not getting it, or you are about to fund an AI project and are not sure the data is ready, talk to us about your project. Tell us what sources you have and what questions you need answered. We will help you map the build-versus-buy call, the scope, and a budget before you spend anything.
What is a data platform?
The infrastructure that moves data from your source systems into one place, cleans and models it, and serves it to the people and systems that need it. It is the plumbing under dashboards and AI, not the dashboards themselves.
How much does it cost to build a data platform?
A starter platform with one warehouse and a handful of connectors typically runs $25,000 to $60,000. An operational platform with governance runs $60,000 to $150,000. A product-grade platform with real-time pipelines and data APIs runs $150,000 to $400,000 or more.
Should I buy a data platform or build one?
Buy the storage and ingestion layers (Snowflake, BigQuery, Databricks, Fivetran, Airbyte). Build the transformation and serving layers, because those encode your business rules. Build more of it in-house only when the data platform is the product or latency is a hard requirement.
What is the difference between a data warehouse and a data lake?
A warehouse stores structured data optimized for fast queries, which is what BI needs. A lake stores raw files cheaply for logs and unprocessed data. A lakehouse combines the two.
How long does it take to build a data platform?
A starter platform with a few sources and core models often ships in six to ten weeks. A full platform with governance, many sources, and real-time pipelines takes months, and the work never really stops because sources and questions keep changing.
Do I need a data scientist or a data engineer first?
A data engineer. Clean, trustworthy data comes before models. A data scientist without good data builds models on garbage; a data engineer makes the data usable first.
Tell us what you are building and where you are today. We typically reply within 24 hours.
