Bringing Real-Time SAP Data to the Databricks Data Intelligence Platform
/How AecorSoft Data Integrator lands SAP data in governed Delta tables, and why Databricks Lakebase is a strong home for a highly available ADI cluster
Finance, supply chain and manufacturing teams increasingly want their SAP data on Databricks, next to IoT, CRM and web data and close to their AI work. That takes more than a copy of the tables: SAP data must arrive complete, stay in sync with the source, and be governed as carefully as in SAP itself.
AecorSoft Data Integrator (ADI) moves SAP data into Databricks in batch and in real time. This article walks through the architecture, the Databricks capabilities that matter most for SAP data, and Lakebase as the database of a highly available ADI cluster.
ADI and SAP Business Data Cloud
SAP Business Data Cloud Connect shares SAP-managed data products with Databricks through Delta Sharing. The share itself is zero-copy, but SAP first replicates the data from the source system into Business Data Cloud, on SAP's schedule and in SAP's data model, and it requires a Business Data Cloud subscription.
ADI loads any SAP data directly from S/4HANA or SAP ECC into your own Delta tables, from custom Z tables to large tables such as ACDOCA: every column including customer extension fields, filtered as needed, with real-time CDC and resumable parallel loads. ADI reads at the SAP application layer, so SAP authorizations apply. Organizations already using Business Data Cloud can use both side by side.
Figure 1. SAP to Databricks with AecorSoft Data Integrator
How the Data Flows
- Extract. ADI reads SAP tables (transparent, cluster and pool), extractors (through OData, the SAP-compliant way), CDS views, BW InfoProviders, queries and Open Hub destinations, BAPIs, RFCs, ABAP reports and T-codes, and InfoSet queries. Full loads run in parallel packages.
- Capture changes. ADI's real-time CDC captures inserts, updates and deletes in SAP as they happen, and CDC delta runs apply them continuously. For objects without a delta mechanism, Super Delta captures changes with proprietary algorithms.
- Stage. ADI writes CSV or Parquet files to a Unity Catalog volume (created automatically) or to ADLS Gen2.
- Load and apply changes. ADI orchestrates the load jobs itself: a Databricks SQL warehouse bulk-loads the staged files into Delta tables, with INSERT for full loads and appends and MERGE for upserts and CDC. Customers usually have to write this ingestion themselves, with Auto Loader and a Lakeflow Spark Declarative Pipeline or a hand-built job; with ADI there is none to build or maintain.
Connections use TLS, and ADI signs in with a token or as a service principal (OAuth), with the secret optionally in a key vault. A failed full load resumes where it stopped.
SAP Business Names in Unity Catalog
SAP's technical names, such as VBAK or VBELN, mean little to most business users, or to AI. ADI creates the Delta tables with the descriptions from the SAP ABAP Dictionary as Unity Catalog comments: the table gets its SAP description (Sales Document: Header Data for VBAK), and every column its business name (Sales Document for VBELN). Analysts find the right data in Catalog Explorer, and Genie Agents read the comments to understand it. Not every replication tool does this, and without it, teams maintain these descriptions by hand.
Databricks Capabilities That Matter for SAP Data
- Unity Catalog governance. SAP data often includes payroll, pricing and customer details. Unity Catalog applies grants, row filters and column masks, and records lineage and audit logs, across all workspaces.
- Delta Lake transactions. MERGE applies SAP changes atomically, so readers never see a half-applied delta. Time travel reads earlier versions within the retention period, and Change Data Feed passes changes on to downstream pipelines. For a permanent history, ADI's append mode keeps every change.
- Large-table performance. Liquid Clustering and Predictive Optimization keep tables like ACDOCA fast without manual tuning.
- AI on governed data. Genie Agents let business users talk to their SAP data in plain language. Teams build AI models and Agent Bricks agents on the same tables, with data and AI governed together by Unity Catalog and Unity AI Gateway.
- Open sharing. The tables stay in open Delta format, and Delta Sharing gives partners, other platforms or SAP Business Data Cloud access to curated SAP data without extra copies.
Lakebase for the ADI Cluster
A single ADI server keeps metadata and logs locally and fits many landscapes. ADI Cluster Edition adds high availability and scale-out: the leader fires schedules and chains, runs are shared across nodes, and CDC tasks and failed loads continue on another node. It is built for near-zero downtime:
- Planned maintenance without downtime. For Windows patching or most ADI upgrades, an admin drains one node at a time. The node finishes its running loads and hands its CDC tasks and schedules to the other nodes, so SAP data keeps flowing.
- Automatic failover. If a node fails, the other nodes take over its schedules and CDC tasks, typically within a minute, and an interrupted table load continues from its last checkpoint. Changes made in SAP meanwhile are not lost: they are delivered once the CDC task continues on another node.
The nodes share a cluster database (PostgreSQL or SQL Server) that also holds the run logs, and it must be highly available with synchronous replication, so a failover never loses committed state. While the database fails over, running loads continue, and new runs start again once it completes.
Databricks Lakebase, the serverless PostgreSQL service generally available since early 2026, meets these requirements with little administration:
- High availability across zones. A primary with one to three secondaries in other availability zones, automatic failover, and the same connection string afterwards. Commits are acknowledged only once durable in a replicated quorum, so none are lost on failover. Lakebase has had an SLA since July 2026.
- No database password. ADI signs in with OAuth, as a service principal or a Microsoft Entra ID identity, and renews the one-hour tokens in the background.
- Branching and point-in-time restore. A branch is a writable copy of the cluster database in seconds, to rehearse an ADI upgrade or investigate a run without touching production.
- One platform. Lakebase databases register in Unity Catalog, and its change data feed (Public Preview) can bring ADI's run logs into Delta tables for monitoring. ADI can also load SAP data straight into Lakebase tables for Databricks Apps.
Scale-to-zero is not available with high availability, so an HA cluster database runs continuously, but it needs little compute: it holds metadata and logs, not SAP data. ADI also supports other PostgreSQL and SQL Server databases that meet the requirements.
Choosing a Deployment
| Single server | Cluster Edition on Lakebase | |
|---|---|---|
| Best for | Getting started, smaller landscapes | Business-critical SAP feeds, many CDC tasks |
| When a server fails | Loads and CDC pause until it is back | Other nodes take over, typically within a minute |
| Planned maintenance | Loads and CDC pause | No downtime: one node at a time |
| Metadata and run logs | Local to the server | Lakebase, replicated across zones |
| Database sign-in | Not needed | OAuth, no stored password |
Conclusion
SAP data reaches its full value on Databricks when it arrives complete, current and governed. ADI brings it from the SAP application layer into Delta tables under Unity Catalog, with SAP business names as comments, real-time CDC and resumable loads. For critical feeds, ADI Cluster Edition on Lakebase removes the single point of failure and keeps downtime near zero, with a managed, zone-redundant database that needs no password and stays within the Databricks platform.
