What is Changed and Why it Matters? Apache Iceberg was built around one core idea: separate the data from the metadata. The data files sit in object storage, and a catalog keeps track of the tables and their current state. That separation
TL;DR: Source databases are constantly evolving. Columns are added, renamed, and dropped. Data types are widened. Usually without warning. When that happens, most replication pipelines break or lose data silently. OLake is built to absorb
OLake is an open-source tool for replicating databases into Apache Iceberg and S3 compaitable destinations, built for fast, reliable Full Refresh, Incremental, and CDC syncs. To know more about how OLake Go works read, deep dive into OLake'…
Health score fix - Restored missing tableRuntime.setTableSummary(pendingInput) updates in TableRuntimeRefreshExecutor after the cron changes. Health score calculation is now scoped to applicable Iceberg tables only. Refresh table and Fusion…
This guide details the process for deploying OLake UI on Kubernetes using the official Helm chart. Ensure the following requirements are met before proceeding: Kubernetes 1.19+: Administrative access to a Kubernetes cluster Helm 3.2.0+: Hel…
A catalog is a entity from where OLake Fusion fetches Iceberg tables. It stores reference to the latest metadata file which stores info such as table names, schemas, and file locations. OLake Fusion must know which catalog (and database) a
Got questions, ideas, or just want to connect with other data engineers? 👉 Join our Slack Community to get real-time support, share feedback, and shape the future of OLake together. 🚀 Your success with OLake is our priority. Don’t hesitat…
AWS DMS is a database migration service with CDC layered on. OLake is a replication engine built for streaming change data into Apache Iceberg. In a published benchmark on just over 4 billion rows, OLake ran the full load about 4.6x faster,…
OLake Go Playground is a self-contained environment for exploring lakehouse architecture using Apache Iceberg. It comes preconfigured with all the required components, allowing you to experience the complete workflow without manual setup. E…
Once your job is created in OLake Go, you can manage and control it using various job-level features available through the Actions menu (⋮) on the Jobs page. These features allow you to trigger immediate syncs, modify stream configurations,…
OLake Go supports data partitioning when writing to S3. You can define a partition_regex during schema changes in OLake UI or in streams.json in OLake CLI. This regex determines how data is partitioned into folders within your S3 bucket. Pa…
The following benchmark evaluates performance, environment configuration, and operational considerations for compacting Apache Iceberg tables using Apache Spark rewrite_data_files and OLake Fusion compaction on a CDC-like TPCH workload. We