Why We Built DIP
By 2025, most mid-to-large enterprises had data distributed across three to seven different systems: a legacy data warehouse, a cloud object store, a Kafka event stream, multiple relational databases, and often a separate ML feature store. The cost and latency of cross-system queries was unsustainable.
DIP solves this by creating a single logical lakehouse layer over all your data — without requiring you to migrate it. Everything is queryable through a single SQL-compatible API endpoint at dip.cogfocus.com.
Core Architecture Components
Storage: Apache Iceberg Table Format
All DIP-managed tables use Apache Iceberg format, giving every query ACID transactions, time-travel (query data as it existed at any past timestamp), partition evolution, and schema evolution without table rewrites.
Compute: Distributed Spark Pods
DIP runs Spark compute in isolated pods — one per tenant. Pods scale from 2 to 128 vCPUs within 90 seconds, and auto-suspend after configurable idle periods to eliminate runaway costs. Spark catalogs are pre-configured to read from DIP's Iceberg metastore.
Query Layer: DIP Intelligence API
The DIP SDK exposes a natural-language query interface layered over a standard SQL engine. Enterprises with existing BI tools (Tableau, Power BI, Metabase) connect via JDBC/ODBC — no migration required.
Migration Without Disruption
DIP's federated connector layer mirrors data from existing sources on a configurable schedule using change-data-capture (CDC) streams. Enterprises migrating from Snowflake, Redshift, or BigQuery achieve zero data loss with continuity of existing dashboards and reporting pipelines.
"The migration that DIP's federated connector handles in hours would have taken our data engineering team months of manual extract-load scripts."
Get Started
Visit dip.cogfocus.com to access the platform documentation, request a trial environment, or schedule an architecture review with our data engineering team.