Hybrid Lakehouse
Hardening Data Platform
|
Outdated and unstable Hadoop and Dremio |
Upgraded Dremio. Upgraded Hadoop involving 30 application teams. Scaled-out Dremio Coordinators. Outcome – stable execution of a workload of 500,000 queries reading 2PB of data daily on a relatively small hardware environment. Developed automation tool for testing Dremio upgrades at production workload scale. |
| Rudimentary Monitoring | Created comprehensive monitoring solution collecting metrics from all Dremio coordinators and executors which enabled us to quickly identify and resolve a thread leakage problem. |
|
No workload analytics and workload management strategy |
Created Dremio Self Analytics solution. Designed workload management utilizing Dremio Engines and Queues to satisfy SLA for critical workload. Identified and optimized inefficient queries. |
|
Spark Optimization
|
Utilized open source SparkScope to optimize Spark applications. |
|
|
Developed a comprehensive set of best Practices and worked with over 30 teams to implement.
|
Modernizing Data Platform
|
Outdated Hadoop Environment |
Implementing Hybrid environment - Cloud Ready on prem and AWS - based on Kubernetes and MinIO infrastructure with Dremio, Spark, Jupyter, and other tools. |
Apache Iceberg Lakehouse
|
Migrating from parquet files to Apache Iceberg |
Centralized Iceberg Catalog allows to easily orchestrate Dremio, Spark and other workload. Deprecating significant portion of ETL code. No more “dynamic” VDS, Iceberg provides data integrity during data ingestion and transformation.No more ”snapshotting” data sets in our code.Ability to audit and troubleshoot old data states out of the box.No more metadata refresh nightmares.Support new use cases that were impossible prior. Schema evolution. |