TLDR Data 2026-06-04

Only 1 in 6 orgs have the data foundation for agentic AI. Fivetran and dbt Labs will now join forces to redefine the future of open data infrastructure, enabling organizations to accelerate their AI initiatives with a robust data foundation.

Alongside the merger, Fivetran has revealed the first combined innovations from Fivetran + dbt:

• dbt Core v2.0: open sourcing of dbt Fusion engine runtime
• dbt State (preview): caching layer for data pipelines that reduces underlying infra costs by >30%
• dbt Wizard (beta): autonomous support for model authoring, refactoring, and debugging
• Agents Schema: open source standard for agentic context

Try Fivetran for free

Zepto built a Cart Contextual Model that treats shopping carts as “sentences” and uses a Transformer-based masked language model (MLM) to infer user intent in real time as items are added. By training on historical cart patterns with temporal, geographical, and product signals plus inverse-frequency masking to handle long-tail items, the model predicts what else the user will likely buy.

Manticore Search argues vector search should be tuned like a production retrieval system, not treated as a default embedding feature. It recommends aligning similarity metrics with models, tuning HNSW for recall, latency, and memory, and using batching, chunk optimization, and physical backups to keep indexes consistent.

After running 8 production-like use cases side-by-side, Ray Data was selected over Daft primarily for superior stability and resilience at scale (especially in complex async LLM inference) while acknowledging Daft's strengths in ergonomic native multimodal primitives and cleaner code for many operations.

Databricks debunks 8 common myths about data layout, arguing that Liquid Clustering is superior to traditional Hive-style partitioning for modern lakehouses. Unlike rigid partitioning, Liquid Clustering dynamically organizes data using clustering keys that evolve over time, supports row-level concurrency, metadata-only operations, and works seamlessly across open table formats.

AI agents are creating more small, bursty data queries, making single-warehouse costs harder to manage. Multi-engine routing cuts cost by sending each query to the best engine while keeping familiar workflows.

Redis Array is a brand-new native data type (introduced in Redis 8.8) designed for constant-time positional access by index, filling a long-standing gap in Redis where position/index itself carries semantic meaning. It efficiently supports both dense and extremely sparse arrays using a hierarchical group-based structure, allowing fast random access, range queries, ring-buffer semantics, pattern matching across sparse data, and fixed memory usage.

QueryFlux is an open-source Rust-based SQL routing proxy that intelligently directs queries across multiple query engines (Trino, Spark, DuckDB, Snowflake, Athena, Flink, etc.), sharing the same Iceberg tables. It handles protocol translation, dialect conversion via SQLGlot, cost-aware routing, concurrency control, and health-based failover.

dbt Core v2.0 alpha makes the Fusion engine's Rust-based runtime open source under Apache 2.0, unifying Core and Fusion around a shared foundation with faster parsing, Parquet artifacts, better local docs, simpler installs, and a tighter language spec. Fusion remains the recommended free CLI for most users, while Core v2 serves teams that need fully open source code or custom OSS builds.

ingestr is a CLI ELT tool for moving data from many databases and SaaS apps into warehouses or storage with simple flags, no backend or custom code required. It supports incremental loads, easy install, and broad connector coverage.

OpenTelemetry launched “Blueprints” to simplify observability with standard patterns and reference implementations for Kubernetes, infrastructure, apps, and centralized telemetry platforms.

MongoDB can run low-latency transactional logic without stored procedures by combining ACID transactions, bulkWrite, validation, indexes, and pipeline updates. This is demonstrated through an example that processes payments with card checks, vendor checks, limits, duplicate prevention, and ledger writes.

Lays out identity, policy, and audit patterns teams need to externally enforce least-privilege agent calls under upcoming regulation.

Pluto 1.0 marks the Julia notebook environment as stable, with major improvements to reproducibility, reactivity, sharing, accessibility, education, docs, and editor tools.

Preview adds persistent, metadata-driven data-quality decorators that fail fast and auto-route remediation in dlt pipelines.