TLDR Data 2026-06-15

Spotify built Vedder, an AI data assistant for 2,100+ users across 177 clusters, to move beyond schema-only RAG across 70,000 datasets. Domain experts curate each cluster with datasets, vetted question-SQL pairs, and business docs. Only 12.5% of mined query pairs were accepted, so health scoring tracks drift, validity, coverage, and reproducibility to keep context reliable.

Feldera treats streams as incremental SQL views, using DBSP to propagate deltas instead of recomputing joins and aggregations. Inserts, deletes, and updates become Z-set changes, so only affected rows are updated. The result is batch-SQL-like semantics for continuous pipelines with lower CPU, less memory pressure, and predictable latency.

LinkedIn built MUSE (Member Understanding Semantic Embeddings) to power semantic search inside Hiring Assistant. It uses a dual-tower Matryoshka embedding model trained on millions of high-quality labels from an LLM Teacher grounded in product policy, combining embedding-based retrieval with a downstream engagement-optimized ranker.

AI coding agents reduce coding labor, but not the hardest parts of software: design judgment, scope control, testing, and maintainability. They reduce accidental complexity, but can create technical debt, architectural drift, and bloated codebases at machine speed. The edge shifts to experts who can steer the model, say no, and keep systems production-ready.

Uber's AI budget blowout shows agentic AI is a task-economics problem, not a token-pricing problem. Claude Code adoption hit 84% across 5,000 engineers, exhausting the annual AI budget by mid-April. With spend hidden in re-sent context, retrieval, orchestration, governance, and retries, teams need to measure value per task, control context, and build stateful agent infrastructure.

Companies need to compound human expertise and AI capability, not just rely on the best model. By owning their workflows, evals, and institutional knowledge, firms can keep improving while avoiding a future where all value flows to a few frontier models.

Your data stack is held together with duct tape. You know it. Your team knows it. On June 24th, Matia CEO Benjamin Segal and Doug Laney, author of Infonomics and Data Juice, are doing a live fireside on what comes next.Register now→

Flights is MotherDuck's new agent-native data pipeline feature that lets AI agents easily build, run, and schedule ingestion and transformation workloads using a secure, general-purpose Python runtime. It has native support for dlt pipelines, direct DuckDB execution, logging, scheduling, and versioning. Agents can create Flights via MCP server, SQL table functions, or the UI.

Omnigent is an open-source Databricks meta-harness that makes agents like Claude Code, Codex, Pi, and custom agents work together through one shared layer. It helps teams compose agents, add security and cost controls, share live sessions, and keep workflows portable as tools change.

Apache DataFusion 54.0.0 adds major SQL upgrades, including LATERAL joins, SQL lambda functions for arrays, a new Arrow-based Avro reader, and spill-to-disk for memory-heavy nested loop joins. Performance also jumps, with near-unique LEFT/FULL sort-merge joins up to 20–50× faster and repartition-heavy operations improving by up to 50%.

Databricks has handed the Delta Sharing protocol over to the Linux Foundation. OpenSharing extends Delta Sharing to AI models, agent skills, and unstructured data across clouds and platforms. It adds standard APIs for discovery, authorization, and access, with support for existing Delta Sharing recipients plus Apache Iceberg/REST Catalog clients. The project aims to replace proprietary marketplaces with a single standard for enterprise AI asset distribution.

Databricks' ai_parse_document + ai_query can turn messy PDFs into structured JSON in a few SQL lines, but the challenge is reliability at scale. Every rerun reopens parsing and LLM costs, corrected documents can create duplicates, and even temperature 0 still produces non-deterministic outputs that undermine auditability. A pipeline design with checkpoints, versioned prompts, and deduplication reduces reprocessing cost and improves reproducibility. Deterministic parsers like OpenDataLoader PDF are more appropriate when document templates are consistent.

Google Research introduced Regularized f-Divergence Kernel Tests to audit machine unlearning and privacy leakage more reliably than standard two-sample tests.

SQL to ER Diagram is a free, open source, browser-only tool that turns pasted SQL schemas into clean, interactive ER diagrams without uploading your data or requiring signup.

DuckDB + Redis deliver a five-component DIY feature store that avoids training-serving skew for real-time ML and RAG systems.