TLDR Data 2026-08-06

Netflix's TimeSeries Abstraction stores and queries multi-petabyte temporal datasets, tiering older, immutable slices from hot Cassandra into cheaper S3 storage. The earlier design used Kafka and Flink to stream Parquet to S3 and daily compaction, serving cold reads with roughly 500 ms p99 latency, but added operational cost. The later Cassandra-native cold tier reads directly from S3 backups, serves 15+ PB of compressed cold data, and improves p90 latency by about 30% while preserving the same query API.

Meta scaled its ads recommendation foundation model 4x in 12 months and doubled training efficiency to 20–25% MFU through five techniques: jagged flash attention to eliminate padding waste, a unified attention kernel with 2–3.5x speedup, MXFP8 precision training, SM-free collective communication that reclaims ~23 SMs, and a load-balancing shuffle that adds 4% efficiency at zero communication cost.

Flipkart's Product Analyser trains a relevance labeling model in two stages: SFT on millions of balanced query-product examples with generated reasoning traces, then GRPO alignment using an 8B reward model that scores explanation quality alongside label correctness. The automated NDCG report it produces differs from the manual one by under 1%, at 30% lower cost.

Polars lets teams develop ETL pipelines locally on a representative data sample, then run the exact same LazyFrame queries across 16 billion rows in the cloud without rewriting the logic. The pipeline pre-aggregates raw Polymarket data into small Parquet artifacts in S3, allowing a Plotly Dash dashboard to remain fast without scanning the full dataset on each request.

A web service latency rollout shows the mean rising 9%, the median falling 46%, and p99 rising 119% within the same dataset, demonstrating how single-metric summaries can hide bimodal distributions. Further analysis like density plots, CDFs, shift functions, and ridgelines provide complementary views that together identify response size as the root cause.

Defining metrics once in a semantic layer reduces the risk of conflicting numbers across Tableau, Power BI, and notebooks, and makes permission controls and change propagation auditable from a single place. The semantic layer is an operational risk mitigation strategy rather than an optional layer.

pyhctsa is a Python toolkit for extracting hundreds of statistical and structural features from single or multiple time series, with support for custom feature sets and variable-length inputs. It also supports direct operation calls and parallel processing across local CPU cores for faster analysis.

pgGraph is a PostgreSQL extension that adds fast graph search, traversal, and shortest-path queries to existing relational tables without moving data into a separate graph database or learning a new query language. It builds a rebuildable, memory-efficient graph index while PostgreSQL remains the authoritative system of record.

Smevals is a lightweight Python CLI for running task-specific evals across multiple models to find the cheapest one that meets a quality bar. You define tasks, write YAML graders with custom checker scripts, run them against a set of model configs, and view scored results in a local dashboard or static site.

DuckLake is now implemented for Apache DataFusion as a production-ready, open-source catalog backend, giving Hotdata transactional metadata, snapshots, and schema management for Parquet tables in object storage. The design supports multiple relational catalog backends and logical catalogs, enabling millions of isolated, ephemeral databases for AI agents without duplicating infrastructure. TPC-H benchmarks across 22 queries and three dataset sizes showed no measurable overhead versus direct Parquet access, with slightly better performance due to less metadata discovery during query planning.

Over 500,000 subscribers read TLDR Data to stay on top of the latest in data science and data engineering. If you work in data and want to help curate it, send your LinkedIn or resume to data@tldr.tech!

DB Trail makes MySQL audit logs queryable by linking row changes to session identity, SQL statements, and before-images. It provides attributed, row-level answers for destructive actions and generates transaction-scoped reversal SQL to restore affected data.

Encoding improves analytical query execution, while compression mainly reduces storage. Dictionary encoding and frame-of-reference enable fast comparisons, pruning, and SIMD operations. CedarDB's results suggest always encoding, then adding zstd only when the disk savings justify it.

Etsy built Claude Code skills for seven Kafka streaming workflows, covering ML feature generation, embeddings, and fan-out pipelines.

GraphRAG works best for multi-hop, relationship-heavy, or explainable queries, while vector RAG is cheaper for semantic lookup.