# Stop Adding Another Database: Scaling Crypto Analytics in Postgres | Mike Freedman - Tiger Database - Channel: [Ethereum Denver](https://streameth.org/ethereum-denver) - Date: 2026-03-09 - Duration: 20:07 - Topics: ETHDenver, Crypto, Web3, Blockchain, Event, Conference, ETHDenver 2025, ETHDenver 2024, Bitcoin, Ethereum - Watch: https://streameth.org/watch/yt-3VVzQkR9BH8 - YouTube: https://www.youtube.com/watch?v=3VVzQkR9BH8 ## Description πŸš€ Get Ready for ETHDenver 2026! πŸš€ We're already hard at work preparing for next year's biggest Web3 event! Keep your eyes peeled for more info on ETHDenver 2026β€”it’s going to be epic! 🌟 ## Transcript Okay, now we have a Tiger Data. You can see their boots right behind you guys. But the CTO right now is here, Mike Freriedman. He's going to talk about stop adding another database. There you go. Take the stage. >> Thank you. Thank you. Uh it's coming up. Okay, great. Uh thanks for being here. My name is Mike Freriedman. I'm the co-founder and CTO at uh Tiger Data who's the creator of Timecale DB. I'm also a professor of computer science at Princeton. Uh today I have a request which is stop adding another database. Scale your crypto analytics with Postgress. Your engineering team will thank you today and your future team will thank you as well. But first let's talk about the problem. Across the crypto industry, real time analytics power customerf facing applications. Decentralized exchanges power order book and candlestick charts in real time. Uh DeFi protocols surface liquidity, lending, staking. Uh NFT marketplaces track mints, royalties and price evolution. Uh node level blockchain infrastructure and analytics platforms with you know real-time and historical insights. Although these products look different, their data characteristics are actually similar and they all need to scale as these systems grows. Uh the default reaction is predictable. A team begins typically with Postgress for transaction transactions and indexing because that's where most applications start today. Uh when analytics queries become expensive, they introduce a separate data warehouse. Uh when search requirements are expand or when they want to perform hybrid search, uh they add an engine for keyword search or a vector store. Each addition solves a specific problem, but they introduce complexity. Data must be duplicated, pipelines must remain synchronized, queries cross system boundaries, and latency and inconsistencies emerge. Uh, and operational overhead compounds. Now your team needs expertise in multiple systems. So while the instinct to add another database is understandable, the question is whether it is necessary or wise. And that's the main argument I want to make today. Don't rush to add a second, third, or even fourth system. Extend the foundation of your general purpose database to handle new use cases. Now, this does not mean denying that crypto workloads can be complex or demanding. Rather, it's to recognize that crypto workloads are still structured in a certain way. So, build specifically with that in mind. For example, blockchain data is ingested as appendheavy event streams. The data is naturally ordered uh either by time stamp or by block height. Your architecture should optimize for that. Queries are are are dominated either by lookups for specific tokens or transactions or by aggregations for some specific time interval. So organize data layouts for those patterns and premputee the rollups that you need to serve dashboards or APIs. And history grows continuously without bound. Although recent data is much more uh of is much more active than historical. So you have to scale but you still want to serve data quickly. So if you have a technology designed explicitly for these properties many of the pressures that lead teams to adopt additional technologies can be avoided and you can instead scale with a unified Postgress architecture. So that's the perspective I want to share today and what hundreds of thousands of developers are using with time scale DB built on Postgress. First a little a bit about us. Uh Tiger Data formerly known as Timecale is the company behind Timecale DB. Our focus has been building modern application database for uh data intensive applications systems that are timeoriented, appendheavy and analytical by default. We enable Postgress to serve new use cases originally for time series analytics and more recently for vector data and keyword search. We have a large open source community and also offer the fully managed tiger cloud for production scale needs with a modern developer experience. Uh time scaleb launched in 2017 as a postgress extension. Uh to be clear it doesn't fork Postgress or replace its core. Rather Postgress extensions are implemented using hooks inside the database and kind of run in the same processor memory space. They extend Postgress while fully preserving its ecosystem uh tooling and semantics. Today, Times Scaleb has 22,000 GitHub stars and a large number of time series and real-time analytic workloads. We're trusted by a thriving community with millions of active databases per month and over 2,000 customers on our managed time scale cloud. Many of those customers operate in the crypto ecosystem including Poly Market, Pancake Scale, Swap, MetaMask, OpenC, Chainlink Labs, Orca, Quicknote, Getoolabs and many many others. So I want I add this just to say this EOS this architecture is not theoretical uh but powers real exchanges, real platforms and real financial infrastructure. So with that context, let's look at this architecture that actually makes it possible. So I want to talk about three key architectural building blocks that help power crypto workloads. The first is hyper tables which automate data partitioning. Data is typically partitioned by time stamp such as for order book information or token prices or anything that behaves monotonically like the block height of a of a of a blockchain. The second is hyper core which introduces columnar storage and compression designed for high cardality time series data. And the third is continuous aggregates which provide automated rollups or in database speak incremental materialized views that you can query with standard SQL. These replace the needs for separate warehouse data pipelines. Together these three capabilities address ingestion, storage efficiency, and analytical query performance in ways that are directly aligned with how crypto data behaves. So I'm going to walk through them oneonone and show this. Hyper tables align storage with a pen mostly data. So data gets partitioned automatically by a time dimension such as timestamp or block height or in included those embedded in UIDs like events. As new data arrives, it is routed to the appropriate partition without any application logic. You can query these hyper tables like any normal Postgress table and the database planner is smart enough to exclude those chunks outside the requested time range and paralyze queries to the remaining. Further, you can apply uh policies declaratively to the partitions such as compression policies, data retention policies and taring and the database engine manages all of this automatically. Hyper tables turn out to be powerful building blocks. Uh poly market for example keeps their real-time P&L charts and pricing charts in hypert platform. This allows them to keep processing high volume market data with really dynamic and bursty patterns. Similarly, OpenC uses uh hypertables to index onchain events and track real-time market data like sales, listings, and price history. But even as data rates rise, when using hypertest ingestion remains predictable and data life cycle management remains simple and automated. Now once data is partitioned, the next question is how is it physically organized within these partitions? Crypto workloads tend to do tend to exhibit two dominant query patterns. Sometimes you need a wide and shallow view such as the entire market over the last hour. Sometimes you need a narrow and deep view such as the behavior of a single token over the past year. These two patterns place different demands on storage layouts. One scans across many assets in a short window. The other scans deeply across time for a single asset. Colinar storage when combined with appropriate uh techniques I'm going to describe handles both efficiently. The engine in time scale DB which we call hyper core quickly converts data to a columnar format and organizes them according to a segment key. You typically choose an asset ID or a token ID and time. So we typically basically end up with a thousand or so of these readings for a given token all collocated together in storage and we can compress that batch of column data together. In fact the compression algorithms that we imply use type specific algorithms for even greater efficiency. As a result, uh, wide recent queries can then scan across compressed segments during a specified time intervals, while narrow historical queries can read tightly packed together data for a single asset without touching unrelated tokens. And compression rates are significant. For example, Orca achieves roughly 88% compression while processing billions of swap and liquidity events on Salana, while Msari achieves 94% compression rates for pricing and candlestick data across thousands of assets. To be clear, this reduction is not merely about cost. By shrinking the working set, it actually also improves cache locality and makes queries much faster. It's not uncommon for these columnar queries to be 10x or even 100x faster than you would get on standard Postgress. Now once storage is aligned and physically organized efficient efficiently, the next dominant workload in crypto is rollups. Dashboards do not typically query only raw data like individual swaps or order book information. They typically query for candles, rolling averages, you know, total value locked and other time bucketed metrics commonly visualized across time. All those red and green candles you're all used to staring at. And these are not ad hoc queries, but the data you serve again and again to customers thousands of times per second or more. So the com comment here is don't recmp compute from raw data each time if you could avoid avoid it. premputee and incrementally incrementally maintain this data. After all, this isn't a warehouse where you are primarily doing random exploratory analytics. Continuous aggregates maintain these rollups incrementally within Postgress. A base hypo table let's say with one minute uh OCLV uh data can feed hierarchal aggregates at 5 minute, 15 minute, hourly and daily intervals. Importantly, these aggregates kind of showing here can be queried just like normal data with normal SQL. They aren't cached data. For example, if the database has one hour rollups, but you want to query for three-hour interval, it'll just perform that aggregation at query time. You aren't limited to what's pre-calculated. More powerfully, they also greatly simplify data management. If you ever update existing raw records or backfill new records to old time periods, the database intelligently tracks the dependent roll-ups that now need to be recmputed. These cause the engines to log in validations and then incrementally recomputee aggregations they need in the background all automatically. Uh perhaps even more exciting in next month's release, the database will automatically detect the right aggregation level from which to serve a query. So your app can just query the same underlying table. But if there's an aggregation that can more efficiently serve that query like an hourly rollup, the database engine is smart enough to pick that. Continuous aggregates are ubiquitous for crypto companies who want real-time dashboards. This is exactly how Pancake Swap and MetaMask maintain multiple granularities of candles across many assets. They ingest raw pricing information to time scale DB and allow it to compute these higher level rollups right in the database. At this point the natural skepticism might arise. All of these sounds architecturally coherent but can Postgress really sustain these at a meaningful rate? Maybe one data point I could give is we do itself which is dog fooding at our best. Uh the Tiger Cloud console includes an observability product called insights shown here on the right. It allows developers to drill down into the behavior of individual queries. Yet the back end of this telemetry platform is a single standard database service running on Tiger Cloud. It just crossed the one quadrillion metric boundary last month and stores more than three pabytes of data largely tiered more on this later and ingests over three trillion metrics per day. So when I hear somebody saying like once you hit a billion records you out Postgress. Well, this always gives me a hold my beer uh kind of feeling. And so with that context, let me talk about how other ways uh people can scale crypto use cases over time. On the right path, crypto systems face two distinct pressures. Sustained high throughput ingestion and periodic largecale backfills. High throughput ingestion is straightforward conceptually but demanding operationally. Blockchain networks generate continuous event streams and that can be a fire hose and bursts that stress both memory and storage. As I discussed, hyper tables are a key enabler here. But in more recent releases, we took this to next level with what we call direct compress. With direct compress, data is never first written to a traditional row store and then asynchronously converted to this columnar format. Incoming rows are instead batched, ordered and segmented only in memory and then written directly to compressed form in the columnar store. Yet this is still maintains transactional guarantees. This reduces write amplification lowers IO pressure particularly for the write ahead log. Now as some numbers in this benchmark with a very simple schema though traditional writes using time scale tob the row store can still reach a very respectful four million rows per second with direct compress enabled throughput continue to scale even as client writes uh concurrently reaching over 60 million rows per second at max load. And these are just standard clients using copy commands to write into a single database. Uh we're very excited about this. Um, but ingestion is really only half the story. Platforms continuously onboard new chains or new tracked assets. This requires replaying or backfilling history often to the chain's genesis block which can be billions of records inserted into a live production system. But a key win because our columnar storage is fully mutable. You could insert, update, or delete at will. Further, when you segment by token ID, ingesting a new token doesn't require rewriting or recompressing existing data. As new data is backfilled, continuous aggregates start immediately rebuilding to include this token. This is both operationally simple and and quite efficient. So the same architectural principles that make ingestion stable also now make backflow routine and that is we think about as right path scale. On the read path, the pressures are different. As query volume grows, especially during market events, horizontal scaling becomes necessary. Replica sets allows reads to scale independently of writes, all behind a single load balanced endpoint. Replicas can be removed or added dynamically up to 10 replicas per database in Tiger Cloud. And built-in load balancing distributes traffic without any application change. As an example, I love uh consider polyarket. When interest in trade volume spiked in the major election cycles in recent years, they quickly scaled up their read replicas on Tiger Cloud to handle the volume of requests that they were receiving. But query volume is not the only readside pressure. Uh historical depth is also significant. After all, blockchain history does not shrink. Even with compression, long-term retention requires new ways to scale. And for that, uh Tiger's Cloud's fully integrated tiered storage is our answer. It allows recent data to remain in primary storage for low latency queries, but it continuously moves older partitions to object storage and yet still preserves a SQL interface. When you query from within Postgress, the database is smart enough to dispatch certain queries to hot data, certain queries to old to cold data, and also known when to take one query and split it across both. And you could continue to build these continuous aggregates in hot storage even if the underlying raw asset or raw data is tiered out to cold storage. So we see many chains in granular token pricing in tiered storage going back years while more recent data and candlestick rollups remain in hot storage. Read path scaling then in crypto had two dimensions horizontal scaling query capacity and scaling historical both can be addressed in an integrated postgress architecture and last what we're working on lately once transactions and analytics can coexist within a single server within a single system adding intelligence becomes a natural extension what do we mean by intelligence additional Postgress extensions that allow new functionality to coexist in the same system. PG vector and PG vector scale, the latter of which we built, allows vector embeddings to live along structured time series data for things like rag and structured search. PG text search, which we also built, enables fast uh keyword indexing, maybe of things like governance proposals, contract metadata or onchain messages. In fact, these can be used together for hybrid search to power AI workflows like MCP tools and other things. In fact, we're beginning to see some DeFi application to use AI to answer just in time questions like what happened in the news that could explain the price volatility of token X. Crucially, these capabilities all operate within the same query engine. You could filter by token ID in time ranges when performing AI search. You do not again need to synchronize across multiple systems to build customerf facing applications. Now I began with the proposal that or the observation that the default response to scale is adding additional system postgress for transactions a warehouse for analytics a search engine a vector store that instinct is understandable but I've talked about the alternative what you can do instead when your trusted and loved database Postgress is extended and reimagined for time series data hopefully I've shown that if you design explicitly for time and block height the architecture begins to align with the shape of blockchain data. If you organize storage around workload shape, wide and shadow queries, deep and narrow scans, you can turn physical layout into a performance advantage. If you compute incrementally, roll up stopping batch jobs and become part of the database itself. If you scale horizontally and tier data seamlessly, growth stops becoming a crisis. At this point, adding a new database technology, you know, we feel is no longer the default answer and is basically another complexity or as to say your crypto protocols are decentralized by design, but your data stack does not need to be fragmented. So, thank you. Uh, I'm happy to take one or two questions. I also want to say we are located right there and in fact, I think right after this, we're doing a raffle on a nano ledger wallet. So, if you haven't, some of you already signed up. If you haven't, come over, get a raffle ticket. We'll do that soon. Maybe time for No, no time for questions. I talked too long. Thank you all for your time. See you again in the raffle.