Skip to main content

· One min read
Garren

I did a presentation at the FoundationDB Monthly Meetup. The talk is about, how building secondary indexes for a database is always about balance. A balance between a system that scales and is easy to manage and an API that is intuitive and delightful for a developer to use. Recently at Tigris Data, we have been adding secondary indexes to our metadata store and have been working hard to achieve a good balance between scale and developer delight. Tigris is a storage platform that leverages FoundationDB as one of its core components. In this talk I cover four aspects we had to balance:

  1. Handling schema changes automatically in our secondary indexes.
  2. The trade-off between auto-indexing all fields and indexing select fields.
  3. Changes we made after performance testing.
  4. How we plan build indexes in the background with minimal conflicts.

· 4 min read
Jigar Joshi

CSE

Client-side encryption refers to the practice of encrypting data on the client side (user's device) before it is transmitted to a server or stored in a remote location. This approach enhances the security and privacy of user data by ensuring that sensitive information is encrypted locally, and only the encrypted form is transferred or stored on the server.

Here's how client-side encryption typically works:

CSE_Flow

· 9 min read
Peter Boros

We are running FoundationDB with the official kubernetes operator. FoundationDB supports logical backups (with backup_agent) and disaster recovery (with dr_agent) through copying the database/streaming changes logically. It also supports binary backups through disk snapshots.

In this blog post, we will describe how to make a backup of FoundationDB via backup_agent. The FoundationDB operator supports making logical backups via backup_agent, but it does not support running DR with dr_agent. We decided to run backup_agent as a separate deployment to allow a symmetric setup with dr_agent.

· 3 min read
Himank Chaudhary

Tigris is a globally available, S3 compatible object storage service. Tigris utilizes FoundationDB's transactional key-value interface for its underlying metadata storage. This blog delves into the topics of serializable transactions, the mechanics of transactions within Tigris, and concurrency control.

· 9 min read
Himank Chaudhary
Ovais Tariq

Building a new storage platform

The most complicated and time-consuming parts of building a new storage system are usually the edge cases and low-level details. Concurrency control, consistency, handling faults, load balancing, that kind of thing. Almost every mature storage system will have to grapple with all of these problems at one point or another. For example, at a high level, load balancing hot partitions across brokers in Kafka is not that different from load balancing hot shards in MongoDB, but each system ends up re-implementing a custom load-balancing solution instead of focusing on their differentiating value to end-developers.

This is one of the most confusing aspects of the modern data infrastructure industry, why does every new system have to completely rebuild (not even reinvent!) the wheel? Most of them decide to reimplement common processes and components without substantially increasing the value gained from reimplementing them. For instance, many database builders start from scratch when building their own storage and query systems, but often merely emulate existing solutions. These items usually take a massive undertaking just to get basic features working, let alone correct.