Berkeley DB Programmer's Reference Guide

Table of Contents

Preface

Conventions Used in this Book

For More Information

Contact Us

1. Introduction

An introduction to data management

Mapping the terrain: theory and practice

Data access and data management

Relational databases

Object-oriented databases

Network databases

Clients and servers

What is Berkeley DB?

Data Access Services

Data management services

Design

What Berkeley DB is not

Berkeley DB is not a relational database

Berkeley DB is not an object-oriented database

Berkeley DB is not a network database

Berkeley DB is not a database server

Do you need Berkeley DB?

What other services does Berkeley DB provide?

What does the Berkeley DB distribution include?

Where does Berkeley DB run?

The Berkeley DB products

Berkeley DB Data Store

Berkeley DB Concurrent Data Store

Berkeley DB Transactional Data Store

Berkeley DB High Availability

2. Access Method Configuration

What are the available access methods?

Btree

Hash

Heap

Queue

Recno

Selecting an access method

Btree or Heap?

Hash or Btree?

Queue or Recno?

Logical record numbers

General access method configuration

Selecting a page size

Selecting a cache size

Selecting a byte order

Duplicate data items

Non-local memory allocation

Btree access method specific configuration

Btree comparison

Btree prefix comparison

Minimum keys per page

Retrieving Btree records by logical record number

Compression

Hash access method specific configuration

Page fill factor

Specifying a database hash

Hash table size

Heap access method specific configuration

Queue and Recno access method specific configuration

Managing record-based databases

Selecting a Queue extent size

Flat-text backing files

Logically renumbering records

3. Access Method Operations

Database open

Opening multiple databases in a single file

Configuring databases sharing a file

Caching databases sharing a file

Locking in databases based on sharing a file

Partitioning databases

Specifying partition keys

Partitioning callback

Placing partition files

Retrieving records

Storing records

Deleting records

Database statistics

Database truncation

Database upgrade

Database verification and salvage

Flushing the database cache

Database close

Secondary indexes

Error Handling With Secondary Indexes

Foreign key indexes

Cursor operations

Retrieving records with a cursor

Storing records with a cursor

Deleting records with a cursor

Duplicating a cursor

Equality Join

Data item count

Cursor close

4. Access Method Wrapup

Data alignment

Retrieving and updating records in bulk

Bulk retrieval

Bulk updates

Bulk deletes

Partial record storage and retrieval

Storing C/C++ structures/objects

Retrieved key/data permanence for C/C++

Error support

Cursor stability

Database limits

Disk space requirements

Btree

Hash

Specifying a Berkeley DB schema using SQL DDL

Access method tuning

Access method FAQ

5. Java API

Java configuration

Compatibility

Java programming notes

Java FAQ

6. C# API

Compatibility

7. Standard Template Library API

Dbstl introduction

Standards compatible

Performance overhead

Portability

Dbstl typical use cases

Dbstl examples

Berkeley DB configuration

Registering database and environment handles

Truncate requirements

Auto commit support

Database and environment identity checks

Products, constructors and configurations

Using advanced Berkeley DB features with dbstl

Using bulk retrieval iterators

Using the DB_RMW flag

Using secondary index database and secondary containers

Using transactions in dbstl

Using dbstl in multithreaded applications

Working with primitive types

Storing strings

Store and Retrieve data or objects of complex types

Storing varying length objects

Storing arbitrary sequences

Notes

Dbstl persistence

Direct database get

Change persistence

Object life time and persistence

Dbstl container specific notes

db_vector specific notes

Associative container specific notes

Using dbstl efficiently

Using iterators efficiently

Using containers efficiently

Dbstl memory management

Freeing memory

Type specific notes

Dbstl miscellaneous notes

Special notes about trivial methods

Using correct container and iterator public types

Dbstl known issues

8. Berkeley DB Architecture

The big picture

Programming model

Programmatic APIs

C

C++

STL

Java

Dbm/Ndbm, Hsearch

Scripting languages

Perl

PHP

Tcl

Supporting utilities

9. The Berkeley DB Environment

Database environment introduction

Creating a database environment

Sizing a database environment

Opening databases within the environment

Error support

DB_CONFIG configuration file

File naming

Specifying file naming to Berkeley DB

Filename resolution in Berkeley DB

Examples

Shared memory regions

Security

Encryption

Remote filesystems

Environment FAQ

10. Berkeley DB Concurrent Data Store Applications

Concurrent Data Store introduction

Handling failure in Data Store and Concurrent Data Store applications

Architecting Data Store and Concurrent Data Store applications

11. Berkeley DB Transactional Data Store Applications

Transactional Data Store introduction

Why transactions?

Terminology

Handling failure in Transactional Data Store applications

Architecting Transactional Data Store applications

Opening the environment

Opening the databases

Recoverability and deadlock handling

Atomicity

Isolation

Degrees of isolation

Snapshot Isolation

Transactional cursors

Nested transactions

Environment infrastructure

Deadlock detection

Checkpoints

Database and log file archival

Log file removal

Recovery procedures

Hot failover

Using Recovery on Journaling Filesystems

Recovery and filesystem operations

Berkeley DB recoverability

Transaction tuning

Transaction throughput

Transaction FAQ

12. Berkeley DB Replication

Replication introduction

Replication environment IDs

Replication environment priorities

Building replicated applications

Replication Manager methods

Base API Methods

Building the communications infrastructure

Connecting to a new site

Managing Replication Manager Group Membership

Adding Sites to a Replication Group

Removing Sites from a Replication Group

Primordial Startups

Upgrading Groups

Managing Replication Files

Running Replication Manager in multiple processes

One replication process and multiple subordinate processes

Persistence of local site network address configuration

Programming considerations

Handling failure

Other miscellaneous rules

Running Replication using the db_replicate Utility

One Replication Process and Multiple Subordinate Processes

Common Use Case

Avoiding Rollback

When to Consider an Integrated HA Application

Choosing a Replication Manager Ack Policy

Elections

Synchronizing with a master

Delaying client synchronization

Client-to-client synchronization

Blocked client operations

Clients too far out-of-date to synchronize

Initializing a new site

Bulk transfer

Transactional guarantees

Master Leases

Changing Group Size

Read your writes consistency

Getting a token

Token handling

Using a token to check or wait for a transaction

Clock Skew

Using Replication Manager message channels

DB_CHANNEL

Sending messages over a message channel

Receiving messages

Special considerations for two-site replication groups

Network partitions

Replication FAQ

Ex_rep: a replication example

Ex_rep_base: a TCP/IP based communication infrastructure

Ex_rep_base: putting it all together

Ex_rep_chan: a Replication Manager channel example

13. Distributed Transactions

Introduction

Berkeley DB XA Implementation

Building a Global Transaction Manager

Communicating with multiple Berkeley DB environments

Recovering from GTM failure

Managing the Global Transaction ID (GID) name space

Maintaining state for each distributed transaction.

Recovering from the failure of a single environment

Recovering from GTM failure

XA Introduction

Configuring Berkeley DB with the Tuxedo System

Update the Resource Manager File in Tuxedo

Build the Transaction Manager Server

Update the UBBCONFIG File

Restrictions on XA Transactions

XA: Frequently Asked Questions

14. Application Specific Logging and Recovery

Introduction to application specific logging and recovery

Defining application-specific log records

Automatically generated functions

Application configuration

15. Programmer Notes

Signal handling

Error returns to applications

Environment variables

Multithreaded applications

Berkeley DB handles

Name spaces

C Language Name Space

Filesystem Name Space

Memory-only or Flash configurations

Disk drive caches

Copying or moving databases

Compatibility with historic UNIX interfaces

Run-time configuration

Performance Event Monitoring

Using the DTrace Provider

Using SystemTap

Example Scripts

Performance Events Reference

Programmer notes FAQ

16. The Locking Subsystem

Introduction to the locking subsystem

Configuring locking

Configuring locking: sizing the system

Standard lock modes

Deadlock detection

Deadlock detection using timers

Deadlock debugging

Locking granularity

Locking without transactions

Locking with transactions: two-phase locking

Berkeley DB Concurrent Data Store locking conventions

Berkeley DB Transactional Data Store locking conventions

Locking and non-Berkeley DB applications

17. The Logging Subsystem

Introduction to the logging subsystem

Configuring logging

Log file limits

18. The Memory Pool Subsystem

Introduction to the memory pool subsystem

Configuring the memory pool

Warming the memory pool

The warm_cache() function

19. The Transaction Subsystem

Introduction to the transaction subsystem

Configuring transactions

Transaction limits

Transaction IDs

Cursors

Multiple Threads of Control

20. Sequences

21. Berkeley DB Extensions: Tcl

Loading Berkeley DB with Tcl

Installing as a Tcl Package

Loading Berkeley DB with Tcl

Using Berkeley DB with Tcl

Tcl API programming notes

Tcl error handling

Tcl FAQ

22. Berkeley DB Extensions

Using Berkeley DB with Apache

Using Berkeley DB with Perl

Using Berkeley DB with PHP

23. Dumping and Reloading Databases

The db_dump and db_load utilities

Dump output formats

Loading text into databases

24. Additional References

Additional references

Technical Papers on Berkeley DB

Background on Berkeley DB Features

Database Systems Theory