Concepts

Building Blocks

Trace
The fundamental object in Braintrust. A trace is a single execution of an instrumented set of code. Traces appear in two places in Braintrust: Experiments (for traces run outside of production on test data) and Logs (for traces run in production on live user inputs).

Span
An atomic unit of computation. Spans form the building blocks of a trace. Components of a Span include:

• Input
• Output
• [Expected output]
• Metadata—This includes system generated metadata like Metrics (e.g. duration) and Context (where in your code this ran), and flexible metadata which you can define under Attributes.

Objects and Organizing Constructs

• I. Organization
  The billing unit of Braintrust. Can represent a company or team.
  • A. Project
    A single AI feature. Experiments run within the same project should generally be able to be compared to each other.
    • 1. Experiment
         A set of traces run to test the behavior of code outside of production. The input data comes from pre-defined data, where each member will result in one Experiment Trace. (Note: Braintrust Datasets are a convenient way to store and manage input data for Experiments.)
      • a. Experiment Trace
        A single trace in an Experiment.
    • 2. Logs
         The set of all Log Traces.
      • a. Log Trace
        A trace run to monitor code in production. The input data comes from user input in an application.
    • 3. Dataset
         A list of data inputs and, optionally, their expected values and metadata. Primarily used for running Experiments.
    • 4. Prompt
         A versioned prompt that can be referenced in your code.
    • 5. Playground
         An environment for quickly iterating on prompts and model parameters and seeing the result.

Helpers

Eval()
A function that makes it easy to run Experiments (and create the set of Experiment Traces inside of them).

Proxy
An abstraction for connecting to LLMs. Provides a single API across AI providers.