braintrust

A Python library for logging data to Braintrust. braintrust is distributed as a library on PyPI. It is open source and available on GitHub.

Quickstart

Install the library with pip.

pip install braintrust

Then, create a file like eval_hello.py with the following content:

from braintrust import Eval
 
def is_equal(expected, output):
    return expected == output
 
Eval(
  "Say Hi Bot",
  data=lambda: [
      {
          "input": "Foo",
          "expected": "Hi Foo",
      },
      {
          "input": "Bar",
          "expected": "Hello Bar",
      },
  ],  # Replace with your eval dataset
  task=lambda input: "Hi " + input,  # Replace with your LLM call
  scores=[is_equal],
)

Finally, run the script with braintrust eval eval_hello.py.

BRAINTRUST_API_KEY=<YOUR_BRAINTRUST_API_KEY> braintrust eval eval_hello.py

API Reference

braintrust.logger

Span Objects

class Span(ABC)

A Span encapsulates logged data and metrics for a unit of work. This interface is shared by all span implementations.

We suggest using one of the various start_span methods, instead of creating Spans directly. See Span.start_span for full details.

id

@property
@abstractmethod
def id() -> str

Row ID of the span.

log

@abstractmethod
def log(**event)

Incrementally update the current span with new data. The event will be batched and uploaded behind the scenes.

Arguments:

• **event: Data to be logged. See Experiment.log for full details.

log_feedback

@abstractmethod
def log_feedback(**event)

Add feedback to the current span. Unlike Experiment.log_feedback and Logger.log_feedback, this method does not accept an id parameter, because it logs feedback to the current span.

Arguments:

• **event: Data to be logged. See Experiment.log_feedback for full details.

start_span

@abstractmethod
def start_span(name=None,
               type=None,
               span_attributes=None,
               start_time=None,
               set_current=None,
               parent=None,
               parent_id=None,
               **event)

Create a new span. This is useful if you want to log more detailed trace information beyond the scope of a single log event. Data logged over several calls to Span.log will be merged into one logical row.

We recommend running spans within context managers (with start_span(...) as span) to automatically mark them as current and ensure they are ended. Only spans run within a context manager will be marked current, so they can be accessed using braintrust.current_span(). If you wish to start a span outside a context manager, be sure to end it with span.end().

Arguments:

• name: Optional name of the span. If not provided, a name will be inferred from the call stack.
• type: Optional type of the span. Use the SpanTypeAttribute enum or just provide a string directly. If not provided, the type will be unset.
• span_attributes: Optional additional attributes to attach to the span, such as a type name.
• start_time: Optional start time of the span, as a timestamp in seconds.
• set_current: If true (the default), the span will be marked as the currently-active span for the duration of the context manager.
• parent: Optional parent info string for the span. The string can be generated from [Span,Experiment,Logger].export. If not provided, the current span will be used (depending on context). This is useful for adding spans to an existing trace.
• parent_id: This option is deprecated and will be removed in a future version of Braintrust. Prefer to use parent instead.
• **event: Data to be logged. See Experiment.log for full details.

Returns:

The newly-created Span

end

@abstractmethod
def end(end_time=None) -> float

Log an end time to the span (defaults to the current time). Returns the logged time.

Will be invoked automatically if the span is bound to a context manager.

Arguments:

• end_time: Optional end time of the span, as a timestamp in seconds.

Returns:

The end time logged to the span metrics.

close

@abstractmethod
def close(end_time=None) -> float

Alias for end.

set_http_adapter

def set_http_adapter(adapter: HTTPAdapter)

Specify a custom HTTP adapter to use for all network requests. This is useful for setting custom retry policies, timeouts, etc.

Braintrust uses the requests library, so the adapter should be an instance of requests.adapters.HTTPAdapter.

Arguments:

• adapter: The adapter to use.

init

def init(project: Optional[str] = None,
         experiment: Optional[str] = None,
         description: Optional[str] = None,
         dataset: Optional["Dataset"] = None,
         open: bool = False,
         base_experiment: Optional[str] = None,
         is_public: bool = False,
         app_url: Optional[str] = None,
         api_key: Optional[str] = None,
         org_name: Optional[str] = None,
         metadata: Optional[Metadata] = None,
         git_metadata_settings: Optional[GitMetadataSettings] = None,
         set_current: bool = True,
         update: Optional[bool] = None,
         project_id: Optional[str] = None,
         base_experiment_id: Optional[str] = None,
         repo_info: Optional[RepoInfo] = None)

Log in, and then initialize a new experiment in a specified project. If the project does not exist, it will be created.

Arguments:

• project: The name of the project to create the experiment in. Must specify at least one of project or project_id.
• experiment: The name of the experiment to create. If not specified, a name will be generated automatically.
• description: (Optional) An optional description of the experiment.
• dataset: (Optional) A dataset to associate with the experiment. The dataset must be initialized with braintrust.init_dataset before passing it into the experiment.
• update: If the experiment already exists, continue logging to it.
• base_experiment: An optional experiment name to use as a base. If specified, the new experiment will be summarized and compared to this experiment. Otherwise, it will pick an experiment by finding the closest ancestor on the default (e.g. main) branch.
• is_public: An optional parameter to control whether the experiment is publicly visible to anybody with the link or privately visible to only members of the organization. Defaults to private.
• app_url: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.
• api_key: The API key to use. If the parameter is not specified, will try to use the BRAINTRUST_API_KEY environment variable. If no API key is specified, will prompt the user to login.
• org_name: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.
• metadata: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the prompt, example's id, or anything else that would be useful to slice/dice later. The values in metadata can be any JSON-serializable type, but its keys must be strings.
• git_metadata_settings: (Optional) Settings for collecting git metadata. By default, will collect all git metadata fields allowed in org-level settings.
• set_current: If true (the default), set the global current-experiment to the newly-created one.
• open: If the experiment already exists, open it in read-only mode.
• project_id: The id of the project to create the experiment in. This takes precedence over project if specified.
• base_experiment_id: An optional experiment id to use as a base. If specified, the new experiment will be summarized and compared to this. This takes precedence over base_experiment if specified.
• repo_info: (Optional) Explicitly specify the git metadata for this experiment. This takes precedence over git_metadata_settings if specified.

Returns:

The experiment object.

init_experiment

def init_experiment(*args, **kwargs)

Alias for init

init_dataset

def init_dataset(project: Optional[str] = None,
                 name: Optional[str] = None,
                 description: Optional[str] = None,
                 version: Optional[Union[str, int]] = None,
                 app_url: Optional[str] = None,
                 api_key: Optional[str] = None,
                 org_name: Optional[str] = None,
                 project_id: Optional[str] = None,
                 use_output: bool = DEFAULT_IS_LEGACY_DATASET)

Create a new dataset in a specified project. If the project does not exist, it will be created.

Arguments:

• project_name: The name of the project to create the dataset in. Must specify at least one of project_name or project_id.
• name: The name of the dataset to create. If not specified, a name will be generated automatically.
• description: An optional description of the dataset.
• version: An optional version of the dataset (to read). If not specified, the latest version will be used.
• app_url: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.
• api_key: The API key to use. If the parameter is not specified, will try to use the BRAINTRUST_API_KEY environment variable. If no API key is specified, will prompt the user to login.
• org_name: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.
• project_id: The id of the project to create the dataset in. This takes precedence over project if specified.
• use_output: If True (the default), records will be fetched from this dataset in the legacy format, with the "expected" field renamed to "output". This will default to False in a future version of Braintrust.

Returns:

The dataset object.

init_logger

def init_logger(project: Optional[str] = None,
                project_id: Optional[str] = None,
                async_flush: bool = True,
                app_url: Optional[str] = None,
                api_key: Optional[str] = None,
                org_name: Optional[str] = None,
                force_login: bool = False,
                set_current: bool = True)

Create a new logger in a specified project. If the project does not exist, it will be created.

Arguments:

• project: The name of the project to log into. If unspecified, will default to the Global project.
• project_id: The id of the project to log into. This takes precedence over project if specified.
• async_flush: If true (the default), log events will be batched and sent asynchronously in a background thread. If false, log events will be sent synchronously. Set to false in serverless environments.
• app_url: The URL of the Braintrust API. Defaults to https://www.braintrustdata.com.
• api_key: The API key to use. If the parameter is not specified, will try to use the BRAINTRUST_API_KEY environment variable. If no API key is specified, will prompt the user to login.
• org_name: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.
• force_login: Login again, even if you have already logged in (by default, the logger will not login if you are already logged in)
• set_current: If true (the default), set the global current-experiment to the newly-created one.

Returns:

The newly created Logger.

load_prompt

def load_prompt(project: Optional[str] = None,
                slug: Optional[str] = None,
                version: Optional[Union[str, int]] = None,
                project_id: Optional[str] = None,
                defaults: Optional[Dict[str, Any]] = None,
                no_trace: bool = False,
                app_url: Optional[str] = None,
                api_key: Optional[str] = None,
                org_name: Optional[str] = None)

Loads a prompt from the specified project.

Arguments:

• project: The name of the project to load the prompt from. Must specify at least one of project or project_id.
• slug: The slug of the prompt to load.
• version: An optional version of the prompt (to read). If not specified, the latest version will be used.
• project_id: The id of the project to load the prompt from. This takes precedence over project if specified.
• defaults: (Optional) A dictionary of default values to use when rendering the prompt. Prompt values will override these defaults.
• no_trace: If true, do not include logging metadata for this prompt when build() is called.
• app_url: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.
• api_key: The API key to use. If the parameter is not specified, will try to use the BRAINTRUST_API_KEY environment variable. If no API key is specified, will prompt the user to login.
• org_name: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.
• project_id: The id of the project to load the prompt from. This takes precedence over project if specified.

Returns:

The prompt object.

login

def login(app_url=None, api_key=None, org_name=None, force_login=False)

Log into Braintrust. This will prompt you for your API token, which you can find at

https://www.braintrustdata.com/app/token. This method is called automatically by init().

Arguments:

• app_url: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.
• api_key: The API key to use. If the parameter is not specified, will try to use the BRAINTRUST_API_KEY environment variable. If no API key is specified, will prompt the user to login.
• org_name: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.
• force_login: Login again, even if you have already logged in (by default, this function will exit quickly if you have already logged in)

log

def log(**event)

Log a single event to the current experiment. The event will be batched and uploaded behind the scenes.

Arguments:

• **event: Data to be logged. See Experiment.log for full details.

Returns:

The id of the logged event.

summarize

def summarize(summarize_scores=True, comparison_experiment_id=None)

Summarize the current experiment, including the scores (compared to the closest reference experiment) and metadata.

Arguments:

• summarize_scores: Whether to summarize the scores. If False, only the metadata will be returned.
• comparison_experiment_id: The experiment to compare against. If None, the most recent experiment on the comparison_commit will be used.

Returns:

ExperimentSummary

current_experiment

def current_experiment() -> Optional["Experiment"]

Returns the currently-active experiment (set by braintrust.init(...)). Returns None if no current experiment has been set.

current_logger

def current_logger() -> Optional["Logger"]

Returns the currently-active logger (set by braintrust.init_logger(...)). Returns None if no current logger has been set.

current_span

def current_span() -> Span

Return the currently-active span for logging (set by running a span under a context manager). If there is no active span, returns a no-op span object, which supports the same interface as spans but does no logging.

See Span for full details.

get_span_parent_object

def get_span_parent_object() -> Union["Logger", "Experiment", Span]

Mainly for internal use. Return the parent object for starting a span in a global context.

traced

def traced(*span_args, **span_kwargs)

Decorator to trace the wrapped function. Can either be applied bare (@traced) or by providing arguments (@traced(*span_args, **span_kwargs)), which will be forwarded to the created span. See Span.start_span for full details on the span arguments.

It checks the following (in precedence order): _ Currently-active span _ Currently-active experiment * Currently-active logger

and creates a span in the first one that is active. If none of these are active, it returns a no-op span object.

The decorator will automatically log the input and output of the wrapped function to the corresponding fields of the created span. Pass the kwarg notrace_io=True to the decorator to prevent this.

Unless a name is explicitly provided in span_args or span_kwargs, the name of the span will be the name of the decorated function.

start_span

def start_span(name=None,
               type: SpanTypeAttribute = None,
               span_attributes=None,
               start_time=None,
               set_current=None,
               parent=None,
               parent_id=None,
               **event) -> Span

Lower-level alternative to @traced for starting a span at the toplevel. It creates a span under the first active object (using the same precedence order as @traced), or if parent is specified, under the specified parent row, or returns a no-op span object.

We recommend running spans bound to a context manager (with start_span) to automatically mark them as current and ensure they are terminated. If you wish to start a span outside a context manager, be sure to terminate it with span.end().

See Span.start_span for full details.

ObjectFetcher Objects

class ObjectFetcher()

fetch

def fetch()

Fetch all records.

for record in object.fetch():
    print(record)
 
# You can also iterate over the object directly.
for record in object:
    print(record)
 
**Returns**:
 
An iterator over the records.
 
<a id="braintrust.logger.Experiment"></a>
 
## Experiment Objects
 
```python
class Experiment(ObjectFetcher)

An experiment is a collection of logged events, such as model inputs and outputs, which represent a snapshot of your application at a particular point in time. An experiment is meant to capture more than just the model you use, and includes the data you use to test, pre- and post- processing code, comparison metrics (scores), and any other metadata you want to include.

Experiments are associated with a project, and two experiments are meant to be easily comparable via their inputs. You can change the attributes of the experiments in a project (e.g. scoring functions) over time, simply by changing what you log.

You should not create Experiment objects directly. Instead, use the braintrust.init() method.

log

def log(input=None,
        output=None,
        expected=None,
        tags=None,
        scores=None,
        metadata=None,
        metrics=None,
        id=None,
        dataset_record_id=None,
        inputs=None,
        allow_concurrent_with_spans=False)

Log a single event to the experiment. The event will be batched and uploaded behind the scenes.

Arguments:

• input: The arguments that uniquely define a test case (an arbitrary, JSON serializable object). Later on, Braintrust will use the input to know whether two test cases are the same between experiments, so they should not contain experiment-specific state. A simple rule of thumb is that if you run the same experiment twice, the input should be identical.
• output: The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the output should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question.
• expected: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare to output to determine if your output value is correct or not. Braintrust currently does not compare output to expected for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate your experiments while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models.
• scores: A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare experiments.
• metadata: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the prompt, example's id, or anything else that would be useful to slice/dice later. The values in metadata can be any JSON-serializable type, but its keys must be strings.
• tags: (Optional) a list of strings that you can use to filter and group records later.
• metrics: (Optional) a dictionary of metrics to log. The following keys are populated automatically: "start", "end".
• id: (Optional) a unique identifier for the event. If you don't provide one, BrainTrust will generate one for you.
• dataset_record_id: (Optional) the id of the dataset record that this event is associated with. This field is required if and only if the experiment is associated with a dataset.
• inputs: (Deprecated) the same as input (will be removed in a future version).
• allow_concurrent_with_spans: (Optional) in rare cases where you need to log at the top level separately from using spans on the experiment elsewhere, set this to True.

Returns:

The id of the logged event.

log_feedback

def log_feedback(id,
                 scores=None,
                 expected=None,
                 tags=None,
                 comment=None,
                 metadata=None,
                 source=None)

Log feedback to an event in the experiment. Feedback is used to save feedback scores, set an expected value, or add a comment.

Arguments:

• id: The id of the event to log feedback for. This is the id returned by log or accessible as the id field of a span.
• scores: (Optional) a dictionary of numeric values (between 0 and 1) to log. These scores will be merged into the existing scores for the event.
• expected: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare to output to determine if your output value is correct or not.
• tags: (Optional) a list of strings that you can use to filter and group records later.
• comment: (Optional) an optional comment string to log about the event.
• metadata: (Optional) a dictionary with additional data about the feedback. If you have a user_id, you can log it here and access it in the Braintrust UI.
• source: (Optional) the source of the feedback. Must be one of "external" (default), "app", or "api".

start_span

def start_span(name=None,
               type=None,
               span_attributes=None,
               start_time=None,
               set_current=None,
               parent=None,
               parent_id=None,
               **event)

Create a new toplevel span underneath the experiment. The name defaults to "root" and the span type to "eval".

See Span.start_span for full details

summarize

def summarize(summarize_scores=True, comparison_experiment_id=None)

Summarize the experiment, including the scores (compared to the closest reference experiment) and metadata.

Arguments:

• summarize_scores: Whether to summarize the scores. If False, only the metadata will be returned.
• comparison_experiment_id: The experiment to compare against. If None, the most recent experiment on the origin's main branch will be used.

Returns:

ExperimentSummary

export

def export() -> str

Return a serialized representation of the experiment that can be used to start subspans in other places. See Span.start_span for more details.

close

def close()

This function is deprecated. You can simply remove it from your code.

flush

def flush()

Flush any pending rows to the server.

ReadonlyExperiment Objects

class ReadonlyExperiment(ObjectFetcher)

A read-only view of an experiment, initialized by passing open=True to init().

SpanImpl Objects

class SpanImpl(Span)

Primary implementation of the Span interface. See the Span interface for full details on each method.

We suggest using one of the various start_span methods, instead of creating Spans directly. See Span.start_span for full details.

export

def export() -> str

Return a serialized representation of the span that can be used to start subspans in other places. See Span.start_span for more details.

flush

def flush()

Flush any pending rows to the server.

Dataset Objects

class Dataset(ObjectFetcher)

A dataset is a collection of records, such as model inputs and outputs, which represent data you can use to evaluate and fine-tune models. You can log production data to datasets, curate them with interesting examples, edit/delete records, and run evaluations against them.

You should not create Dataset objects directly. Instead, use the braintrust.init_dataset() method.

insert

def insert(input,
           expected=None,
           tags=None,
           metadata=None,
           id=None,
           output=None)

Insert a single record to the dataset. The record will be batched and uploaded behind the scenes. If you pass in an id,

and a record with that id already exists, it will be overwritten (upsert).

Arguments:

• input: The argument that uniquely define an input case (an arbitrary, JSON serializable object).
• expected: The output of your application, including post-processing (an arbitrary, JSON serializable object).
• tags: (Optional) a list of strings that you can use to filter and group records later.
• metadata: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the prompt, example's id, or anything else that would be useful to slice/dice later. The values in metadata can be any JSON-serializable type, but its keys must be strings.
• id: (Optional) a unique identifier for the event. If you don't provide one, Braintrust will generate one for you.
• output: (Deprecated) The output of your application. Use expected instead.

Returns:

The id of the logged record.

delete

def delete(id)

Delete a record from the dataset.

Arguments:

• id: The id of the record to delete.

summarize

def summarize(summarize_data=True)

Summarize the dataset, including high level metrics about its size and other metadata.

Arguments:

• summarize_data: Whether to summarize the data. If False, only the metadata will be returned.

Returns:

DatasetSummary

close

def close()

This function is deprecated. You can simply remove it from your code.

flush

def flush()

Flush any pending rows to the server.

Prompt Objects

class Prompt()

A prompt object consists of prompt text, a model, and model parameters (such as temperature), which can be used to generate completions or chat messages. The prompt object supports calling .build() which uses mustache templating to build the prompt with the given formatting options and returns a plain dictionary that includes the built prompt and arguments. The dictionary can be passed as kwargs to the OpenAI client or modified as you see fit.

You should not create Prompt objects directly. Instead, use the braintrust.load_prompt() method.

build

def build(**build_args)

Build the prompt with the given formatting options. The args you pass in will

be forwarded to the mustache template that defines the prompt and rendered with the chevron library.

Returns:

A dictionary that includes the rendered prompt and arguments, that can be passed as kwargs to the OpenAI client.

Logger Objects

class Logger()

log

def log(input=None,
        output=None,
        expected=None,
        tags=None,
        scores=None,
        metadata=None,
        metrics=None,
        id=None,
        allow_concurrent_with_spans=False)

Log a single event. The event will be batched and uploaded behind the scenes.

Arguments:

• input: (Optional) the arguments that uniquely define a user input(an arbitrary, JSON serializable object).
• output: (Optional) the output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, the output should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question.
• expected: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare to output to determine if your output value is correct or not. Braintrust currently does not compare output to expected for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models.
• tags: (Optional) a list of strings that you can use to filter and group records later.
• scores: (Optional) a dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare logs.
• metadata: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the prompt, example's id, or anything else that would be useful to slice/dice later. The values in metadata can be any JSON-serializable type, but its keys must be strings.
• metrics: (Optional) a dictionary of metrics to log. The following keys are populated automatically: "start", "end".
• id: (Optional) a unique identifier for the event. If you don't provide one, BrainTrust will generate one for you.
• allow_concurrent_with_spans: (Optional) in rare cases where you need to log at the top level separately from using spans on the logger elsewhere, set this to True.

log_feedback

def log_feedback(id,
                 scores=None,
                 expected=None,
                 tags=None,
                 comment=None,
                 metadata=None,
                 source=None)

Log feedback to an event. Feedback is used to save feedback scores, set an expected value, or add a comment.

Arguments:

• id: The id of the event to log feedback for. This is the id returned by log or accessible as the id field of a span.
• scores: (Optional) a dictionary of numeric values (between 0 and 1) to log. These scores will be merged into the existing scores for the event.
• expected: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare to output to determine if your output value is correct or not.
• tags: (Optional) a list of strings that you can use to filter and group records later.
• comment: (Optional) an optional comment string to log about the event.
• metadata: (Optional) a dictionary with additional data about the feedback. If you have a user_id, you can log it here and access it in the Braintrust UI.
• source: (Optional) the source of the feedback. Must be one of "external" (default), "app", or "api".

start_span

def start_span(name=None,
               type=None,
               span_attributes=None,
               start_time=None,
               set_current=None,
               parent=None,
               parent_id=None,
               **event)

Create a new toplevel span underneath the logger. The name defaults to "root" and the span type to "task".

See Span.start_span for full details

export

def export() -> str

Return a serialized representation of the logger that can be used to start subspans in other places. See Span.start_span for more details.

flush

def flush()

Flush any pending logs to the server.

ScoreSummary Objects

@dataclasses.dataclass
class ScoreSummary(SerializableDataClass)

Summary of a score's performance.

name

Name of the score.

score

Average score across all examples.

diff

Difference in score between the current and reference experiment.

improvements

Number of improvements in the score.

regressions

Number of regressions in the score.

MetricSummary Objects

@dataclasses.dataclass
class MetricSummary(SerializableDataClass)

Summary of a metric's performance.

name

Name of the metric.

metric

Average metric across all examples.

unit

Unit label for the metric.

diff

Difference in metric between the current and reference experiment.

improvements

Number of improvements in the metric.

regressions

Number of regressions in the metric.

ExperimentSummary Objects

@dataclasses.dataclass
class ExperimentSummary(SerializableDataClass)

Summary of an experiment's scores and metadata.

project_name

Name of the project that the experiment belongs to.

experiment_name

Name of the experiment.

project_url

URL to the project's page in the Braintrust app.

experiment_url

URL to the experiment's page in the Braintrust app.

comparison_experiment_name

The experiment scores are baselined against.

scores

Summary of the experiment's scores.

metrics

Summary of the experiment's metrics.

DataSummary Objects

@dataclasses.dataclass
class DataSummary(SerializableDataClass)

Summary of a dataset's data.

new_records

New or updated records added in this session.

total_records

Total records in the dataset.

DatasetSummary Objects

@dataclasses.dataclass
class DatasetSummary(SerializableDataClass)

Summary of a dataset's scores and metadata.

project_name

Name of the project that the dataset belongs to.

dataset_name

Name of the dataset.

project_url

URL to the project's page in the Braintrust app.

dataset_url

URL to the experiment's page in the Braintrust app.

data_summary

Summary of the dataset's data.

braintrust.framework

EvalCase Objects

@dataclasses.dataclass(kw_only=True)
class EvalCase(SerializableDataClass)

An evaluation case. This is a single input to the evaluation task, along with an optional expected output, metadata, and tags.

EvalResult Objects

@dataclasses.dataclass(kw_only=True)
class EvalResult(SerializableDataClass)

The result of an evaluation. This includes the input, expected output, actual output, and metadata.

EvalHooks Objects

class EvalHooks(abc.ABC)

An object that can be used to add metadata to an evaluation. This is passed to the task function.

span

@property
@abc.abstractmethod
def span() -> Span

Access the span under which the task is run. Also accessible via braintrust.current_span()

meta

@abc.abstractmethod
def meta(**info) -> None

Adds metadata to the evaluation. This metadata will be logged to the Braintrust. You can pass in metadaa as keyword arguments, e.g. hooks.meta(foo="bar").

EvalScorerArgs Objects

class EvalScorerArgs(SerializableDataClass)

Arguments passed to an evaluator scorer. This includes the input, expected output, actual output, and metadata.

BaseExperiment Objects

@dataclasses.dataclass
class BaseExperiment()

Use this to specify that the dataset should actually be the data from a previous (base) experiment. If you do not specify a name, Braintrust will automatically figure out the best base experiment to use based on your git history (or fall back to timestamps).

name

The name of the base experiment to use. If unspecified, Braintrust will automatically figure out the best base using your git history (or fall back to timestamps).

Evaluator Objects

@dataclasses.dataclass
class Evaluator()

An evaluator is an abstraction that defines an evaluation dataset, a task to run on the dataset, and a set of scorers to evaluate the results of the task. Each method attribute can be synchronous or asynchronous (for optimal performance, it is recommended to provide asynchronous implementations).

You should not create Evaluators directly if you plan to use the Braintrust eval framework. Instead, you should create them using the Eval() method, which will register them so that braintrust eval ... can find them.

project_name

The name of the project the eval falls under.

eval_name

A name that describes the experiment. You do not need to change it each time the experiment runs.

data

Returns an iterator over the evaluation dataset. Each element of the iterator should be an EvalCase or a dict with the same fields as an EvalCase (input, expected, metadata).

task

Runs the evaluation task on a single input. The hooks object can be used to add metadata to the evaluation.

scores

A list of scorers to evaluate the results of the task. Each scorer can be a Scorer object or a function that takes input, output, and expected arguments and returns a Score object. The function can be async.

experiment_name

Optional experiment name. If not specified, a name will be generated automatically.

metadata

A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the prompt, example's id, or anything else that would be useful to slice/dice later. The values in metadata can be any JSON-serializable type, but its keys must be strings.

trial_count

The number of times to run the evaluator per input. This is useful for evaluating applications that have non-deterministic behavior and gives you both a stronger aggregate measure and a sense of the variance in the results.

is_public

Whether the experiment should be public. Defaults to false.

ReporterDef Objects

@dataclasses.dataclass
class ReporterDef(SerializableDataClass)

A reporter takes an evaluator and its result and returns a report.

name

The name of the reporter.

report_eval

A function that takes an evaluator and its result and returns a report.

report_run

A function that takes all evaluator results and returns a boolean indicating whether the run was successful. If you return false, the braintrust eval command will exit with a non-zero status code.

Eval

def Eval(name: str,
         data: Callable[[], Union[Iterator[EvalCase],
                                  AsyncIterator[EvalCase]]],
         task: Callable[[Input, EvalHooks], Union[Output, Awaitable[Output]]],
         scores: List[EvalScorer],
         experiment_name: Optional[str] = None,
         trial_count: int = 1,
         metadata: Optional[Metadata] = None,
         is_public: bool = False,
         reporter: Optional[Union[ReporterDef, str]] = None)

A function you can use to define an evaluator. This is a convenience wrapper around the Evaluator class.

Example:

Eval(
    name="my-evaluator",
    data=lambda: [
        EvalCase(input=1, expected=2),
        EvalCase(input=2, expected=4),
    ],
    task=lambda input, hooks: input * 2,
    scores=[
        NumericDiff,
    ],
)

Arguments:

• name: The name of the evaluator. This corresponds to a project name in Braintrust.
• data: Returns an iterator over the evaluation dataset. Each element of the iterator should be a EvalCase.
• task: Runs the evaluation task on a single input. The hooks object can be used to add metadata to the evaluation.
• scores: A list of scorers to evaluate the results of the task. Each scorer can be a Scorer object or a function that takes an EvalScorerArgs object and returns a Score object.
• experiment_name: (Optional) Experiment name. If not specified, a name will be generated automatically.
• trial_count: The number of times to run the evaluator per input. This is useful for evaluating applications that have non-deterministic behavior and gives you both a stronger aggregate measure and a sense of the variance in the results.
• metadata: (Optional) A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log the prompt, example's id, or anything else that would be useful to slice/dice later. The values in metadata can be any JSON-serializable type, but its keys must be strings.
• is_public: (Optional) Whether the experiment should be public. Defaults to false.
• reporter: (Optional) A reporter that takes an evaluator and its result and returns a report.

Returns:

An Evaluator object.

Reporter

def Reporter(
    name: str,
    report_eval: Callable[[Evaluator, EvalResultWithSummary, bool, bool],
                          Union[EvalReport, Awaitable[EvalReport]]],
    report_run: Callable[[List[EvalReport], bool, bool],
                         Union[bool, Awaitable[bool]]])

A function you can use to define a reporter. This is a convenience wrapper around the ReporterDef class.

Example:

def report_eval(evaluator, result, verbose, jsonl):
    return str(result.summary)
 
def report_run(results, verbose, jsonl):
    return True
 
Reporter(
    name="my-reporter",
    report_eval=report_eval,
    report_run=report_run,
)

Arguments:

• name: The name of the reporter.
• report_eval: A function that takes an evaluator and its result and returns a report.
• report_run: A function that takes all evaluator results and returns a boolean indicating whether the run was successful.

set_thread_pool_max_workers

def set_thread_pool_max_workers(max_workers)

Set the maximum number of threads to use for running evaluators. By default, this is the number of CPUs on the machine.