braintrust
A Python library for logging data to Braintrust. braintrust
is distributed as
a library on PyPI. It is open source and
available on GitHub.
Quickstart
Install the library with pip.
pip install braintrust
Then, create a file like eval_hello.py
with the following content:
from braintrust import Eval
def is_equal(expected, output):
return expected == output
Eval(
"Say Hi Bot",
data=lambda: [
{
"input": "Foo",
"expected": "Hi Foo",
},
{
"input": "Bar",
"expected": "Hello Bar",
},
], # Replace with your eval dataset
task=lambda input: "Hi " + input, # Replace with your LLM call
scores=[is_equal],
)
Finally, run the script with braintrust eval eval_hello.py
.
BRAINTRUST_API_KEY=<YOUR_BRAINTRUST_API_KEY> braintrust eval eval_hello.py
API Reference
braintrust.logger
Span Objects
class Span(ABC)
A Span encapsulates logged data and metrics for a unit of work. This interface is shared by all span implementations.
We suggest using one of the various start_span
methods, instead of creating Spans directly. See Span.start_span
for full details.
id
@property
@abstractmethod
def id() -> str
Row ID of the span.
log
@abstractmethod
def log(**event)
Incrementally update the current span with new data. The event will be batched and uploaded behind the scenes.
Arguments:
**event
: Data to be logged. SeeExperiment.log
for full details.
log_feedback
@abstractmethod
def log_feedback(**event)
Add feedback to the current span. Unlike Experiment.log_feedback
and Logger.log_feedback
, this method does not accept an id parameter, because it logs feedback to the current span.
Arguments:
**event
: Data to be logged. SeeExperiment.log_feedback
for full details.
start_span
@abstractmethod
def start_span(name=None,
type=None,
span_attributes=None,
start_time=None,
set_current=None,
parent=None,
parent_id=None,
**event)
Create a new span. This is useful if you want to log more detailed trace information beyond the scope of a single log event. Data logged over several calls to Span.log
will be merged into one logical row.
We recommend running spans within context managers (with start_span(...) as span
) to automatically mark them as current and ensure they are ended. Only spans run within a context manager will be marked current, so they can be accessed using braintrust.current_span()
. If you wish to start a span outside a context manager, be sure to end it with span.end()
.
Arguments:
name
: Optional name of the span. If not provided, a name will be inferred from the call stack.type
: Optional type of the span. Use theSpanTypeAttribute
enum or just provide a string directly. If not provided, the type will be unset.span_attributes
: Optional additional attributes to attach to the span, such as a type name.start_time
: Optional start time of the span, as a timestamp in seconds.set_current
: If true (the default), the span will be marked as the currently-active span for the duration of the context manager.parent
: Optional parent info string for the span. The string can be generated from[Span,Experiment,Logger].export
. If not provided, the current span will be used (depending on context). This is useful for adding spans to an existing trace.parent_id
: This option is deprecated and will be removed in a future version of Braintrust. Prefer to useparent
instead.**event
: Data to be logged. SeeExperiment.log
for full details.
Returns:
The newly-created Span
end
@abstractmethod
def end(end_time=None) -> float
Log an end time to the span (defaults to the current time). Returns the logged time.
Will be invoked automatically if the span is bound to a context manager.
Arguments:
end_time
: Optional end time of the span, as a timestamp in seconds.
Returns:
The end time logged to the span metrics.
close
@abstractmethod
def close(end_time=None) -> float
Alias for end
.
set_http_adapter
def set_http_adapter(adapter: HTTPAdapter)
Specify a custom HTTP adapter to use for all network requests. This is useful for setting custom retry policies, timeouts, etc.
Braintrust uses the requests
library, so the adapter should be an instance of requests.adapters.HTTPAdapter
.
Arguments:
adapter
: The adapter to use.
init
def init(project: Optional[str] = None,
experiment: Optional[str] = None,
description: Optional[str] = None,
dataset: Optional["Dataset"] = None,
open: bool = False,
base_experiment: Optional[str] = None,
is_public: bool = False,
app_url: Optional[str] = None,
api_key: Optional[str] = None,
org_name: Optional[str] = None,
metadata: Optional[Metadata] = None,
git_metadata_settings: Optional[GitMetadataSettings] = None,
set_current: bool = True,
update: Optional[bool] = None,
project_id: Optional[str] = None,
base_experiment_id: Optional[str] = None,
repo_info: Optional[RepoInfo] = None)
Log in, and then initialize a new experiment in a specified project. If the project does not exist, it will be created.
Arguments:
project
: The name of the project to create the experiment in. Must specify at least one ofproject
orproject_id
.experiment
: The name of the experiment to create. If not specified, a name will be generated automatically.description
: (Optional) An optional description of the experiment.dataset
: (Optional) A dataset to associate with the experiment. The dataset must be initialized withbraintrust.init_dataset
before passing it into the experiment.update
: If the experiment already exists, continue logging to it.base_experiment
: An optional experiment name to use as a base. If specified, the new experiment will be summarized and compared to this experiment. Otherwise, it will pick an experiment by finding the closest ancestor on the default (e.g. main) branch.is_public
: An optional parameter to control whether the experiment is publicly visible to anybody with the link or privately visible to only members of the organization. Defaults to private.app_url
: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.api_key
: The API key to use. If the parameter is not specified, will try to use theBRAINTRUST_API_KEY
environment variable. If no API key is specified, will prompt the user to login.org_name
: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.metadata
: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log theprompt
, example'sid
, or anything else that would be useful to slice/dice later. The values inmetadata
can be any JSON-serializable type, but its keys must be strings.git_metadata_settings
: (Optional) Settings for collecting git metadata. By default, will collect all git metadata fields allowed in org-level settings.set_current
: If true (the default), set the global current-experiment to the newly-created one.open
: If the experiment already exists, open it in read-only mode.project_id
: The id of the project to create the experiment in. This takes precedence overproject
if specified.base_experiment_id
: An optional experiment id to use as a base. If specified, the new experiment will be summarized and compared to this. This takes precedence overbase_experiment
if specified.repo_info
: (Optional) Explicitly specify the git metadata for this experiment. This takes precedence overgit_metadata_settings
if specified.
Returns:
The experiment object.
init_experiment
def init_experiment(*args, **kwargs)
Alias for init
init_dataset
def init_dataset(project: Optional[str] = None,
name: Optional[str] = None,
description: Optional[str] = None,
version: Optional[Union[str, int]] = None,
app_url: Optional[str] = None,
api_key: Optional[str] = None,
org_name: Optional[str] = None,
project_id: Optional[str] = None,
use_output: bool = DEFAULT_IS_LEGACY_DATASET)
Create a new dataset in a specified project. If the project does not exist, it will be created.
Arguments:
project_name
: The name of the project to create the dataset in. Must specify at least one ofproject_name
orproject_id
.name
: The name of the dataset to create. If not specified, a name will be generated automatically.description
: An optional description of the dataset.version
: An optional version of the dataset (to read). If not specified, the latest version will be used.app_url
: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.api_key
: The API key to use. If the parameter is not specified, will try to use theBRAINTRUST_API_KEY
environment variable. If no API key is specified, will prompt the user to login.org_name
: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.project_id
: The id of the project to create the dataset in. This takes precedence overproject
if specified.use_output
: If True (the default), records will be fetched from this dataset in the legacy format, with the "expected" field renamed to "output". This will default to False in a future version of Braintrust.
Returns:
The dataset object.
init_logger
def init_logger(project: Optional[str] = None,
project_id: Optional[str] = None,
async_flush: bool = True,
app_url: Optional[str] = None,
api_key: Optional[str] = None,
org_name: Optional[str] = None,
force_login: bool = False,
set_current: bool = True)
Create a new logger in a specified project. If the project does not exist, it will be created.
Arguments:
project
: The name of the project to log into. If unspecified, will default to the Global project.project_id
: The id of the project to log into. This takes precedence over project if specified.async_flush
: If true (the default), log events will be batched and sent asynchronously in a background thread. If false, log events will be sent synchronously. Set to false in serverless environments.app_url
: The URL of the Braintrust API. Defaults to https://www.braintrustdata.com.api_key
: The API key to use. If the parameter is not specified, will try to use theBRAINTRUST_API_KEY
environment variable. If no API key is specified, will prompt the user to login.org_name
: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.force_login
: Login again, even if you have already logged in (by default, the logger will not login if you are already logged in)set_current
: If true (the default), set the global current-experiment to the newly-created one.
Returns:
The newly created Logger.
load_prompt
def load_prompt(project: Optional[str] = None,
slug: Optional[str] = None,
version: Optional[Union[str, int]] = None,
project_id: Optional[str] = None,
defaults: Optional[Dict[str, Any]] = None,
no_trace: bool = False,
app_url: Optional[str] = None,
api_key: Optional[str] = None,
org_name: Optional[str] = None)
Loads a prompt from the specified project.
Arguments:
project
: The name of the project to load the prompt from. Must specify at least one ofproject
orproject_id
.slug
: The slug of the prompt to load.version
: An optional version of the prompt (to read). If not specified, the latest version will be used.project_id
: The id of the project to load the prompt from. This takes precedence overproject
if specified.defaults
: (Optional) A dictionary of default values to use when rendering the prompt. Prompt values will override these defaults.no_trace
: If true, do not include logging metadata for this prompt when build() is called.app_url
: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.api_key
: The API key to use. If the parameter is not specified, will try to use theBRAINTRUST_API_KEY
environment variable. If no API key is specified, will prompt the user to login.org_name
: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.project_id
: The id of the project to load the prompt from. This takes precedence overproject
if specified.
Returns:
The prompt object.
login
def login(app_url=None, api_key=None, org_name=None, force_login=False)
Log into Braintrust. This will prompt you for your API token, which you can find at
https://www.braintrustdata.com/app/token. This method is called automatically by init()
.
Arguments:
app_url
: The URL of the Braintrust App. Defaults to https://www.braintrustdata.com.api_key
: The API key to use. If the parameter is not specified, will try to use theBRAINTRUST_API_KEY
environment variable. If no API key is specified, will prompt the user to login.org_name
: (Optional) The name of a specific organization to connect to. This is useful if you belong to multiple.force_login
: Login again, even if you have already logged in (by default, this function will exit quickly if you have already logged in)
log
def log(**event)
Log a single event to the current experiment. The event will be batched and uploaded behind the scenes.
Arguments:
**event
: Data to be logged. SeeExperiment.log
for full details.
Returns:
The id
of the logged event.
summarize
def summarize(summarize_scores=True, comparison_experiment_id=None)
Summarize the current experiment, including the scores (compared to the closest reference experiment) and metadata.
Arguments:
summarize_scores
: Whether to summarize the scores. If False, only the metadata will be returned.comparison_experiment_id
: The experiment to compare against. If None, the most recent experiment on the comparison_commit will be used.
Returns:
ExperimentSummary
current_experiment
def current_experiment() -> Optional["Experiment"]
Returns the currently-active experiment (set by braintrust.init(...)
). Returns None if no current experiment has been set.
current_logger
def current_logger() -> Optional["Logger"]
Returns the currently-active logger (set by braintrust.init_logger(...)
). Returns None if no current logger has been set.
current_span
def current_span() -> Span
Return the currently-active span for logging (set by running a span under a context manager). If there is no active span, returns a no-op span object, which supports the same interface as spans but does no logging.
See Span
for full details.
get_span_parent_object
def get_span_parent_object() -> Union["Logger", "Experiment", Span]
Mainly for internal use. Return the parent object for starting a span in a global context.
traced
def traced(*span_args, **span_kwargs)
Decorator to trace the wrapped function. Can either be applied bare (@traced
) or by providing arguments (@traced(*span_args, **span_kwargs)
), which will be forwarded to the created span. See Span.start_span
for full details on the span arguments.
It checks the following (in precedence order): _ Currently-active span _ Currently-active experiment * Currently-active logger
and creates a span in the first one that is active. If none of these are active, it returns a no-op span object.
The decorator will automatically log the input and output of the wrapped function to the corresponding fields of the created span. Pass the kwarg notrace_io=True
to the decorator to prevent this.
Unless a name is explicitly provided in span_args
or span_kwargs
, the name of the span will be the name of the decorated function.
start_span
def start_span(name=None,
type: SpanTypeAttribute = None,
span_attributes=None,
start_time=None,
set_current=None,
parent=None,
parent_id=None,
**event) -> Span
Lower-level alternative to @traced
for starting a span at the toplevel. It creates a span under the first active object (using the same precedence order as @traced
), or if parent
is specified, under the specified parent row, or returns a no-op span object.
We recommend running spans bound to a context manager (with start_span
) to automatically mark them as current and ensure they are terminated. If you wish to start a span outside a context manager, be sure to terminate it with span.end()
.
See Span.start_span
for full details.
ObjectFetcher Objects
class ObjectFetcher()
fetch
def fetch()
Fetch all records.
for record in object.fetch():
print(record)
# You can also iterate over the object directly.
for record in object:
print(record)
**Returns**:
An iterator over the records.
<a id="braintrust.logger.Experiment"></a>
## Experiment Objects
```python
class Experiment(ObjectFetcher)
An experiment is a collection of logged events, such as model inputs and outputs, which represent a snapshot of your application at a particular point in time. An experiment is meant to capture more than just the model you use, and includes the data you use to test, pre- and post- processing code, comparison metrics (scores), and any other metadata you want to include.
Experiments are associated with a project, and two experiments are meant to be easily comparable via
their inputs
. You can change the attributes of the experiments in a project (e.g. scoring functions)
over time, simply by changing what you log.
You should not create Experiment
objects directly. Instead, use the braintrust.init()
method.
log
def log(input=None,
output=None,
expected=None,
tags=None,
scores=None,
metadata=None,
metrics=None,
id=None,
dataset_record_id=None,
inputs=None,
allow_concurrent_with_spans=False)
Log a single event to the experiment. The event will be batched and uploaded behind the scenes.
Arguments:
input
: The arguments that uniquely define a test case (an arbitrary, JSON serializable object). Later on, Braintrust will use theinput
to know whether two test cases are the same between experiments, so they should not contain experiment-specific state. A simple rule of thumb is that if you run the same experiment twice, theinput
should be identical.output
: The output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, theoutput
should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question.expected
: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare tooutput
to determine if youroutput
value is correct or not. Braintrust currently does not compareoutput
toexpected
for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate your experiments while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models.scores
: A dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare experiments.metadata
: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log theprompt
, example'sid
, or anything else that would be useful to slice/dice later. The values inmetadata
can be any JSON-serializable type, but its keys must be strings.tags
: (Optional) a list of strings that you can use to filter and group records later.metrics
: (Optional) a dictionary of metrics to log. The following keys are populated automatically: "start", "end".id
: (Optional) a unique identifier for the event. If you don't provide one, BrainTrust will generate one for you.dataset_record_id
: (Optional) the id of the dataset record that this event is associated with. This field is required if and only if the experiment is associated with a dataset.inputs
: (Deprecated) the same asinput
(will be removed in a future version).allow_concurrent_with_spans
: (Optional) in rare cases where you need to log at the top level separately from using spans on the experiment elsewhere, set this to True.
Returns:
The id
of the logged event.
log_feedback
def log_feedback(id,
scores=None,
expected=None,
tags=None,
comment=None,
metadata=None,
source=None)
Log feedback to an event in the experiment. Feedback is used to save feedback scores, set an expected value, or add a comment.
Arguments:
id
: The id of the event to log feedback for. This is theid
returned bylog
or accessible as theid
field of a span.scores
: (Optional) a dictionary of numeric values (between 0 and 1) to log. These scores will be merged into the existing scores for the event.expected
: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare tooutput
to determine if youroutput
value is correct or not.tags
: (Optional) a list of strings that you can use to filter and group records later.comment
: (Optional) an optional comment string to log about the event.metadata
: (Optional) a dictionary with additional data about the feedback. If you have auser_id
, you can log it here and access it in the Braintrust UI.source
: (Optional) the source of the feedback. Must be one of "external" (default), "app", or "api".
start_span
def start_span(name=None,
type=None,
span_attributes=None,
start_time=None,
set_current=None,
parent=None,
parent_id=None,
**event)
Create a new toplevel span underneath the experiment. The name defaults to "root" and the span type to "eval".
See Span.start_span
for full details
summarize
def summarize(summarize_scores=True, comparison_experiment_id=None)
Summarize the experiment, including the scores (compared to the closest reference experiment) and metadata.
Arguments:
summarize_scores
: Whether to summarize the scores. If False, only the metadata will be returned.comparison_experiment_id
: The experiment to compare against. If None, the most recent experiment on the origin's main branch will be used.
Returns:
ExperimentSummary
export
def export() -> str
Return a serialized representation of the experiment that can be used to start subspans in other places. See Span.start_span
for more details.
close
def close()
This function is deprecated. You can simply remove it from your code.
flush
def flush()
Flush any pending rows to the server.
ReadonlyExperiment Objects
class ReadonlyExperiment(ObjectFetcher)
A read-only view of an experiment, initialized by passing open=True
to init()
.
SpanImpl Objects
class SpanImpl(Span)
Primary implementation of the Span
interface. See the Span
interface for full details on each method.
We suggest using one of the various start_span
methods, instead of creating Spans directly. See Span.start_span
for full details.
export
def export() -> str
Return a serialized representation of the span that can be used to start subspans in other places. See Span.start_span
for more details.
flush
def flush()
Flush any pending rows to the server.
Dataset Objects
class Dataset(ObjectFetcher)
A dataset is a collection of records, such as model inputs and outputs, which represent data you can use to evaluate and fine-tune models. You can log production data to datasets, curate them with interesting examples, edit/delete records, and run evaluations against them.
You should not create Dataset
objects directly. Instead, use the braintrust.init_dataset()
method.
insert
def insert(input,
expected=None,
tags=None,
metadata=None,
id=None,
output=None)
Insert a single record to the dataset. The record will be batched and uploaded behind the scenes. If you pass in an id
,
and a record with that id
already exists, it will be overwritten (upsert).
Arguments:
input
: The argument that uniquely define an input case (an arbitrary, JSON serializable object).expected
: The output of your application, including post-processing (an arbitrary, JSON serializable object).tags
: (Optional) a list of strings that you can use to filter and group records later.metadata
: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log theprompt
, example'sid
, or anything else that would be useful to slice/dice later. The values inmetadata
can be any JSON-serializable type, but its keys must be strings.id
: (Optional) a unique identifier for the event. If you don't provide one, Braintrust will generate one for you.output
: (Deprecated) The output of your application. Useexpected
instead.
Returns:
The id
of the logged record.
delete
def delete(id)
Delete a record from the dataset.
Arguments:
id
: Theid
of the record to delete.
summarize
def summarize(summarize_data=True)
Summarize the dataset, including high level metrics about its size and other metadata.
Arguments:
summarize_data
: Whether to summarize the data. If False, only the metadata will be returned.
Returns:
DatasetSummary
close
def close()
This function is deprecated. You can simply remove it from your code.
flush
def flush()
Flush any pending rows to the server.
Prompt Objects
class Prompt()
A prompt object consists of prompt text, a model, and model parameters (such as temperature), which
can be used to generate completions or chat messages. The prompt object supports calling .build()
which uses mustache templating to build the prompt with the given formatting options and returns a
plain dictionary that includes the built prompt and arguments. The dictionary can be passed as
kwargs to the OpenAI client or modified as you see fit.
You should not create Prompt
objects directly. Instead, use the braintrust.load_prompt()
method.
build
def build(**build_args)
Build the prompt with the given formatting options. The args you pass in will
be forwarded to the mustache template that defines the prompt and rendered with
the chevron
library.
Returns:
A dictionary that includes the rendered prompt and arguments, that can be passed as kwargs to the OpenAI client.
Logger Objects
class Logger()
log
def log(input=None,
output=None,
expected=None,
tags=None,
scores=None,
metadata=None,
metrics=None,
id=None,
allow_concurrent_with_spans=False)
Log a single event. The event will be batched and uploaded behind the scenes.
Arguments:
input
: (Optional) the arguments that uniquely define a user input(an arbitrary, JSON serializable object).output
: (Optional) the output of your application, including post-processing (an arbitrary, JSON serializable object), that allows you to determine whether the result is correct or not. For example, in an app that generates SQL queries, theoutput
should be the result of the SQL query generated by the model, not the query itself, because there may be multiple valid queries that answer a single question.expected
: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare tooutput
to determine if youroutput
value is correct or not. Braintrust currently does not compareoutput
toexpected
for you, since there are so many different ways to do that correctly. Instead, these values are just used to help you navigate while digging into analyses. However, we may later use these values to re-score outputs or fine-tune your models.tags
: (Optional) a list of strings that you can use to filter and group records later.scores
: (Optional) a dictionary of numeric values (between 0 and 1) to log. The scores should give you a variety of signals that help you determine how accurate the outputs are compared to what you expect and diagnose failures. For example, a summarization app might have one score that tells you how accurate the summary is, and another that measures the word similarity between the generated and grouth truth summary. The word similarity score could help you determine whether the summarization was covering similar concepts or not. You can use these scores to help you sort, filter, and compare logs.metadata
: (Optional) a dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log theprompt
, example'sid
, or anything else that would be useful to slice/dice later. The values inmetadata
can be any JSON-serializable type, but its keys must be strings.metrics
: (Optional) a dictionary of metrics to log. The following keys are populated automatically: "start", "end".id
: (Optional) a unique identifier for the event. If you don't provide one, BrainTrust will generate one for you.allow_concurrent_with_spans
: (Optional) in rare cases where you need to log at the top level separately from using spans on the logger elsewhere, set this to True.
log_feedback
def log_feedback(id,
scores=None,
expected=None,
tags=None,
comment=None,
metadata=None,
source=None)
Log feedback to an event. Feedback is used to save feedback scores, set an expected value, or add a comment.
Arguments:
id
: The id of the event to log feedback for. This is theid
returned bylog
or accessible as theid
field of a span.scores
: (Optional) a dictionary of numeric values (between 0 and 1) to log. These scores will be merged into the existing scores for the event.expected
: (Optional) the ground truth value (an arbitrary, JSON serializable object) that you'd compare tooutput
to determine if youroutput
value is correct or not.tags
: (Optional) a list of strings that you can use to filter and group records later.comment
: (Optional) an optional comment string to log about the event.metadata
: (Optional) a dictionary with additional data about the feedback. If you have auser_id
, you can log it here and access it in the Braintrust UI.source
: (Optional) the source of the feedback. Must be one of "external" (default), "app", or "api".
start_span
def start_span(name=None,
type=None,
span_attributes=None,
start_time=None,
set_current=None,
parent=None,
parent_id=None,
**event)
Create a new toplevel span underneath the logger. The name defaults to "root" and the span type to "task".
See Span.start_span
for full details
export
def export() -> str
Return a serialized representation of the logger that can be used to start subspans in other places. See Span.start_span
for more details.
flush
def flush()
Flush any pending logs to the server.
ScoreSummary Objects
@dataclasses.dataclass
class ScoreSummary(SerializableDataClass)
Summary of a score's performance.
name
Name of the score.
score
Average score across all examples.
diff
Difference in score between the current and reference experiment.
improvements
Number of improvements in the score.
regressions
Number of regressions in the score.
MetricSummary Objects
@dataclasses.dataclass
class MetricSummary(SerializableDataClass)
Summary of a metric's performance.
name
Name of the metric.
metric
Average metric across all examples.
unit
Unit label for the metric.
diff
Difference in metric between the current and reference experiment.
improvements
Number of improvements in the metric.
regressions
Number of regressions in the metric.
ExperimentSummary Objects
@dataclasses.dataclass
class ExperimentSummary(SerializableDataClass)
Summary of an experiment's scores and metadata.
project_name
Name of the project that the experiment belongs to.
experiment_name
Name of the experiment.
project_url
URL to the project's page in the Braintrust app.
experiment_url
URL to the experiment's page in the Braintrust app.
comparison_experiment_name
The experiment scores are baselined against.
scores
Summary of the experiment's scores.
metrics
Summary of the experiment's metrics.
DataSummary Objects
@dataclasses.dataclass
class DataSummary(SerializableDataClass)
Summary of a dataset's data.
new_records
New or updated records added in this session.
total_records
Total records in the dataset.
DatasetSummary Objects
@dataclasses.dataclass
class DatasetSummary(SerializableDataClass)
Summary of a dataset's scores and metadata.
project_name
Name of the project that the dataset belongs to.
dataset_name
Name of the dataset.
project_url
URL to the project's page in the Braintrust app.
dataset_url
URL to the experiment's page in the Braintrust app.
data_summary
Summary of the dataset's data.
braintrust.framework
EvalCase Objects
@dataclasses.dataclass(kw_only=True)
class EvalCase(SerializableDataClass)
An evaluation case. This is a single input to the evaluation task, along with an optional expected output, metadata, and tags.
EvalResult Objects
@dataclasses.dataclass(kw_only=True)
class EvalResult(SerializableDataClass)
The result of an evaluation. This includes the input, expected output, actual output, and metadata.
EvalHooks Objects
class EvalHooks(abc.ABC)
An object that can be used to add metadata to an evaluation. This is passed to the task
function.
span
@property
@abc.abstractmethod
def span() -> Span
Access the span under which the task is run. Also accessible via braintrust.current_span()
meta
@abc.abstractmethod
def meta(**info) -> None
Adds metadata to the evaluation. This metadata will be logged to the Braintrust. You can pass in metadaa
as keyword arguments, e.g. hooks.meta(foo="bar")
.
EvalScorerArgs Objects
class EvalScorerArgs(SerializableDataClass)
Arguments passed to an evaluator scorer. This includes the input, expected output, actual output, and metadata.
BaseExperiment Objects
@dataclasses.dataclass
class BaseExperiment()
Use this to specify that the dataset should actually be the data from a previous (base) experiment. If you do not specify a name, Braintrust will automatically figure out the best base experiment to use based on your git history (or fall back to timestamps).
name
The name of the base experiment to use. If unspecified, Braintrust will automatically figure out the best base using your git history (or fall back to timestamps).
Evaluator Objects
@dataclasses.dataclass
class Evaluator()
An evaluator is an abstraction that defines an evaluation dataset, a task to run on the dataset, and a set of scorers to evaluate the results of the task. Each method attribute can be synchronous or asynchronous (for optimal performance, it is recommended to provide asynchronous implementations).
You should not create Evaluators directly if you plan to use the Braintrust eval framework. Instead, you should
create them using the Eval()
method, which will register them so that braintrust eval ...
can find them.
project_name
The name of the project the eval falls under.
eval_name
A name that describes the experiment. You do not need to change it each time the experiment runs.
data
Returns an iterator over the evaluation dataset. Each element of the iterator should be an EvalCase
or a dict
with the same fields as an EvalCase
(input
, expected
, metadata
).
task
Runs the evaluation task on a single input. The hooks
object can be used to add metadata to the evaluation.
scores
A list of scorers to evaluate the results of the task. Each scorer can be a Scorer object or a function
that takes input
, output
, and expected
arguments and returns a Score
object. The function can be async.
experiment_name
Optional experiment name. If not specified, a name will be generated automatically.
metadata
A dictionary with additional data about the test example, model outputs, or just about anything else that's
relevant, that you can use to help find and analyze examples later. For example, you could log the prompt
,
example's id
, or anything else that would be useful to slice/dice later. The values in metadata
can be any
JSON-serializable type, but its keys must be strings.
trial_count
The number of times to run the evaluator per input. This is useful for evaluating applications that have non-deterministic behavior and gives you both a stronger aggregate measure and a sense of the variance in the results.
is_public
Whether the experiment should be public. Defaults to false.
ReporterDef Objects
@dataclasses.dataclass
class ReporterDef(SerializableDataClass)
A reporter takes an evaluator and its result and returns a report.
name
The name of the reporter.
report_eval
A function that takes an evaluator and its result and returns a report.
report_run
A function that takes all evaluator results and returns a boolean indicating whether the run was successful.
If you return false, the braintrust eval
command will exit with a non-zero status code.
Eval
def Eval(name: str,
data: Callable[[], Union[Iterator[EvalCase],
AsyncIterator[EvalCase]]],
task: Callable[[Input, EvalHooks], Union[Output, Awaitable[Output]]],
scores: List[EvalScorer],
experiment_name: Optional[str] = None,
trial_count: int = 1,
metadata: Optional[Metadata] = None,
is_public: bool = False,
reporter: Optional[Union[ReporterDef, str]] = None)
A function you can use to define an evaluator. This is a convenience wrapper around the Evaluator
class.
Example:
Eval(
name="my-evaluator",
data=lambda: [
EvalCase(input=1, expected=2),
EvalCase(input=2, expected=4),
],
task=lambda input, hooks: input * 2,
scores=[
NumericDiff,
],
)
Arguments:
name
: The name of the evaluator. This corresponds to a project name in Braintrust.data
: Returns an iterator over the evaluation dataset. Each element of the iterator should be aEvalCase
.task
: Runs the evaluation task on a single input. Thehooks
object can be used to add metadata to the evaluation.scores
: A list of scorers to evaluate the results of the task. Each scorer can be a Scorer object or a function that takes anEvalScorerArgs
object and returns aScore
object.experiment_name
: (Optional) Experiment name. If not specified, a name will be generated automatically.trial_count
: The number of times to run the evaluator per input. This is useful for evaluating applications that have non-deterministic behavior and gives you both a stronger aggregate measure and a sense of the variance in the results.metadata
: (Optional) A dictionary with additional data about the test example, model outputs, or just about anything else that's relevant, that you can use to help find and analyze examples later. For example, you could log theprompt
, example'sid
, or anything else that would be useful to slice/dice later. The values inmetadata
can be any JSON-serializable type, but its keys must be strings.is_public
: (Optional) Whether the experiment should be public. Defaults to false.reporter
: (Optional) A reporter that takes an evaluator and its result and returns a report.
Returns:
An Evaluator
object.
Reporter
def Reporter(
name: str,
report_eval: Callable[[Evaluator, EvalResultWithSummary, bool, bool],
Union[EvalReport, Awaitable[EvalReport]]],
report_run: Callable[[List[EvalReport], bool, bool],
Union[bool, Awaitable[bool]]])
A function you can use to define a reporter. This is a convenience wrapper around the ReporterDef
class.
Example:
def report_eval(evaluator, result, verbose, jsonl):
return str(result.summary)
def report_run(results, verbose, jsonl):
return True
Reporter(
name="my-reporter",
report_eval=report_eval,
report_run=report_run,
)
Arguments:
name
: The name of the reporter.report_eval
: A function that takes an evaluator and its result and returns a report.report_run
: A function that takes all evaluator results and returns a boolean indicating whether the run was successful.
set_thread_pool_max_workers
def set_thread_pool_max_workers(max_workers)
Set the maximum number of threads to use for running evaluators. By default, this is the number of CPUs on the machine.