Official dataset collectionProvider-published evidence
TypeSafe WorkflowEvals
Four provider-published datasets for testing model decisions inside invoice, customer-service, security, and agent-review workflows.
WorkflowEvals pairs workflow cases, policy-conditioned questions, reference decisions, and published model runs. Type Safe catalogs the collection as evidence material; these datasets do not create an independent Type Safe leaderboard or endorsement.
- Publisher
- TypeSafe AI
- Published or submitted
- Not specified by the source
- Last verified
- October 8, 2026
Type Safe catalogs sources and evidence boundaries. Provider datasets and third-party preprints do not become an independent leaderboard.
Key facts
- Datasets
- 4
- Default model in repository
- typesafe:jev-1.13.0
- Published metrics
- exact_actions and primary_action
Method and usage
- Each workflow provides cases, policy-conditioned question instances, reference labels, and published run results.
- The repository recomputes agreement against selected references and supports exact action-set and primary-action comparisons where the workflow exposes them.
- Published providers include OpenAI, Anthropic, Fireworks, Groq, Cerebras, and TypeSafe; missing cost rates remain unknown.
Limitations and evidence boundaries
- The provider publishes the datasets and evaluation code, so this directory does not treat the results as an independent benchmark.
- Agreement with a selected reference is not a universal measure of real-world workflow quality.
- Dataset licenses are repository-specific and must not be inferred from the Apache-2.0 code license.