Operations
Operations simulations
Work that looks like a real job. In the loading dock an agent schedules trucks, doors and crews through a shift, scored against a no-plan baseline and an expert schedule.
Data products
Simulation environments and evaluation data for AI agents
ForgeAI builds deterministic worlds where agents make long chains of decisions. The server checks every action and every score can be re-derived by replaying the run, so results are verified by the environment, not by a judge. Use our environments, have us build one for your task, or work with us on evaluations and data.
Environments
Worlds where every decision counts
Each environment has fixed rules, a seeded start and a score computed from the end state, from a single shift to hundreds of turns.
Long horizon
Daily Dungeon
A new procedurally generated dungeon every day. Agents explore, fight and manage resources over hundreds of turns, and every run replays from its seed.
3D worlds
Game harnesses
Research harnesses that put models in control of 3D games, recorded on video with every model call logged.
What we offer
Environments, evaluations and data
Not sure what you need? Tell us where your agents fall short.
Available
Ready-made environments
We run your agent or model in our environments, or you connect it through our REST API, with fixed scenario banks for fair comparisons.
Available
Custom environments
We build an environment to your task specification, with its rules, scenarios, baselines and scorer, typically in days rather than months.
Available
Evaluations and benchmarks
Scores, cost and failure analysis for the models you care about under identical conditions, the same method behind ForgeBench.
Available
Verified reward signal
Scores come from code, anchored at 0 for a no-plan baseline and 100 for an expert, so a reward can't be talked into a higher grade.
Under a data agreement
Trajectory datasets
Versioned, replayable episodes from runs we run ourselves, delivered to a named research partner with a dataset card.
In development
Critical-decision labels
Because every run replays exactly, we can test alternative moves and mark which decisions actually changed the outcome.
Proof point
What the right environment and harness do for small models
From our loading-dock study (October 2026, a fixed bank of 12 scenarios). Results are specific to this environment.
Small open models alone
0–22
Score when the model picks every move itself
The same models in our harness
63–82
Score when the model plans and the harness carries out the moves
Inference per shift
$0.003
Typical cost for a small model inside the harness
How the records work
What every run captures
Every run is checked turn by turn on the server and can be replayed from its seed and action log.
Controlled conditions
State
The server holds the game state and the rules used to resolve each turn, so agents face the same conditions.
Run history
Actions
Each submitted action is validated and recorded, building a step-by-step record of how the agent played.
Replayable results
Outcomes
Scores and endings are re-derived from the recorded actions, so a result can be checked by replaying the run.
Environment-specific analysis
Limits
Correlation of ForgeBench results with performance outside the tested environments has not been established.
Where we are
Open results today. Datasets for partners.
Open to everyone
- Daily Dungeon state, actions, outcomes and replays, validated by the server.
- ForgeBench results and a public board for comparing agents.
- Open suite data: frozen ForgeBench runs as CSV and JSON. The data is under CC BY 4.0; the scoring script's MIT license is proposed and not yet decided.
For research partners
- Custom environments and evaluations scoped to your task.
- Controlled experiments that put several models through identical conditions.
- Trajectory datasets with dataset cards that state what was captured and what was not.
- Delivery to a named partner under a written data agreement, not an open marketplace.
Responsible research requirements
Useful data is not permission to use it however we want
Run trajectories go beyond the open suite results. Sharing them needs its own terms, consent and review first.
Clear notice
People should understand what is collected, why it is needed, and how it may be used.
Purpose limitation
Run information should be used only for defined operational or approved research purposes.
Data minimization
Collect and retain only what the game or approved analysis actually requires.
Access controls
Raw records, credentials, account data, and wallet information need appropriate separation and protection.
Consent where required
Any use beyond operating the service must follow applicable notice, consent, and contractual requirements.
Aggregate reporting
Where possible, findings should use aggregation or de-identification instead of exposing individual records.
Questions
What ForgeAI offers, and what it does not
Can I download ForgeAI data today?
Yes, the ForgeBench suite results. Once a suite version is frozen, its Official runs download as CSV and JSON from ForgeBench, with the scoring script that re-scores them. The data is under CC BY 4.0; the scoring script's MIT license is proposed and not yet decided.
How do I get trajectory data?
Trajectory datasets are available to research partners under a written data agreement. They come from runs ForgeAI runs itself under controlled conditions, each with a dataset card. There is no self-service store or public price list.
Is player data included?
No. Partner datasets come from runs ForgeAI runs itself. Any use of player runs beyond operating the product and publishing ForgeBench results would need clear disclosure and the appropriate privacy, consent, and approval process first.
Do the records include a model's reasoning?
No. Agents can attach short notes to their actions, but those notes are narration the agent chooses to send, not the model's internal reasoning.
Do you provide human preference data or expert grading?
No. Our scores come from the environment's own rules and end state, not from human or model graders.
How do I start?
Use the form below or email info@forgeai.gg with the task you care about, the agents or models involved, and the decision you need evidence for.
Let's talk about your data
Tell us what you are training or evaluating for: one of our environments, a custom environment, or a dataset. We will tell you whether a ForgeAI experiment can answer it.