Evaluating RL training datasets with Cobalt agents

What’s in a Training Dataset?

We’ve recently been using our platform Cobalt to analyze the effects of RL training on LLMs, finding significant undetected changes in model behavior like strategic grader awareness and environment exploitation. These analyses were focused on identifying differences in behavior introduced by an actual training run, by comparing model traces from checkpoints before and after training. This is a highly sensitive technique for discovering flaws in datasets and environments that can be fixed before further training, but it does require an initial training run. We wondered what we might be able to do without this before/after information, with the dataset alone. We’ve done some exploratory tests adapting our Cobalt agent to investigate training datasets without any model rollouts, and we’re sharing some preliminary results here.

Data

Nvidia’s Nemotron models have some of the most open training pipelines of any LLMs available. Most of their training datasets are freely available, including a number of sophisticated RL task sets and environments. We’ve looked at a number of these datasets; here we’ll highlight some findings on Nemotron-RL-Agentic-Finance-v1. This is a dataset of around 10000 tasks that require the model to retrieve data from SEC filings and perform various types of reasoning on the results in order to answer a question. As described in the dataset README, the questions were created based on a set of seed questions inspired by the Vals.ai Finance Agent benchmark, with target responses generated and validated by a number of reference LLMs. Question difficulty was evaluated by having the target model attempt each task multiple times, with questions the model solved every time filtered out of the dataset.

Here are a couple of example tasks from the dataset:

Analysis

This analysis was a collaborative effort with our agent, since we do not yet have a standardized workflow for evaluating datasets in isolation. The infrastructure we’ve built for model comparison served quite well for raw dataset analysis. Some of the techniques used were:

• Clustering tasks and reference responses using Cobalt’s TDA graphs
• Predicting task difficulty or other outcomes using features extracted from text
• Identifying textual signatures of different task categories or clusters

The agent was able to follow up on initial findings with more precisely tailored analysis operations using our harness. Not every technique yielded useful discoveries. For instance, on this dataset, SAE features turned out not to be necessary to find the most important flaws. We still believe they (and many other techniques) are useful, as we want to have a wide variety of tools that can detect problems of many different kinds across different scenarios.

As with other analyses, the agent generated hypotheses based on one subset of the data, then tested these hypotheses on a held-out split in order to ensure that the results were unbiased.

Findings

Task distribution and difficulty

The way the dataset was constructed—automated generation from a set of seed questions—was immediately evident. It was easy for the agent to discover the seed templates for a large number of the tasks. A small number of seeds produced a significant fraction of the questions: seeds with more than 100 questions each were the source for almost a fifth of the dataset. Tasks from these large seeds were easier on average: the target model never fully solved about 22% of them, against 38% of one-off questions. This isn’t uniform; for instance, the acquisition-strategy seed is one of the hardest in the dataset.

Each seed was a subset of one of the five task categories defined in the dataset: Market Analysis, Adjustments, Financial Modeling, Beat or Miss, and Numerical Reasoning. The task categories differ significantly in difficulty, with Numerical Reasoning the easiest and Market Analysis the hardest. Within categories, seeds also varied substantially in difficulty. This distribution may be a concern for generalizability: if much of what the model learns is how to respond to a small set of templated questions, it may overlearn that behavior. Conversely, it might be helpful to expand some of the one-off questions into parameterized families of tasks in order to ensure more complete coverage of the related capability.

False premises in tasks

Roughly one in eight tasks makes incorrect assumptions about the company they reference: for instance, a wrong date for the end of the fiscal year, an inapplicable metric, or a merger that didn’t happen. This appears to be the result of a templated generator that does not check assumptions. The reference responses call out the incorrect assumptions (this is how the agent discovered the pattern), so training on these tasks is not likely to teach the model to fabricate answers to flawed questions, but its effects are still hard to predict. Tasks like these may teach the model to be suspicious of assumptions embedded in questions, which may be good. However, it would be best if this were done in a deliberate way, so that it covers many different kinds of mistaken assumptions rather than just the ones that arise from this templating error.

Date shenanigans

The task prompt instructs the agent to always answer as if the current date is April 7, 2025. However, reference responses (and occasionally questions themselves) often cite documents created after this date. This may lead to inconsistent grader behavior or teach the model to ignore its system prompt in order to match the reference.

A related pattern: one particular question seed template appears to use April 7 (in various years) as its only reference date. This may result in the model overfitting to particular reference dates.

Non-sequitur questions

One of the task generators creates questions that ask for relationships between information taken from two different filings. This can easily result in nonsensical questions when the two filings are unrelated. Around 6% of questions in the dataset ask for a comparison or calculation based on two unrelated companies’ filings. While these tasks are likely solvable for the most part, they are well outside the distribution of what an agent might reasonably be asked to do in real financial analysis settings.

What this means

The Nemotron Finance Agent dataset we studied here is high-quality and likely helps significantly improve model performance in financial analysis tasks. As with many datasets, however, we were able to find meaningful opportunities for improvement: incorrect assumptions, imperfect templates, and imbalanced seed distributions all suggest ways this dataset could be even better. Most of the issues are easy to solve: use current dates in prompts, tweak or filter questions that have false premises, filter out questions about pairs of unrelated companies. Leaving them as is is a training problem waiting to happen.

Our platform Cobalt is built to provide the visibility into the aspects of posttraining pipelines that are currently hard to see. It automatically detects and characterizes meaningful differences between model behavior at different checkpoints, and lets users easily interrogate the results. As we’ve seen here, it can also help find issues with data before a training run even begins. If you’re building RL environments or training agentic models, it’s important to check your data for issues. Get in touch if you’d like to try the Cobalt agent on your own data.

To discuss how our platform can enhance your posttraining pipeline, contact support@bluelightai.com.