Curious whether this framework extends to agentic systems where advice shapes the planner's internal search over many iterations. For example, suppose an agent repeatedly asks an untrusted adviser which subproblem to tackle next. Each recommendation may be locally reasonable, but a slight bias in those recommendations can gradually steer the planner away from important parts of the search space. Here the cumulative effect is a drift in the agent's internal planning process that might eventually change the final outcome. It seems like, in this setting, the object to control is not individual advice but cumulative influence over the planner's search.
Yes, it's definitely important to consider the cumulative influence of advice across an entire trajectory!
In our experiments, we only enforce an information bottleneck on the per-message level, but we measure information usage on the per-trajectory level, and it would be possible to explicitly limit information per-trajectory. As mentioned briefly in the post, "an information bottleneck can apply to a single advice message, a single trajectory, or even across U’s entire deployment."
When you mention that the advisor could "steer the planner away from important parts of the search space," that sounds like a problem of diffuse control (https://blog.redwoodresearch.org/p/how-can-we-solve-diffuse-threats). The untrusted advice protocol doesn't prevent diffuse problems, like U sandbagging on certain kinds of tasks, since U can easily decide to give worse advice whenever it wants. In this post, we instead focus on high-stakes control, where U wins by causing a single catastrophe.
It's surely easier to cause a catastrophe if you can use multiple pieces of advice to gradually steer the executor over the course of a trajectory, but it's still pretty hard. For example, it's hard to imagine how to subtly nudge the model's search into downloading malware from a specific URL.
While I agree that reducing "channel capacity" is in principle a promising path towards control, as soon as it becomes a selection-force bottleneck, the compression ratio of your encoding rises massively (e.g., bits per character). We might define a regularizer, or an Occam's razor, in this way: i.e., an information bottleneck constraining a selection force.
This, however, cuts against interpretability:
By analogy, suppose the layers 1, ..., N - 1 are an untrusted model U, which must steer the world through the last layer N, our trusted model T. The information bottleneck between them is the activation vector dimension.
Every interpretability method can be described as taking a dense encoding (e.g., the activation vector) and projecting it into a redundant one (e.g., a sparse vector, recently natural language). So the tradeoff is kind of "less stuff to review in total" but "much more dense stuff to review".
Concretely: Do you think LLM-surprisal-limiting can solve that, or will it push down dense thought deep into natural-sounding text? (i.e., steganography that sounds like natural language to an LLM, that actually contains strongly compressed abstractions)
Curious whether this framework extends to agentic systems where advice shapes the planner's internal search over many iterations. For example, suppose an agent repeatedly asks an untrusted adviser which subproblem to tackle next. Each recommendation may be locally reasonable, but a slight bias in those recommendations can gradually steer the planner away from important parts of the search space. Here the cumulative effect is a drift in the agent's internal planning process that might eventually change the final outcome. It seems like, in this setting, the object to control is not individual advice but cumulative influence over the planner's search.
Yes, it's definitely important to consider the cumulative influence of advice across an entire trajectory!
In our experiments, we only enforce an information bottleneck on the per-message level, but we measure information usage on the per-trajectory level, and it would be possible to explicitly limit information per-trajectory. As mentioned briefly in the post, "an information bottleneck can apply to a single advice message, a single trajectory, or even across U’s entire deployment."
When you mention that the advisor could "steer the planner away from important parts of the search space," that sounds like a problem of diffuse control (https://blog.redwoodresearch.org/p/how-can-we-solve-diffuse-threats). The untrusted advice protocol doesn't prevent diffuse problems, like U sandbagging on certain kinds of tasks, since U can easily decide to give worse advice whenever it wants. In this post, we instead focus on high-stakes control, where U wins by causing a single catastrophe.
It's surely easier to cause a catastrophe if you can use multiple pieces of advice to gradually steer the executor over the course of a trajectory, but it's still pretty hard. For example, it's hard to imagine how to subtly nudge the model's search into downloading malware from a specific URL.
While I agree that reducing "channel capacity" is in principle a promising path towards control, as soon as it becomes a selection-force bottleneck, the compression ratio of your encoding rises massively (e.g., bits per character). We might define a regularizer, or an Occam's razor, in this way: i.e., an information bottleneck constraining a selection force.
This, however, cuts against interpretability:
By analogy, suppose the layers 1, ..., N - 1 are an untrusted model U, which must steer the world through the last layer N, our trusted model T. The information bottleneck between them is the activation vector dimension.
Every interpretability method can be described as taking a dense encoding (e.g., the activation vector) and projecting it into a redundant one (e.g., a sparse vector, recently natural language). So the tradeoff is kind of "less stuff to review in total" but "much more dense stuff to review".
Concretely: Do you think LLM-surprisal-limiting can solve that, or will it push down dense thought deep into natural-sounding text? (i.e., steganography that sounds like natural language to an LLM, that actually contains strongly compressed abstractions)