Meeting Many Goals with Smarter Reinforcement Learning
Problem
Reinforcement learning (RL) trains an agent to make a sequence of decisions toward a goal, expressed as a reward. In business, the goal rarely stays fixed. A trading system may need to balance return against risk differently next quarter. A manufacturing line may switch from maximising throughput to minimising energy use. A care-planning tool may need to weigh outcomes differently for different patients.
Standard RL ties what an agent learns closely to one reward. Change the goal and much of the learning has to be redone. Real environments add a second difficulty: the agent often cannot see everything it needs to know, so it has to gather information and reason about hidden factors. Researchers describe this setting as partially observable, and model it as a POMDP.
A family of methods called successor representations helps with the first problem. They let an agent generalise from past experience to new goals. But, as the paper explains, they are not optimised to generalise across policies, meaning different ways of behaving, and they typically make no provision for gathering information or reasoning about hidden variables.
Also applicable to
- Finance: decision policies that can be re-targeted as objectives and constraints change.
- Healthcare: treatment and care planning that balances several outcomes.
- Manufacturing: control systems that switch between goals such as speed, quality and energy use.
- Robotics and automation: agents that take on new tasks in the same environment.
- Operations and logistics: planning that adapts when priorities shift.
Solution / Approach
The Elementera team's research on multi-goal reinforcement learning appears in Successor Feature Sets: Generalizing Successor Representations Across Policies, a paper co-authored by Elementera co-founder Soroush Mehri with Kianté Brantley and Geoff J. Gordon, published at AAAI 2021.
The paper brings together ideas from four areas: predictive state representations, belief-space value iteration (a way of planning when the true state is hidden), successor features and convex analysis. From these it develops:
- A new, general successor-style representation that covers many policies at once, not just a short list.
- A Bellman equation that connects multiple sources of information within this representation, including different hidden states, policies and reward functions.
The result is a representation that, in the paper's words, is highly expressive. For example, it lets an agent efficiently read off an optimal policy for a new reward function, or a policy that imitates a new demonstration. In other words, the learning is done once, and new goals can be served from it.
The paper focuses on exact computation of the representation in small, known environments, building the mathematical foundation for this kind of goal-flexible decision-making.
Why this matters for a company using AI: decision systems are expensive to build. A representation that can be re-used when the goal changes, rather than retrained, makes RL a better fit for businesses whose priorities move.
Impact
This work extends successor representations, which the paper notes form a natural bridge between model-based and model-free RL, so that they generalise across policies and handle hidden information. For businesses, that means:
- One model, many goals: an optimal policy for a new reward can be read from the representation.
- Learning from demonstrations: the same representation supports imitating a new example of good behaviour.
- Decisions under uncertainty: the framework reasons about hidden states, as real operations require.
- Deep RL expertise: together with Dr. Maziar Gomrokchi's published work on exploration and private policy evaluation, this gives the Elementera team strong grounding in how decision-making systems learn.
Technologies
- Reinforcement learning: sequential decision-making from rewards.
- Successor features and representations: predicting future outcomes in a goal-independent way.
- POMDPs and belief-space planning: acting when the full state is not observed.
- Predictive state representations: describing state through predictions of future observations.
Read the paper: Successor Feature Sets: Generalizing Successor Representations Across Policies (AAAI 2021).
Ex-Microsoft. 10+ years of experience in AI/ML, MSc from Université de Montréal (Mila). US patent holder.
AI/ML scientist with 20+ years of research & industry experience. Ex-Google DeepMind, MIT, Harvard, Mila. McGill-Mila PhD.