Tensor Kernel Hilbert Space for Safe Reinforcement Learning

In this article, we propose a new approach to reinforcement learning called Safe Kernel RL, which combines the concepts of Predictive State Representations (PSR) and Tensor Reproducing Kernel Hilbert Spaces (RKHS). PSR helps us represent the relationships between observations, histories, and policy information in a more structured way, while RKHS provides a universal function approximation framework to handle complex problems.
To implement Safe Kernel RL, we propose five operators that describe how the value/risk functions are transformed from the latent state to the observation space. These operators enable us to achieve ϵ-sub-optimal solutions with polynomial sample complexity, which is more efficient than other methods. The key insight is that even though there may be multiple link functions between the observation and latent state spaces, the induced value and risk functions are unique.
To understand how this works, imagine you’re a chef trying to create a new recipe for a dish. You have a set of ingredients (observations) and a set of cooking techniques (policy information). The recipe (value/risk function) tells you how to combine the ingredients in the right way to make a delicious meal (optimal policy). PSR helps you structure the recipe by defining the relationships between the ingredients, while RKHS provides a way to approximate the recipe using different cooking techniques.
The proposed operators allow us to map the value/risk functions from the latent state space to the observation space using link functions, which are like secret ingredients that enhance the flavor of the dish. These link functions are well-defined and unique, so we can use them to create a wide range of delicious meals (policies). By combining PSR and RKHS, Safe Kernel RL provides a powerful framework for solving reinforcement learning problems in complex environments.
In summary, Safe Kernel RL is a new approach to reinforcement learning that combines the structured representation of Predictive State Representations with the flexibility of Tensor Reproducing Kernel Hilbert Spaces. By leveraging these powerful tools, we can create more efficient and effective policies in complex environments, making it easier for agents to learn and make decisions in a wide range of domains.