Improving Data Dependence Estimation in Markov Decision Processes via Weak Dependence

In this paper, we propose a new framework called concentrability to address the issue of adversarial offline reinforcement learning (RL). Concentrability is a way of measuring how well an RL algorithm can learn from a dataset without being influenced by any potential biases or errors in the data.
To understand concentrability, imagine you are trying to find the best recipe for a dish based on a set of ingredients. The recipe represents the policy (i.e., what action to take given the state of the environment) and the ingredients represent the data (i.e., the state of the environment). The goal is to find a policy that works well across different batches of data, similar to how a good recipe should work with different sets of ingredients.
Existing works on offline RL assume that the dataset is fully covered, which means that all possible states and actions are included in the data. However, this assumption is unrealistic as real-world datasets often have gaps or biases. Concentrability relaxes this assumption by considering partial coverage, where only some states and actions are included in the data.
We define concentrability as the maximum expected value of the policy under a set of possible future observations, given the current state of the environment. This definition allows us to evaluate how well an RL algorithm can handle different scenarios and adapt to new information.
Our main results show that concentrability is closely related to the concept of robustness in RL. A policy with high concentrability is more robust to changes in the data distribution, which means it can generalize better to new situations. We also demonstrate that concentrability can be used to analyze and improve the performance of existing offline RL algorithms.
In summary, concentrability provides a unified framework for understanding adversarial offline RL by measuring how well an algorithm can learn from a dataset without being biased towards any particular policy. By focusing on partial coverage and robustness, we demonstrate that concentrability is a valuable tool for analyzing and improving the performance of offline RL algorithms.