Evaluating Model Performance via Black-Box Sampling

In this article, we dive into the world of generative models and explore how to evaluate their performance. Generative models are like magicians who can create new samples that look real, but we need to know if they’re any good or not. The problem is, these models don’t come with a instruction manual, so we need to find new ways to measure their quality.
We propose a new method called Wang-Landau sampling, which is like a game of chance where we sample from the model’s output distribution. We use this method to estimate how confident the model is in its predictions and if it’s too confident or not.
Other researchers have also studied performance characterization, but they focus on different aspects of generative models. Some use simple models like toy cars, while others use mathematical morphological operators like building blocks. Our approach is similar to these methods but with a twist: we focus on the output distribution rather than the internal workings of the model.
The trade-off between precision and recall in generative models is like a game of give-and-take. The model can be very precise but miss some details, or it can be more recalling but less precise. Our method helps us understand this trade-off better so we can improve the models.
In summary, this article presents a new way to evaluate the performance of generative models by using Wang-Landau sampling and explores how these models make trades between precision and recall. By understanding these trades, we can create better models that are both precise and recalling.