Formalizing Nondeterminism in Coq Semantics

In this article, we explore the concept of stack canaries in the context of formal verification, specifically within the CompCert system. We delve into the semantics of how stack canaries operate and their role in preventing buffer overflow attacks on variables, focusing on arrays allocated on the stack. The author presents a step-by-step explanation of how to implement canaries inside a verified compilation chain using the Frama-C toolset.
To begin with, we need to understand that the stack is like a limited resource, where programs can store data temporarily. In some cases, this data may contain sensitive information or critical system details, making it essential to protect them from tampering. Canaries are a countermeasure against buffer overflow attacks on variables, especially arrays allocated on the stack.
The author explains that CompCert treats stack-allocated data as a single block, which simplifies the implementation of canaries. The informal argument for this code transformation working is based on the assumption that if the program executed correctly prior to the transformation (i.e., writing to the canary was considered undefined behavior), then this execution is preserved after the transformation.
The article highlights that installing canaries on all functions may lead to unnecessary slowdown, so a more effective approach is to only apply them to “vulnerable” functions based on some heuristic for vulnerability.
To formalize these concepts, the author leverages existing work in the field, including Nickel’s WP Plug-in Manual for Frama-C and Zhao et al.’s paper on formalizing the LLVM intermediate representation for verified program transformations. These works provide a foundation for understanding how to model stack frames as a single memory block and how to reason about program behavior using monotonic definiteness.
In conclusion, this article demystifies complex concepts related to stack canaries and formal verification by using everyday language and engaging analogies. By breaking down the technical details into manageable chunks, readers can gain a deeper understanding of how these techniques work together to protect against buffer overflow attacks and ensure the correctness of compiled code.