exec() System Call Within Transactions

One can execute an exec() system call while holding a lock, and also from within an RCU read-side critical section. The exact semantics depends on the type of primitive.

In the case of non-persistent primitives (including pthread_mutex_lock(), pthread_rwlock_rdlock(), and RCU), if the exec() succeeds, the whole address space vanishes, along with any locks being held. Of course, if the exec() fails, the address space still lives, so any associated locks would also still live. A bit strange perhaps, but reasonably well defined.

On the other hand, persistent primitives (including the flock family, lockf(), System V semaphores, and the O_CREAT flag to open()) would survive regardless of whether the exec() succeeded or failed, so that the exec()ed program might well release them. (Non-persistent primitives represented by data structures in mmap() regions of memory are an interesting intermediate point that is left to the reader.)

What happens when you attempt to execute an exec() system call from within a transaction?

Here are some options available to TM:

1. Disallow exec() within transactions, so that the enclosing transactions abort upon encountering the exec(). This is well defined, but clearly requires non-TM synchronization primitives for use in conjunction with exec().
2. Disallow exec() within transactions, with the compiler enforcing this prohibition. There is a draft specification for TM in C++ that takes this approach, allowing functions to be decorated with the transaction_safe and transaction_unsafe attributes. (Thanks to Mark Moir for pointing me at this spec, and to Michael Wong for having pointed me at an earlier revision some time back.) This approach has some advantages over aborting the transaction at runtime, but again requires non-TM synchronization primitives for use in conjuntion with exec().
3. Treat the transaction in a manner similar to non-persistent locking primitives, so that the transaction survives if exec() fails, and silently commits if the exec() succeeds. The case were some of the variables affected by the transaction reside in mmap()ed memory (and thus could survive a successful exec() system call) is left as an exercise for the reader.
4. Abort the transaction (and the exec() system call) if the exec() system call would have succeeded, but allow the transaction to continue if the exec() system call would fail. This is in some sense the “correct” approach, but it would require considerable work for a rather unsatisfying result.

The exec() system call is perhaps the strangest of these examples, as it is not completely clear what approach makes sense, and some might argue that this is merely a reflection of the perils of interacting with execs in real life. That said, the two options prohibiting exec() within transactions are perhaps the most logical of the group.