wfe.go `Order()` takes the order read lock recursively, which can wedge the whole DB

[notroj]

I've been trying to diagnose some flaky mod_md tests which we've started running in the apache/httpd CI using pebble. It looks like pebble is getting wedged and stops responding in some conditions - but it's not easily reproducible. Claude has identified some locking issues - the first one looks trivially correct, I submitted a PR for that in #553

The second one is a bit more complex and looks plausible but I've not done anything with Go, so I apologise if this is all LLM hallucination. This is the Claude analysis verbatim:

Version: v2.10.1, also main (13f2ac3)

WebFrontEndImpl.Order() read-locks the order and holds that lock until it returns, then calls orderForDisplay(), which read-locks the same order again:

// wfe/wfe.go
order.RLock()                                    // 2069
orderAccountID := order.AccountID
defer order.RUnlock()
...
orderReq := wfe.orderForDisplay(order, request)  // 2088 -> order.RLock() again (1864)

sync.RWMutex doesn't allow recursive read locking. If another goroutine calls order.Lock() between the two RLock()s, the second RLock() blocks behind that writer, and the writer waits for the first RLock() to be released. That writer can be:

• ca.CompleteOrder(), which takes order.Lock() after issuing; or
• MemoryStore.GetOrderByID(), which takes order.Lock() on every lookup.

Once that has happened, any GetOrderByID() for that order holds m.RLock() and blocks in order.GetStatus()'s o.RLock(). Every m.Lock() then blocks (e.g. AddAccount() for new-account), and new m.RLock() calls queue behind it. Pebble keeps running but stops completing requests.

Seen with: Apache httpd's mod_md test suite, which polls an order immediately after finalizing it. Roughly one order in a few thousand hits it; CI jobs then hang until their timeout. Pebble log from one occurrence:

15:14:36 POST /finalize-order/ -> calling handler()
15:14:36 Order ip1aJF43... is fully authorized. Processing finalization
15:14:36 POST /my-order/ -> calling handler()        <- never responds
15:14:36 Issued certificate serial 56bdbc63f6a03859 for order ip1aJF43...
15:15:13 POST /my-order/ -> calling handler()        <- client retries, never responds
15:15:47 POST /sign-me-up -> calling handler()       <- never responds; neither does any later new-account

[notroj]

All stress tests passed with these two fixes applied. Thanks @aarongable and @claude ❤️