All allocations within MuPDF (and its sub-libraries) call fz_malloc and family. These functions ultimately call down to the custom allocator functions passed into the fz_new_context call (or to malloc and family if no custom allocators were supplied). (See chapter 5 The Context for details).
If a call to the underlying custom allocator fails, MuPDF will automatically seek to evict the least recently used objects from the store that are not currently being used, and then will retry the allocation. This can happen several times, with more and more objects being freed between each attempt.
Allocation failures are therefore only fatal to MuPDF if there are no remaining objects to be freed in the store.
This ‘just in time’ scavenging of memory means that the store limit can safely be set to a high level (or to be unlimited), and MuPDF will still operate within safe bounds.