[Iradicator]

Hi,

I want to build some sort of API monitor by hooking every ntdll function. So each hooked function will call the original implementation, and than add this call to the std based data structure.

However, I encountered a scenario of recursive hooking where my hooked function is indirectly calling itself.

Therefore, I tried to use the Tls memory to set a bit per thread that indicate that current function is called from hooked function, so avoid calling the hook again (execute the original function only).

But my recursive hook guard also indirectly triggers calls for memory allocation functions like NtAllocateVirtualMemory, and therefore i'm currently avoid hooking those functions.

Perhaps anybody has encountered a similar issue and implemented hook reentrancy guard in a way that doesn't triggered any memory allocation function (which might be imposible since even if you call a new function and your stack
memory is insufficient, it should allocate more memory).

thanks

[Siro]

You mean that inside the callback you have

Code: Select all

void MarkInsideMyHook(unsigned int Hook)
{
    auto ptr = TlsGetValue(g_tlsIndex);
    if (ptr == nullptr)
        ptr = LocalAlloc(....)  // <--- this calls NtAllocateVirtualMemory ?
     ...
}

You could build a static solution (which has its own drawbacks).
If you assume you handle at most MAX threads at once, then you can pre-allocate a global hashset on ATTACH_PROCESS, and use `idx = hash(tid) ` where you insert the relevant tid on first entry, and remove it when you leave.

Then you check the same hashtable for presense of tid.

If you have a present MAX size , you can avoid using allocations, and even work safely by using interlocked functions to set/get the tid inside the hashmap.