Why does Kernel32 OpenProcess function return null?

I’m trying to make an application that reads the memory of another (non-Java & 32bit) application using JNA. So far I know how to find process ID and base address of modules. And right before reading memory I need to open process and the OpenProcess function simply returns null. Also, I’m using Windows 10.

    // process id (pid) is known

    final int PROCESS_VM_READ=0x0010;
    final int PROCESS_QUERY_INFORMATION=0x0400;
      
    WinNT.HANDLE processHandle = Kernel32.INSTANCE.OpenProcess(PROCESS_VM_READ | PROCESS_QUERY_INFORMATION, true, pid);

How can I get a process handle?

Answer

You need to enable debug privilege for your current process in order to query information for processes owned by anyone other than the current user. The link shows the code in C, but you can port that code to JNA.

This is a one-time method call when your program starts up.

Here’s how I do it (hat tip to @RbMm for improvements):

/**
 * Enables debug privileges for this process, required for OpenProcess() to get
 * processes other than the current user
 *
 * @return {@code true} if debug privileges were successfully enabled.
 */
private static boolean enableDebugPrivilege() {
    HANDLEByReference hToken = new HANDLEByReference();
    boolean success = Advapi32.INSTANCE.OpenProcessToken(Kernel32.INSTANCE.GetCurrentProcess(),
            WinNT.TOKEN_QUERY | WinNT.TOKEN_ADJUST_PRIVILEGES, hToken);
    if (!success) {
        LOG.error("OpenProcessToken failed. Error: {}", Native.getLastError());
        return false;
    }
    try {
        WinNT.LUID luid = new WinNT.LUID();
        success = Advapi32.INSTANCE.LookupPrivilegeValue(null, WinNT.SE_DEBUG_NAME, luid);
        if (!success) {
            LOG.error("LookupPrivilegeValue failed. Error: {}", Native.getLastError());
            return false;
        }
        WinNT.TOKEN_PRIVILEGES tkp = new WinNT.TOKEN_PRIVILEGES(1);
        tkp.Privileges[0] = new WinNT.LUID_AND_ATTRIBUTES(luid, new DWORD(WinNT.SE_PRIVILEGE_ENABLED));
        success = Advapi32.INSTANCE.AdjustTokenPrivileges(hToken.getValue(), false, tkp, 0, null, null);
        int err = Native.getLastError();
        if (!success) {
            LOG.error("AdjustTokenPrivileges failed. Error: {}", err);
            return false;
        } else if (err == WinError.ERROR_NOT_ALL_ASSIGNED) {
            LOG.debug("Debug privileges not enabled.");
            return false;
        }
    } finally {
        Kernel32.INSTANCE.CloseHandle(hToken.getValue());
    }
    return true;
}