Friday, June 23, 2017

Hyper-V sockets and AfdConnect

  When we use sockets and call CONNECT method, it calls mswsock!WSPConnect. Then mswsock!WSPConnect calls NtDeviceIoControlFile. NtDeviceIoControlFile has following structure:
NTSTATUS WINAPI NtDeviceIoControlFile(
 _In_  HANDLE           FileHandle, - point to \Device\Afd
 _In_  HANDLE           Event,
 _In_  PIO_APC_ROUTINE  ApcRoutine, - 0
 _In_  PVOID            ApcContext, - 0
 _Out_ PIO_STATUS_BLOCK IoStatusBlock,- 0036F4C0
 _In_  ULONG            IoControlCode – 00012007 - IOCTL code
 _In_  PVOID            InputBuffer,- 0036F4E8
 _In_  ULONG            InputBufferLength, 30
 _Out_ PVOID            OutputBuffer,- 0
 _In_  ULONG            OutputBufferLength - 0
NtDeviceIoControlFile goes to execute of afd!AfdConnect function. It analyze InputBufferLength and, if it above AfdStandardAddressLength constant (eq 0x1C in Windows Server 2016) function, calls nt!ExAllocatePoolWithTagPriority
PVOID ExAllocatePoolWithTagPriority(
 _In_ POOL_TYPE        PoolType,- NonPagedPoolNx
 _In_ SIZE_T           NumberOfBytes,- (InputBufferLength-0xC)
 _In_ ULONG            Tag,- AfdR
 _In_ EX_POOL_PRIORITY Priority - LowPoolPriority

If we change InputBufferLength parameter of ntdll!NtDeviceIoControlFile from 0x30 to 0x4FFFFFFF before execution, we get:
kd> k
# Child-SP          RetAddr           Call Site
00 ffff8b80`7489c678 fffff805`53c46d9f nt!ExAllocatePoolWithTagPriority
01 ffff8b80`7489c680 fffff803`57293180 afd!AfdConnect+0x36f
02 ffff8b80`7489c820 fffff803`57292064 nt!IopSynchronousServiceTail+0x1a0
03 ffff8b80`7489c8e0 fffff803`572919e6 nt!IopXxxControlFile+0x674
04 ffff8b80`7489ca20 fffff803`56fd5493 nt!NtDeviceIoControlFile+0x56
05 ffff8b80`7489ca90 00000000`52dd222c nt!KiSystemServiceCopyEnd+0x13

kd> r
rdx=000000004ffffff3 – buffer size

kd> !poolused 2 AfdR
Sorting by NonPaged Pool Consumed
              NonPaged                  Paged
Tag     Allocs         Used     Allocs         Used
AfdR         6   1 342 178 160          0            0    Afd remote address buffer , Binary: afd.sys
TOTAL         6   1 342 178 160          0            0

Then user buffer (InputBuffer parameter) will be copy to allocated kernel pool by memove. If you don’t allocate user buffer page fault exception may be generated during of copy, especially if buffer size was set to hundreds of megabytes (exception will be handled by afd!AfdExceptionFilter). This kernel pool is freed by nt!ExFreePoolWithTag at the end of afd!AfdConnect.
It was tested with Hyper-V Sockets, but probably, it will work with standard TCP\IP sockets too.
I believe, that LowPoolPriority parameter of nt!ExAllocatePoolWithTagPriority will not allow operation system to hang because of limited kernel pool size.

Tested on afd version:

WINDBG>lmvm afd
   Image path: \SystemRoot\system32\drivers\afd.sys
   Image name: afd.sys
   Timestamp:        Sat Oct 15 06:53:45 2016 (5801A849)

Saturday, June 17, 2017

Hyper-V Sockets and PnP

How does Powershell Direct session work internally? When application, which uses Hyper-V socket, makes CONNECT call, root partition sends message to guest OS using winhvr!WinHvPostMessage:
kd> !dc @rdx – before vmcall (part of message body)
#227b36000 00000001 00000000 00000001 000000c4 ................
#227b36010 00000001 00000000 999e53d4 4c3e3d5c .........S..\=>L
#227b36020 d0be7987 e156c06e

If we look at the parameters of vmicvmsession, we find out, that it has trigger for start.

After the message was delivered to guest OS, vmbus driver added new interface:

kd> kcn
# Call Site
00 nt!IoRegisterDeviceInterface
01 Wdf01000!Mx::MxRegisterDeviceInterface
02 Wdf01000!FxDeviceInterface::Register
03 Wdf01000!FxDeviceInterface::Register
04 Wdf01000!imp_WdfDeviceCreateDeviceInterface
05 vmbus!RootStartDeviceInterfaceByContext
06 hvsocket!VmbusTlXPartProcessNewConnection
07 vmbus!RootNotifyDeviceInterfaceArrival
08 Wdf01000!FxWorkItem::WorkItemHandler
09 Wdf01000!FxWorkItem::WorkItemThunk
0a nt!IopProcessWorkItem
0b nt!ExpWorkerThread
0c nt!PspSystemThreadStartup
0d nt!KiStartSystemThread

And PnP mechanism works:
kd> kcn
# Call Site
00 nt!PnpNotifyDeviceClassChange
01 nt!PnpDeviceEventWorker
02 nt!ExpWorkerThread
03 nt!PspSystemThreadStartup

Service trigger works and service starts. After that icsvc.dll configures socket and begins communication with Host OS.
It is interesting that we can change that GUID to GUID of another device. We can get list of services, that have trigger on DEVICE INTERFACE ARRIVAL event:

All Hyper-V guest services
Bluetooth Support Service
Windows Camera Frame Server
Human Interface Device Service
Geolocation Service
Microsoft Passport
Portable Device Enumerator Service
Sensor Service
Sensor Monitoring Service
Storage Service
Touch Keyboard and Handwriting (probably)

And if we point that GUID in CONNECT call

clsid_str = L"{53f56307-b6bf-11d0-94f2-00a0c91efb8b}";
CLSIDFromString(clsid_str, &ServiceID); //when you call CONNECT with that GUID Portable Device Enumerator Service will start in guest OS (even in shielded VM)

guest OS starts this service, because it believes, that triggered device was appeared in system.
One point: before do this you must add GUID to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\GuestCommunicationServices as key in Host OS (need local admin rights)

It works on Shielded VM too (tested in Admin-Trusted mode), besides Hyper-V Powershell Direct Service – it not started

               It very interesting to see integration between Hyper-V sockets and PnP subsystem in Windows.

Powershell Direct (few internals). Part 2

According Powershell Direct session needs Hyper-V administrator privilege for user, which start that session. Yes, it is true, if you use Enter-PSSession cmdlet, but if you will be use Hyper-V sockets natively you don’t need any privileges for it.

Thursday, May 11, 2017

PowerShell Direct (few internals)

Some thoughts about new Windows Server 2016 and Windows 10 Hyper-V feature – PowerShell Direct. This feature is used for sending commands from host OS to guest OS without network communication. You can read about it on
In host OS you can enter
Enter-PSSession -VMName VMName
Invoke-Command -VMName VMName -ScriptBlock { Commands }

And then you can execute commands in guest OS. Before you can run command, you need enter guest OS credentials with local admin privileges.
After you enter it, PowerShell process in host OS send data to guest OS using Hyper-V sockets.
Byte[] domain = Encoding.Unicode.GetBytes(networkCredential.Domain);
Byte[] userName = Encoding.Unicode.GetBytes(networkCredential.UserName);
Byte[] password = Encoding.Unicode.GetBytes(networkCredential.Password);




In guest OS it handles by icsvc.dll (virtual machine integration component services), which runs as a service.
Guest OS administrator can attach WinDBG to this instance of svchost.exe and make command bp sspicli!LogonUserExExW, then g (yes, before it he can enter .logopen and then go to sleep). After host OS administrator enters guest OS credentials, WinDBG stops on sspicli!LogonUserExExW and you can see clear text login and password.
These credentials are needed for starting powershell.exe process (using CreateProcessAsUserW function).
It is not a bug, but, it seems, creates short nuance for enterprise environment - you can’t use PowerShell Direct if guest OS have more than one administrator and that administrator is not Hyper-V admin. He can catch you credentials very easy without needs to brute force it.
So, it not bad scenario attack continuation in VDI and Terminal Services virtual machines environment, where simple user can increase privileges to local admin. You can mitigate risks if you use unique local credentials for every virtual machine. Yes, if you use domain account for PowerShell Direct , don’t forget limit logon machines in account properties.

Thursday, February 23, 2017

Hyper-V sockets (without internals)

This is small article, which shows how convert simple network socket application to application, which uses Hyper-V socket for communication.
    Using software
  • Windows Server 2016 as Host;
  • Windows Server 2016 as Guest;
  • Visual Studio 2015 and Windows 10 as development environment.

  1. Go to msdn and read about Hyper-V Sockets (1). Read that you need Windows SDK build 10.0.14393.795 (recently QFE for 10.0.14393.033)
  2. Run PowerShell script on Host Server

$friendlyName = "HV Socket Application"

# Create a new random GUID and add it to the services list then add the name as a value

$service = New-Item -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\GuestCommunicationServices" -Name ((New-Guid).Guid)

$service.SetValue("ElementName", $friendlyName)

# Copy GUID to clipboard for later use
Write-Host "Service GUID: " $service.PSChildName

  1. Remember GUID
  2. Google (or wrote) application which works with Hyper-V Sockets. I get example from (2)
We open project in Visual Studio 2015 and convert it. We see two projects in solution: Client Example and Server Example. For every project we need change Target Platform Version to 10.0.14393.0
MSDN article gives GUID definitions
Listeners should bind to this VmId to accept connection from all partitions.

Wildcard address for children. Listeners should bind to this VmId to accept connection from its children.
Loopback address. Using this VmId connects to the same partition as the connector.
Parent address. Using this VmId connects to the parent partition of the connector.*

But not define HV_PROTOCOL_RAW constant. Value HV_PROTOCOL_RAW is 1.
Now we change code in client:
1. Add definitions:
    USHORT Reserved;
    GUID VmId;
    GUID ServiceId;
  1. Define GUIDs
#include <combaseapi.h>
wchar_t* clsid_str = L"{a42e7cda-d03f-480c-9cc2-a4de20abb878}"; // HV_PARENT_GUID
CLSIDFromString(clsid_str, &VmID);
    clsid_str = L"{b1d00d3e-fe10-4570-ad62-7648779d7a1b}";
CLSIDFromString(clsid_str, &ServiceID); //GUID of Service, generated by powershell
        //configure protocol
        ZeroMemory(&clientService, sizeof(clientService));
        clientService.Family = AF_HYPERV;
        clientService.VmId = *vmId;
        clientService.ServiceId = *serviceId;

  1. Change protocol parameters:

        hints.ai_family = AF_HYPERV;
        hints.ai_socktype = SOCK_STREAM;   
        hints.ai_protocol = HV_PROTOCOL_RAW;
  1. Comment getaddrinfo block.
  2. Replace socket connection string
//ConnectSocket = socket(ptr->ai_family, ptr->ai_socktype, ptr->ai_protocol); old code
ConnectSocket = socket(hints.ai_family, hints.ai_socktype, hints.ai_protocol);
  1. Comment Stop() function and its calls

Server code:
  1. Add GUIDS
DEFINE_GUID(HV_GUID_PARENT,0xa42e7cda,0xd03f,0x480c, 0x9c, 0xc2, 0xa, 0x4, 0xde, 0x20, 0xab, 0xb8, 0x78);
DEFINE_GUID(HV_GUID_ZERO,0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0);
  1. Initialize GUIDS

    CLSID ServiceID;
    wchar_t* clsid_str = L"{b1d00d3e-fe10-4570-ad62-7648779d7a1b}";
    CLSIDFromString(clsid_str, &ServiceID); //GUID of Service, generated by powershell
  1. Change protocol parameters
ZeroMemory(&clientService, sizeof(clientService));
    clientService.Family = AF_HYPERV;
    clientService.VmId = HV_GUID_ZERO;
    clientService.ServiceId = *serviceId;

    ZeroMemory(&hints, sizeof(hints));
    hints.ai_family = AF_HYPERV;        // Internet address family is unspecified so that either an IPv6 or IPv4 address can be returned
hints.ai_socktype = SOCK_STREAM;    // Requests the socket type to be a stream socket for the TCP protocol
hints.ai_protocol = HV_PROTOCOL_RAW;

hints.ai_addrlen = sizeof(SOCKADDR_HV);
    hints.ai_addr = reinterpret_cast<SOCKADDR *>(&clientService);
  1. Replace code
//ListenSocket = socket(result->ai_family, result->ai_socktype, result->ai_protocol); - orig.
ListenSocket = socket(hints.ai_family, hints.ai_socktype, hints.ai_protocol);

  1. Replace code
//iResult = bind(ListenSocket, result->ai_addr, (int)result->ai_addrlen); - orig.
iResult = bind(ListenSocket, hints.ai_addr, (int)hints.ai_addrlen);

Compile solution. You get ClientExample.exe and ServerExample.exe. Run ServerExample on host server and ClientExample in guest virtual machine. Enter “test” in ClientExample.exe and see it as output of ServerExample.

            After you register service you can run applications with simple user privileges on host server or virtual machine. It works like usual network application and doesn't need any special permissions for work.
            P.S. If you use one of existring GUID as ServiceID:
            7FDFD0EA-CEA8-4576-92D6-E072DDD2C422 - Machine Provisioning Service
            ACEF5661-84A1-4E44-856B-6245E69F4620 - Host Compute Service
            999E53D4-3D5C-4C3E-8779-BED06EC056E1 - VM Session Service 1
            A5201C21-2770-4C11-A68E-F182EDB29220 - VM Session Service 2
            you don't need administrative privileges for using Hyper-V sockets

  1. - very good article. But code compiles with errors.
  2. - source code from article.