Gerhart
@gerhart_x
|
Terms and definitions
- Root-partition (parent OS, root OS) — Windows Server 2016 with the installed Hyper-V;
- Guest OS (child partition, guest OS) — the virtual machine with Windows Server 2016 Gen2;
- Hyper-V TLFS – Hyper-V Top level functional specification.
- Intro
2006. WinHec conference. Microsoft actively advances the own variant of a hypervisor (then it had no name yet, and it was designated just as “Windows hypervisor”) and even does hints on the fact, that developers will be able to create its own decisions on the basis of new virtualization technology:
Really, certain steps were taken in this direction and at developers got:
- header files hvgdk.h, vid.h, VidDefs.h (Windows WDK 6.0, 7.1, Singularity OS);
- Hyper-V Top Level Functional Specification;
- documentation on msdn which generally coincided with TLFS, but contained more detailed information;
- on osronline.com the architect of Hyper-V Jake Oshins answered questions of developers of the drivers concerning the Hyper-V environment.
But nevertheless, the published information was obvious insufficiently in order that someone began to develop new products basing of Hyper-V (only livecloudkd from moonsols is remembered and, seemingly, the most part of information developers got from reverse engineering). Perhaps, in this regard, the policy of Microsoft sharply changed:
- Header files were removed from WDK (part still exists in singularity OS);
- documentation from MSDN disappeared (on osronline.com for formality conducted https://www.osronline.com/showthread.cfm survey? link=254171, is necessary documentation or not);
- Hyper-V extentions for WinDBG (network virtualization kernel debugger extension nvkd.dll), hvexts.dll, which mentioned by WinDBG at connection to hvix64.exe (hvax64.exe), was not laid out in the general access;
- Jake Oshins (the Hyper-V Architect) disappeared from the osronline.com forum.
Nevertheless, Microsoft independently began to develop Linux Integration Services - set of modules and drivers allowing to start Linux in Hyper-V. Its source codes are integrated into Linux kernel (2.6.32 and above), and, respectively, are uploaded publicly.
TLFS remains the only source of hypervisor internals information, however the specification issued in February, 2017 for Windows Server 2016 contains already 238 pages, but not 420 as it was in the previous specification for Windows Server 2012 R2 (from 23 sections remained 16, descriptions of many hyper calls disappeared, however two sections describing VSM work - Virtual Secure Mode, and the enclosed virtualization which support appeared in Windows Server 2016, were added)
But in 2016 in the ws2def.h file (core definitions for the Winsock2 specification) in the Windows SDK 10.0.10586 the line appeared
#define AF_HYPERV 34
and the HvSocket.h header was added to the Windows SDK 10.0.14393. In what purposes it was made?
New function - Powershell Direct which allows to carry out Powershell commands in a guest operating system without network connection was added to Windows Server 2016, transmitting all necessary data through the VMBus. This mechanism works, using so-called Hyper-V sockets, which were integrated into a network stack of Windows. This article became result of attempt to understand, how the mechanism of network interaction in Windows works and how constructs built-in support of Hyper-V sockets and that is carried out by an operating system during the work of the new protocol.
At first, we will consider in what way the virtualization subsystem was integrated with a network stack of Windows, then we will investigate operation of application, which working with Hyper-V sockets, and we learn how PowerShell Direct technology works. Windows 10 also supports Hyper-V sockets, but in the article it is practically not considered - the emphasis is placed on server OS, but, presumably, essential difference in realization should not be.
Before reading of article is recommended to study section 7 "Network" of the book "Windows Internals, the 6th edition". Perhaps, by the time of the publication of article in sale there will already be the 7th edition with that chapter (Part II or III), and also with excellent article "The network program Windows Vista/2008 interface: the internal device, use and breaking", earlier available on wasm.ru. Now it can be found on various websites (for example, https://vxlab.info/wasm/article.php-article=npi_subvert.htm). MSDN also quite in detail lights the materials mentioned in article, but without concrete of Hyper-V sockets.
- Components of an operating system
At first we will look at the list of the providers (Layered Service Provider - LSP), established in an operating system. We see two records, which name contains Hyper-V.
PS C:\Windows\system32> netsh winsock show catalog
Winsock Catalog Provider Entry
------------------------------------------------------
Entry Type: Base Service Provider
Description: Hyper-V RAW
Provider ID: {1234191B-4BF7-4CA7-86E0-DFD7C32B5445}
Provider Path: %SystemRoot%\system32\mswsock.dll
Catalog Entry ID: 1001
Version: 2
Address Family: 34
Max Address Length: 36
Min Address Length: 36
Socket Type: 1
Protocol: 1
Service Flags: 0x20026
Protocol Chain Length: 1
If we decipher Service Flags according to the description WSAPROTOCOL_INFO, then we will receive 0x20026 = XP1_GUARANTEED_DELIVERY | XP1_GUARANTEED_ORDER | XP1_GRACEFUL_CLOSE | XP1_IFS_HANDLES
In the registry for each provider one section is created (32-bit and 64-bit):
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters\Protocol_Catalog9\Catalog_Entries\000000000001 и HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters\Protocol_Catalog9\Catalog_Entries64\000000000001
In the section HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Winsock\Parameters vmbus transport was added (irda and RFCOMM in Windows Server 2016 in default installation are absent and is only in Windows 10):
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\vmbus\Parameters key contains the subsection Winsock, containing the HelperDllName parameter, in which the name of the wshhyperv.dll library loaded by the main provider mswsock.dll at a stage of the socket creation.
In the Windows Internals 6th edition is written: "Note The Raw transport protocol is not really a protocol and does not perform any encapsulation of the user data. This allows the client to directly control the contents of the frames transmitted and received by the network interface." In our case the Hyper-V RAW protocol is used. However, in spite of the fact that the name of protocol contains RAW, when operation of socket create is executed and socket function is called SOCK_STREAM second parameter (socket type - stream socket), though at WinSock2.h contains a separate type of a socket - SOCK_RAW.
There was a new NPI provider - hvsocket.sys. Is present at import of vmbus.sys, vmbusr.sys and netio.sys drivers and, respectively, it is loaded together with one of the specified modules, which started first (usual netio.sys). Provider is registered by vmbusr.sys driver by a call of the imported function hvsocket!HvSocketProviderStart, which next calls netio! NmrRegisterProvider.
Detail working with providers was described in article "The network program Windows Vista/2008 interface: internal device, use and breaking" (https://vxlab.info/wasm/print.php-article=npi_subvert.htm)
Using WinDBG it is possible to receive the list of all registered providers and their clients. For providers, it is enough to write a script (we put bp on netio!NmrRegisterProvider and write value of parameters in log file):
Windbg> bu netio!NmrRegisterProvider
Windbg>.logopen D:\ida_files\2016\log.txt
Windbg>bp netio!NmrRegisterProvider ".echo **********bp netio!NmrRegisterProvider********; .echo kc;kc; .echo dps rcx;dps rcx; .echo NpiId GUID; dt _GUID poi(rcx+28h); .echo NPI_MODULEID_TYPE GUID; dt _GUID poi(rcx+30h)+8; g"
NTSTATUS NmrRegisterProvider(
_In_ PNPI_PROVIDER_CHARACTERISTICS ProviderCharacteristics,
_In_ PVOID ProviderContext,
_Out_ PHANDLE NmrProviderHandle
);
typedef struct _NPI_REGISTRATION_INSTANCE {
USHORT Version;
USHORT Size;
PNPIID NpiId;
PNPI_MODULEID ModuleId;
ULONG Number;
const VOID *NpiSpecificCharacteristics;
} NPI_REGISTRATION_INSTANCE, *PNPI_REGISTRATION_INSTANCE;
typedef struct _NPI_MODULEID {
USHORT Length;
NPI_MODULEID_TYPE Type;
union {
GUID Guid;
LUID IfLuid;
};
} NPI_MODULEID, *PNPI_MODULEID;
Registration of hvsocket.sys as provider will look so
kc
# Call Site
00 NETIO!NmrRegisterProvider
01 hvsocket!HvSocketProviderStart
02 vmbusr!RootDeviceAdd
03 Wdf01000!FxDriverDeviceAdd::Invoke
04 Wdf01000!FxDriver::AddDevice
05 nt!PpvUtilCallAddDevice
06 nt!PnpCallAddDevice
07 nt!PipCallDriverAddDevice
08 nt!PipProcessDevNodeTree
09 nt!PiProcessStartSystemDevices
0a nt!PnpDeviceActionWorker
0b nt!ExpWorkerThread
0c nt!PspSystemThreadStartup
0d nt!KiStartSystemThread
dps rcx
fffff806`0dece010 00000000`00480000
fffff806`0dece018 fffff806`0ded1640 hvsocket!HvSocketNotifyAttachClient - ProviderAttachClient
fffff806`0dece020 fffff806`0ded18c0 hvsocket!HvSocketNotifyDetachClient - ProviderDetachClient
fffff806`0dece028 fffff806`0ded19a0 hvsocket!HvSocketNotifyCleanupClientContext - ProviderCleanUpBindingContext
fffff806`0dece030 00000000`00280000 – Begin of NPI_REGISTRATION_INSTANCE (Version+Size)
fffff806`0dece038 fffff806`0decc3e0 hvsocket!NPI_TRANSPORT_LAYER_ID – pointer to NpiId (GUID NPIID) - dt _GUID poi(rcx+28h)
fffff806`0dece040 fffff806`0decc3f0 hvsocket!NPI_MS_VMBUS_MODULEID – pointer to ModuleId – dt _GUID poi(rcx+30h)+8
fffff806`0dece048 00000000`00000000 - Number
fffff806`0dece050 fffff806`0decc2e0 hvsocket!VmbusTlProviderCharacteristics - NpiSpecificCharacteristics
fffff806`0dece058 00000000`00000000
fffff806`0dece060 00000500`00000000
fffff806`0dece068 0000ef8b`4509d61c
fffff806`0dece070 00000000`00000000
fffff806`0dece078 00000000`00000000
fffff806`0dece080 00000000`00000000
fffff806`0dece088 fffff803`6c322884 nt!EtwRegisterClassicProvider
NpiId GUID
ntdll!_GUID
{2227e804-8d8b-11d4-abad-009027719e09}
+0x000 Data1 : 0x2227e804
+0x004 Data2 : 0x8d8b
+0x006 Data3 : 0x11d4
+0x008 Data4 : [8] "???"
NPI_MODULEID_TYPE GUID
ntdll!_GUID
{eb004a27-9b1a-11d4-9123-0050047759bc}
+0x000 Data1 : 0xeb004a27
+0x004 Data2 : 0x9b1a
+0x006 Data3 : 0x11d4
+0x008 Data4 : [8] "???"
Logging of registration of NPI clients is similarly carried out. The only thing that breakpoint necessary to be put on netio! NmrRegisterClient. It is interesting that registration of hvsocket.sys as client is not noticed anywhere. The only registered components of virtualization is NDIS! NPI_NDIS_VBUS_INTERFACE_ID (registration goes from NDIS!DriverEntry) and vmswitch!NPI_PKTCAP_INTERFACE_ID (registration from vmswitch!DriverEntry).
The complete list of the providers and clients registered by Windows Server 2016 is provided on Gihub (https://github.com/gerhart01/HyperV-sockets/blob/master/RegisteredProviders.xlsx) in the following format:
Windows Server 2016 in default installation in virtual lab registers 54 providers.
In afd.sys there is an function afd!AfdTlNotifyAttachProvider (client module's ClientAttachProvider callback function) which works with structure AfdTlTransportListHead. We can use a small script for pykd which get a part of the address family elements and function of processing of each element:
2: kd> !py D:\afd_parse_AfdTlTransportListHead.py
cs:AfdTlTransportListHead address is 0xffffb089b4579040L
----Address family 0x0 [ AF_UNSPEC ]
--Dispatch function tcpip!TcpTlProviderDispatch
----Address family 0x0 [ AF_UNSPEC ]
--Dispatch function tcpip!UdpTlProviderDispatch
----Address family 0x0 [ AF_UNSPEC ]
--Dispatch function tcpip!RawTlProviderDispatch
----Address family 0x22 [ AF_HYPERV ]
--Dispatch function hvsocket!VmbusTlProviderDispatch
In principle, the afd command of mex extention for WinDBG has to output similar information, but at my stand it for any reason did not work (perhaps, private symbols are necessary).
At start of vmbusr.sys we see start of hvsocket!HvSocketProviderStart after which afd!AfdTlNotifyAttachProvider is caused.
0>kc
# Call Site
00 NETIO!NmrClientAttachProvider
01 afd!AfdTlNotifyAttachProvider
02 NETIO!NmrpProposeAttachment
03 NETIO!NmrpAttachArray
04 NETIO!NmrpRegisterModule
05 NETIO!NmrRegisterProvider
06 hvsocket!HvSocketProviderStart
07 vmbusr!RootDeviceAdd
In hvsocket!HvSocketProviderStart occurs a call of next functions:
netio!NetioInitializeWorkQueue
netio!NmrRegisterProvider
hvsocket.sys – NPI-провайдер
the hvsocket.sys driver was added for checks to netio!NmrpVerifyModule (in compare with early mentioned wasm article, which describe 3 module)
RtlInitString(&strAfd, "\\systemroot\\system32\\drivers\\afd.sys");
RtlInitString(&strTdx, "\\systemroot\\system32\\drivers\\tdx.sys");
RtlInitString(&strTcpip, "\\systemroot\\system32\\drivers\\tcpip.sys");
RtlInitString(&strHvsocket, "\\systemroot\\system32\\drivers\\hvsocket.sys");
Script afd_parse_AfdEndpointListHead_pykd.py can get the list of the objects created when opening each socket. If the socket is closed, then an object disappears from the list. The script, in principle, displays the same, as the utility of tcpconnect from Sysinternals Suite (but it, unfortunately, does not display opened Hyper-V sockets) or http://codemachine.com/article_findafdendpoints.html script, with an additional conclusion of the driver name and the procedure processing operations with a socket and also a name of process and PID.
For example, there is contents of lists in guest and root OS after successful execution of a Enter-PSSession cmdlet:
kd> !py C:\Tools\Scripts\afd_parse_AfdEndpointListHead_pykd.py – в гостевой ОС
afd!AfdEndpointListHead address is 0xfffff80490ef74e0L
----AfdEndpoint 0xfffff80490ef74e0L 0xda10
----AfdEndpoint 0xffffd00d64e7c130L 0xafd2 tcpip!TcpTlProviderEndpointDispatch explorer.exe 0x3c0
----AfdEndpoint 0xffffd00d6421af60L 0xafd2 hvsocket!VmbusTlProviderEndpointDispatch powershell.exe 0x920
----AfdEndpoint 0xffffd00d64846ea0L 0xafd4 hvsocket!VmbusTlProviderListenDispatch powershell.exe 0x920
----AfdEndpoint 0xffffd00d64d082a0L 0xafd1 tcpip!UdpTlProviderEndpointDispatch lsass.exe 0x204
----AfdEndpoint 0xffffd00d642b8960L 0xafd1 tcpip!UdpTlProviderEndpointDispatch lsass.exe 0x204
----AfdEndpoint 0xffffd00d640c8ba0L 0xafd4 hvsocket!VmbusTlProviderListenDispatch svchost.exe 0x35c (в состав процесса входит служба vmicsession)
----AfdEndpoint 0xffffd00d650c3c30L 0xaafd 0 explorer.exe 0x3c0
----AfdEndpoint 0xffffd00d64340f60L 0xaafd 0 explorer.exe 0x3c0
---------------------------------------------------------------------------------------------
kd> !py D:\ida_files\afd_parse_AfdEndpointListHead_pykd.py – в родительской ОС
afd!AfdEndpointListHead address is 0xfffff807668b74e0L
----AfdEndpoint 0xfffff807668b74e0L 0xe4a0
----AfdEndpoint 0xffff958f202eb300L 0xafd2 hvsocket!VmbusTlProviderEndpointDispatch powershell.exe 0xcc4L
----AfdEndpoint 0xffff958f21d7eac0L 0xafd1 tcpip!UdpTlProviderMessageDispatch svchost.exe 0x438L
----AfdEndpoint 0xffff958f207dd9e0L 0xafd2 hvsocket!VmbusTlProviderEndpointDispatch powershell_ise 0x394L
----AfdEndpoint 0xffff958f1fbc5f60L 0xafd1 tcpip!UdpTlProviderMessageDispatch svchost.exe 0x498L
---------------------------------------------------------------------------------------------
It interesting, we can see one more socket created by process of svchost.exe at early stages of operating system loading:
kd> !py D:\ida_files\ParseAfdEndpointListHead.py
---------------------------------------------------------------------------------------------
----AfdEndpoint 0xffffd1851637e130L 0xafd0 tcpip!TcpTlProviderEndpointDispatch svchost.exe 0x410L
----AfdEndpoint 0xffffd1851627ed60L 0xafd0 hvsocket!VmbusTlProviderEndpointDispatch svchost.exe 0x384L
----AfdEndpoint 0xffffd185161d7330L 0xafd0 tcpip!TcpTlProviderEndpointDispatch wininit.exe 0x2a4L
---------------------------------------------------------------------------------------------
This socket is created by the RPC service, namely the function! RPCRT4TransportProtocol::HandlePnPStateChange.
Stack:
kd> k – bp on wshhyperv.dll load
13 mswsock!SockGetTdiName+0x2b1
14 mswsock!SockSocket+0x117
15 mswsock!WSPSocket+0x220
16 WS2_32!WSASocketW+0x1f0
17 RPCRT4!TransportProtocol::OpenAddressChangeRequestSocket+0x43
18 RPCRT4!TransportProtocol::VerifyProtocolIsFunctional+0x14
19 RPCRT4!TransportProtocol::HandleProtocolChange+0x100
1a RPCRT4!TransportProtocol::HandlePnPStateChange+0x72
1b RPCRT4!ProcessNewAddressEvent+0x21
1c RPCRT4!COMMON_AddressChangeThreadPoolCallback+0x25
1d KERNELBASE!BasepTpIoCallback+0x50
1e ntdll!TppIopExecuteCallback+0x118
1f ntdll!TppWorkerThread+0x8ed
20 KERNEL32!BaseThreadInitThunk+0x14
21 ntdll!RtlUserThreadStart+0x21
Function ws2_32!WSAEnumProtocols called from RPCRT4!TransportProtocol::HandlePnPStateChange. Works with rpcrt4!TransportProtocolArray based on result of ws2_32!WSAEnumProtocols. TransportProtocol::HandleProtocolChange called for each element of the table TransportProtocol (the second parameter - structure of WSAPROTOCOL_INFOW). The size of each TransportProtocolArray element - 72 bytes. But type of this socket - 0xafd0. The structure _AFD_CONNECTION, describes a condition of such socket, up to the end is not filled, and on offset +e0 from the beginning of structure there are only zero. To have a possibility of connection to Hyper-V sockets, its type has to be at least 0xafd2.
There are many functions working with Hyper-V sockets in rpcrt4.dll symbols:
HVSOCKET_QueryClientID
HVSOCKET_BuildAddressVector
HVSOCKET_Open
HVSOCKET_QueryClientAddress
HVSOCKET_REsolveAddress
HVSOCKET_ResolveVmId
HVSOCKET_ServerListen
HVSOCKET_SetSocketOption
The purposes of addition of support in RPC library are unknown. According to MSDN PowerShell Direct has to work locally and actually it does not use RPC for the work.
Perhaps, these functions are necessary for work in the Docker environment, or for future support of distant work of PowerShell Direct.
- Hyper-V socket work
On msdn only one page describes steps which need to be executed to create the application for work with Hyper-V sockets (https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/make-integration-service). As an example, we will take the simple application found in Internet and showing work with standard network sockets, and we modify it so that it for data transmission used sockets of Hyper-V (https://github.com/gerhart01/HyperV-sockets). The application consists of a client and server part. Client part transfers the text typed in the console to server, using Hyper-V sockets for communication.
According to MSDN Hyper-V sockets support the following commands: Socket, Bind, Connect, Send, Listen, Accept.
However in practice it is visible that the bigger number of commands is supported:
Server part of the application execute socket, bind, listen, accept, recv, closesocket.
Client part - socket, connect, send, recv, shutdown, closesocket.
Let's consider our applcation as there is a work with Hyper-V sockets. At once I will tell that in general interfaces of interaction are very similar to usual network sockets, only realization details differ.
In general, the logic of interaction (from the point of view of interaction with windows kernel) looks as follows:
Server
|
Client
|
Afd!AfdCreate
00 ntdll!NtCreateFile
01 mswsock!SockSocket
02 mswsock!WSPSocket
03 WS2_32!WSASocketW
04 WS2_32!socket
05 ServerExample!main
| |
afd!AfdFastIoDeviceControl
00 ntdll!NtDeviceIoControlFile (IOCTL - 1207b)
01 mswsock!SockGetInformation
02 mswsock!SockSocket
03 mswsock!WSPSocket
04 WS2_32!WSASocketW
05 WS2_32!socket
06 ServerExample!main
| |
afd!AfdFastIoDeviceControl
00 ntdll!NtDeviceIoControlFile (IOCTL - 1207b)
01 mswsock!SockGetInformation
02 mswsock!SockSocket
03 mswsock!WSPSocket
04 WS2_32!WSASocketW
05 WS2_32!socket
06 ServerExample!main
| |
afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL - 12047)
01 mswsock!SockSetHandleContext
02 mswsock!WSPSocket
03 WS2_32!WSASocketW
04 WS2_32!socket
05 ServerExample!main
| |
afd!AfdBind
00 ntdll!NtDeviceIoControlFile (IOCTL - 12003)
01 mswsock!WSPBind
02 WS2_32!bind
03 ServerExample!main
| |
afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL - 12047)
01 mswsock!SockSetHandleContext
02 mswsock!WSPBind
03 WS2_32!bind
04 ServerExample!main
| |
afd!AfdStartListen
00 ntdll!NtDeviceIoControlFile (IOCTL - 1200b)
01 mswsock!WSPListen
02 WS2_32!listen
03 ServerExample!main
| |
afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL – 12047)
01 mswsock!SockSetHandleContext
02 mswsock!WSPListen
03 WS2_32!listen
04 ServerExample!main
| |
afd!AfdWaitForListen
00 ntdll!NtDeviceIoControlFile (IOCTL - 1200c)
01 mswsock!WSPAccept
02 WS2_32!WSAAccept
03 WS2_32!accept
04 ServerExample!main
| |
Afd!AfdCreate
01 mswsock!SockSocket
02 mswsock!WSPSocket
03 WS2_32!WSASocketW
04 WS2_32!socket
05 ClientExample!Client::Start
06 ClientExample!main
| |
afd!AfdFastIoDeviceControl
00 ntdll!NtDeviceIoControlFile (IOCTL - 1207b)
01 mswsock!SockGetInformation
02 mswsock!SockSocket
03 mswsock!WSPSocket
04 WS2_32!WSASocketW
05 WS2_32!socket
06 ClientExample!Client::Start
07 ClientExample!main
| |
afd!AfdFastIoDeviceControl
00 ntdll!NtDeviceIoControlFile (IOCTL - 1207b)
01 mswsock!SockGetInformation
02 mswsock!SockSocket
03 mswsock!WSPSocket
04 WS2_32!WSASocketW
05 WS2_32!socket
06 ClientExample!Client::Start
07 ClientExample!main
| |
afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL - 12047)
01 mswsock!SockSetHandleContext
02 mswsock!WSPSocket
03 WS2_32!WSASocketW
04 WS2_32!socket
05 ClientExample!Client::Start
06 ClientExample!main
| |
afd!AfdBind
00 ntdll!NtDeviceIoControlFile (IOCTL - 12003)
01 mswsock!WSPBind
02 mswsock!SockDoConnect
03 mswsock!WSPConnect
04 WS2_32!connect
05 ClientExample!Client::Start
06 ClientExample!main
| |
afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL - 12047)
01 mswsock!SockSetHandleContext
02 mswsock!WSPBind
03 mswsock!SockDoConnect
04 mswsock!WSPConnect
05 WS2_32!connect
06 ClientExample!Client::Start
07 ClientExample!main
| |
afd!AfdConnect
00 ntdll!NtDeviceIoControlFile (IOCTL - 12007)
01 mswsock!SockDoConnectReal
02 mswsock!SockDoConnect
03 mswsock!WSPConnect
04 WS2_32!connect
05 ClientExample!Client::Start
06 ClientExample!main
| |
Afd!AfdCreate
01 mswsock!SockSocket
02 mswsock!WSPAccept
03 WS2_32!WSAAccept
04 WS2_32!accept
05 ServerExample!main
| |
afd!AfdAccept
00 ntdll!NtDeviceIoControlFile (IOCTL - 12010)
01 mswsock!WSPAccept
02 WS2_32!WSAAccept
03 WS2_32!accept
04 ServerExample!main
|
afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL - 12047)
01 mswsock!SockSetHandleContext
02 mswsock!SockPostProcessConnect
03 mswsock!SockDoConnectReal
04 mswsock!SockDoConnect
05 mswsock!WSPConnect
06 WS2_32!connect
07 ClientExample!Client::Start
08 ClientExample!main
|
Afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL - 12037)
01 mswsock!SockCoreAccept
02 mswsock!WSPAccept
03 WS2_32!WSAAccept
04 WS2_32!accept
05 ServerExample!main
| |
afd!AfdDispatchImmediateIrp
00 ntdll!NtDeviceIoControlFile (IOCTL - 12047)
01 mswsock!SockSetHandleContext
02 mswsock!SockCoreAccept
03 mswsock!WSPAccept
04 WS2_32!WSAAccept
05 WS2_32!accept
06 ServerExample!main
| |
afd!AfdFastIoDeviceControl
00 ntdll!NtDeviceIoControlFile (IOCTL - 12017)
01 mswsock!WSPRecv
02 WS2_32!recv
03 ServerExample!main
| |
afd!AfdFastConnectionSend
00 ntdll!NtDeviceIoControlFile (IOCTL – 1201F)
01 mswsock!WSPSend
02 WS2_32!send
03 ClientExample!Client::Send
04 ClientExample!main
| |
afd!AfdFastConnectionSend
00 ntdll!NtDeviceIoControlFile (IOCTL – 1201F)
01 mswsock!WSPSend
02 WS2_32!send
03 ServerExample!main
| |
afd!AfdFastIoDeviceControl
00 ntdll!NtDeviceIoControlFile (IOCTL - 12017)
01 mswsock!WSPRecv
02 WS2_32!recv
03 ServerExample!main
|
afd!AfdFastIoDeviceControl
00 ntdll!NtDeviceIoControlFile (IOCTL - 12017)
01 mswsock!WSPRecv
02 WS2_32!recv
03 ClientExample!Client::Recv
04 СlientExample!main
|
I will note some differences on identical calls in the text. Also, some features of a client part will be reflected further within PowerShell Direct analysis.
In Hyper-V sockets there are no IP addresses, but there are in advance defined GUIDs:
Name
|
GUID
|
Description
|
HV_GUID_ZERO, HV_GUID_WILDCARD
|
00000000-0000-0000-0000-000000000000
|
Listeners should bind to this VmId to accept connection from all partitions.
|
HV_GUID_BROADCAST
|
FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF
| |
HV_GUID_CHILDREN
|
90db8b89-0d35-4f79-8ce9-49ea0ac8b7cd
|
Wildcard address for children. Listeners should bind to this VmId to accept connection from its children.
|
HV_GUID_LOOPBACK
|
e0e16197-dd56-4a10-9195-5ee7a155a838
|
Loopback address. Using this VmId connects to the same partition as the connector.
|
HV_GUID_PARENT
|
a42e7cda-d03f-480c-9cc2-a4de20abb878
|
Parent address. Using this VmId connects to the parent partition of the connector.*
|
In our case we will take HV_GUID_PARENT. The second GUID which is required to us is specially generated for the powershell service. For this purpose we start next PowerShell-script:
$friendlyName = "HV Socket Application"
# Create a new random GUID and add it to the services list then add the name as a value
$service = New-Item -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\GuestCommunicationServices" -Name ((New-Guid).Guid)
$service.SetValue("ElementName", $friendlyName)
# Copy GUID to clipboard for later use
Write-Host "Service GUID: " $service.PSChildName
and remember the received GUID. But, in principle, it is possible to use the existing GUID which are already created during Windows installation in the same section of the register:
HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\GuestCommunicationServices
GUID
|
Name
|
7FDFD0EA-CEA8-4576-92D6-E072DDD2C422
|
Machine Provisioning Service
|
ACEF5661-84A1-4E44-856B-6245E69F4620
|
Host Compute Service
|
999E53D4-3D5C-4C3E-8779-BED06EC056E1
|
VM Session Service 1
|
A5201C21-2770-4C11-A68E-F182EDB29220
|
VM Session Service 2
|
“VM Session Service 1” and “VM Session Service 2” are used for PowerShell Direct work (the second GUID is used before Hyper-V socket duplication mechanism will be released. If within the same PowerShell-session 2 connections open using New-PSSession, then 2 GUID is used).
If to try to open 3 connections at once, then the mistake will be returned:
and only two sessions will be open. When sending messages of guest OS, we can see both GUID.
But we will have only one channel for communication, and, respectively, one GUID is necessary: B1D00D3E-FE10-4570-AD62-7648779D7A1B
int iResult = WSAStartup(MAKEWORD(2,2), &wsaData);
we pass a call of WSAStartup because it hasn’t specific parameters for work with Hyper-V as sockets, we go to the socket function at once
- Socket
Values of all parameters look in the source code of the ServerExample, and we will pass directly to socket call:
ZeroMemory(&hints, sizeof(hints));
hints.ai_family = AF_HYPERV;
hints.ai_socktype = SOCK_STREAM;
hints.ai_protocol = HV_PROTOCOL_RAW;
ListenSocket = socket(hints.ai_family, hints.ai_socktype, hints.ai_protocol);
The code will be compiled in the following:
From ws2_32! WSASocketA is caused ws2_32! WSASocketW, from which was called ws2_32!DPROVIDER __ Initialize
WINDBG>dc poi(esp+4) L100 - – the 2nd parameter of the DPROVIDER function:: Initialize (value of the registry key Protocol_Catalog9\Catalog_Entries\000000000001)
0122b48c 00020026 00000000 00000000 00000000 &...............
0122b49c 00000008 1234191b 4ca74bf7 d7dfe086 ......4..K.L....
0122b4ac 45542bc3 000003e9 00000001 00000000 .+TE............
0122b4bc 00000000 00000000 00000000 00000000 ................
0122b4cc 00000000 00000000 00000002 00000022 ............"...
0122b4dc 00000024 00000024 00000001 00000001 $...$...........
0122b4ec 00000000 00000000 00000000 00000000 ................
0122b4fc 00000000 00790048 00650070 002d0072 ....H.y.p.e.r.-.
0122b50c 00200056 00410052 00000057 00000000 V. .R.A.W.......
0122b51c 00000000 00000000 00000000 00000000 ................
Further we see initialization of pointer on socket helper-functions
 | 
WINDBG>du poi(esp)
00d5f178 "C:\Windows\system32\mswsock.dll"
|
In general in usermode enough operations is carried out, but I will try to specify only the most important of them.
Further by means of LoadLibraryEx mswsock.dll is loaded, then GetProcAddress returns mswsock!WSPStartup address then execution is transferred to this function. Inside execute mswsock_initialize, then ws32SQMinit and WahCreateContextTable.
After completion of mswsock!WSPStartup, is caused the procedure mswsock!WSPSocket (through call esi) from which the function mswsock!socksocket is caused, then mswsock!sockGetTdiName, at the same time the first parameter goes:
WINDBG>dtx _GUID poi(esp+4)
(*((_GUID *)0xf6f4a0)) : {1234191B-4BF7-4CA7-86E0-DFD7C32B5445} [Type: _GUID]
WINDBG> dc poi(esp+4)
00f6f4a0 1234191b 4ca74bf7 d7dfe086 45542bc3 ..4..K.L.....+TE –Hyper-V RAW GUID (we can see in netsh output)
Then mswsock!SockLoadTransportList is caused, which reads out value of the section of the registry "SYSTEM\CurrentControlSet\Services\Winsock\Parameters\Transports"
The following values is returned:
WINDBG>dc @ebx – (in ebx the pointer on the memory block transferred to mswsock!SockLoadTransportList)
01224ee0 006d0076 00750062 00000073 00730050 v.m.b.u.s...P.s.
01224ef0 00680063 00640065 00540000 00700063 c.h.e.d...T.c.p.
01224f00 00700069 00540000 00700063 00700069 i.p...T.c.p.i.p.
01224f10 00000036 abab0000 abababab feeeabab 6..............
mswsock!SockLoadHelperDll is called, HKLM\System\CurrentControlSet\Services\vmbus\Parameters\Winsock\HelperDllName value is requested and the C:\Windows\SysWoW64\wshhyperv.dll library is loaded (the ServerExample is compiled as 32-bit)
When we return from mswsock!SockGetTdiName address of wshhyperv!WSHOpenSocket2 is returned. This function contains only checks of correctness transfer of a socket parameters.
Further GetCurrentProcess\OpenProcessToken is consecutive and then GetTokenInformation. We can see that, as _TOKEN_INFORMATION_CLASS is transferred 0x1D:
WINDBG>dt _TOKEN_INFORMATION_CLASS @esp+8
combase!_TOKEN_INFORMATION_CLASS
1d ( TokenIsAppContainer ) – probably it adaptation of sockets in the environment of the containers Docker or application will be compiled with /APPCONTAINER Visual studio key
then the result is stored in the variable mswsock!SockIsAppContainter, next we see initialization of \\Device\\Afd\\Endpoint string which is transferred to ntdll! NtCreateFile
WINDBG>dtx OBJECT_ATTRIBUTES poi(@esp+0x8) (3rd parameter ntdll!NtCreateFile)
(*((OBJECT_ATTRIBUTES *)0xf6f348)) [Type: OBJECT_ATTRIBUTES]
[+0x000] Length : 0x18 [Type: unsigned long]
[+0x004] RootDirectory : 0x0 [Type: void *]
[+0x008] ObjectName : 0xf6f33c : "\Device\Afd\Endpoint" [Type: _UNICODE_STRING *]
[+0x00c] Attributes : 0x42 [Type: unsigned long]
[+0x010] SecurityDescriptor : 0x0 [Type: void *]
[+0x014] SecurityQualityOfService : 0x0 [Type: void *]
WINDBG>dtx _IO_STATUS_BLOCK @esp+0xc -r (the 4th parameter of ntdll!NtCreateFile)
(*((_IO_STATUS_BLOCK *)0xf6f30c)) [Type: _IO_STATUS_BLOCK] – not initialized structure
[+0x000] Status : 16184168 [Type: long]
[+0x000] Pointer : 0xf6f368 [Type: void *]
[+0x004] Information : 0x0 [Type: unsigned long] – after execution will be returned the status of operation (FILE_CREATED, FILE_OPENED, FILE_OVERWRITTEN, FILE_SUPERSEDED, FILE_EXISTS, FILE_DOES_NOT_EXIST)
Further execution passes into a kernel to the afd.sys driver. At initialization this driver registers next handlers of IRP:
WINDBG>!drvobj afd 2
Driver object (ffffda8527de19c0) is for:
\Driver\AFD
DriverEntry: fffff803db38a000 afd!GsDriverEntry
DriverStartIo: 00000000
DriverUnload: fffff803db34c380 afd!AfdUnload
AddDevice: 00000000
Dispatch routines:
[00] IRP_MJ_CREATE fffff803db357e90 afd!AfdDispatch
[01] IRP_MJ_CREATE_NAMED_PIPE fffff803db357e90 afd!AfdDispatch
[02] IRP_MJ_CLOSE fffff803db357e90 afd!AfdDispatch
[03] IRP_MJ_READ fffff803db357e90 afd!AfdDispatch
[04] IRP_MJ_WRITE fffff803db357e90 afd!AfdDispatch
WINDBG>kn
# Child-SP RetAddr Call Site
00 afd!AfdDispatch
01 nt!IopParseDevice+0x1655
02 nt!ObpLookupObjectName+0x8b2
03 nt!ObOpenObjectByNameEx+0x1dd
04 nt!IopCreateFile+0x3d9
05 nt!NtCreateFile+0x79
06 nt!KiSystemServiceCopyEnd+0x13
07 ntdll!NtCreateFile+0x14
Respectively after call ntdll!NtCreateFile from our program we will go to Afd!AfdDispatch. First parameter of it:
WINDBG>!devobj @rcx
Device object (ffffda8527de29d0) is for:
Afd \Driver\AFD DriverObject ffffda8527de19c0
Current Irp 00000000 RefCount 79 Type 00000011 Flags 00000050
Dacl ffffcb8a7e8ccd11 DevExt 00000000 DevObjExt ffffda8527de2b20
ExtensionFlags (0x00000800) DOE_DEFAULT_SD_PRESENT
Characteristics (0x00020000) FILE_DEVICE_ALLOW_APPCONTAINER_TRAVERSAL
Device queue is not busy.
WINDBG>!devstack ffffda8527de29d0 – in a stack only one device
!DevObj !DrvObj !DevExt ObjectName
> ffffda8527de29d0 \Driver\AFD 00000000 Afd
Second parameter of Afd!AfdDispatch is IRP
WINDBG>!irp @rdx
Irp is active with 4 stacks 4 is current (= 0xffffda8529720378)
No Mdl: System buffer=ffffda8527088910: Thread ffffda8528714080: Irp stack trace.
cmd flg cl Device File Completion-Context
>[IRP_MJ_CREATE(0), N/A(0)]
0 0 ffffda8527de29d0 ffffda8528bc3550 00000000-00000000
\Driver\AFD
Args: ffff908026e7b5d0 03000020 00030000 00000039
We can be convinced that the package is sent by our application
WINDBG>!thread ffffda8528714080
THREAD ffffda8528714080 Cid 0f74.0ca8 Teb: 0000000000ddf000 Win32Thread: ffffda85272b94e0 RUNNING on processor 0
IRP List:
ffffda85297201d0: (0006,0310) Flags: 00000884 Mdl: 00000000
Not impersonating
DeviceMap ffffca89863843b0
Owning Process ffffda8527121080 Image: ServerExample.exe
Attached Process N/A Image: N/A
Wait Start TickCount 416475 Ticks: 1 (0:00:00:00.015)
Context Switch Count 3006 IdealProcessor: 0
Further Afd!AfdCreate is caused. The first parameter - the same IRP. Next - Afd!AfdCheckTDIFilter.
WINDBG>r – параметры Afd!AfdCheckTDIFilter
rcx=0000000000000001
rdx=0000000000000022 (Address Family - AF_HYPERV)
r8=0000000000000001
r9=0000000000000000
It make the search addresses family AF_HYPERV, which was passed this function as parameter, in structure AfdTdiMapping (6 elements, the size of an element 20h of bytes). The structure contains references to standard network Windows devices.
Any of these devices is not used for AF_HYPERV. Pointer to structure AfdTdiMapping was returned. Further - Afd! AfdAllocateEndpoint from which is caused Afd!AfdTlFindAndReferenceTransport.
rcx=0000000000000022 (Address Family - AF_HYPERV)
rdx=000000000000001
r8=0000000000000001
r9=0000000000000001
In this function there is a work with structure AfdTlTransportListHead. It contains the linked list of pointers on transports objects. The instruction of mov rbx, [rbx] occurs loading the next element and comparison of family of the addresses AF_HYPERV (0x22) with [rbx+16h] is carried out if coincides, then function returns the address of structure
WINDBG>dps @rax
ffffda85`27f6b8c0 fffff803`db337530 afd!AfdTlTransportListHead
ffffda85`27f6b8c8 ffffda85`27dfeca0
ffffda85`27f6b8d0 00220000`00000006
ffffda85`27f6b8d8 00000001`00000001
ffffda85`27f6b8e0 ffffda85`27f6ba80
ffffda85`27f6b8e8 fffff803`db95c000 hvsocket!VmbusTlProviderDispatch
ffffda85`27f6b8f0 11d49b1a`eb004a27
ffffda85`27f6b8f8 bc597704`50002391
ffffda85`27f6b900 ffffda85`27f6b96c
ffffda85`27f6b908 00000000`00000000
ffffda85`27f6b910 62524d4e`02080006
ffffda85`27f6b918 8ab20db4`3a180386
WINDBG>dt _GUID @rax+30h – содержит GUID NPI_MS_VMBUS_MODULEID
ServerExample!_GUID
{eb004a27-9b1a-11d4-9123-0050047759bc}
Further afd!PplGenericAllocationFunction is caused (allocation of necessary memory, after filling of necessary structures, next nt!NtAllocatePoolEx is caused). Then nt!ObjDerefernceObject is caused, at the same time is loaded the pointer on process (ServerExample.exe) into rcx.
WINDBG>!object @rcx
Object: ffffda8527121080 Type: (ffffda8527096f20) Process
ObjectHeader: ffffda8527121050 (new version)
HandleCount: 8 PointerCount: 207781
Further increases on 1 global AfdEndpointsOpened variable. At the time of debugging:
WINDBG>dd afd!AfdEndpointsOpened L4
fffff803`db3378e8 000009af 00000000 00000000 00000000
There is AfdEndpointListHead check whether it is empty
Then the new element is inserted into this structure
AfdEndpointListHead as we saw earlier, contains the created objects of sockets.
In principle, aim of afd!AfdAllocateEndpoint consists in creation of a new _AFD_CONNECTION element (the name of structure is not documented, but in Windows 2003, as far as I know, it was called quite so) and its addition in the AfdEndpointListHead array.
Then the condition of a socket changes (in type field constant 0xAFD was written)
The constants AFD, AFD1, AFD2, AFD4, AFD8, AAFD, etc. are indicators of a condition of connection. But it looks like no compliance with RFC 793 which describes possible conditions of sockets - LISTEN, SYN-SENT, SYN-RECEIVED, ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT. Also the byte to the left of a constant changes depending on call type 0001afd0. For example, after bind call its value will become equal to 3.
WINDBG>dc @rdi – pointer to structure _AFD_CONNECTION
ffffda85`2943e9e0 0001afd0 00000000 00000100 00000000 ................
ffffda85`2943e9f0 00000000 00000000 00000000 00000000 ................
ffffda85`2943ea00 00000000 00000000 27121080 ffffda85 ...........'....
Further hvsocket!VmbusTlEndpointIsPrivileged is caused, then afd!AfdTLCreateEndpoint.
WINDBG>r
rcx= ffffda852943e9e0 – pointer to _AFD_CONNECTION with which work is performed
rdx=0000000000000022 – Address Family
r8=0000000000000001 – Hyper-V RAW
r9=ffffda8527f6b8c0 – pointer to the one of AfdTlTransportListHead elements
In this function in a stack 50h byte are nullified and then parameters for the function hvsocket!VmbusTlProviderEndpoint are placed
WINDBG>dps @rsp+20
ffff9080`26e7b250 fffff803`db362930 afd!AfdTLCreateEndpointComplete
ffff9080`26e7b258 ffffda85`297201d0 - - IRP
ffff9080`26e7b260 00000000`00000000
ffff9080`26e7b268 00000001`00010022
ffff9080`26e7b270 ffffda85`2943e9e0 - pointer to _AFD_CONNECTION structure
ffff9080`26e7b278 ffffda85`27121080 - Process Object (it not really need. In hvsocket if it eq zero driver calls PsGetCurrentProcess)
ffff9080`26e7b280 ffffda85`28714080 - pointer to THREAD structure of ServerExmple.exe process (if it zero later will be called ndis!NdisGetProcessObjectCompartmentId before in rcx load Process struct)
ffff9080`26e7b288 ffffca89`8568e280
ffff9080`26e7b290 00000000`00000000
ffff9080`26e7b298 00000000`00000000
then the pointer to hvsocket!VmBusTlProviderEndpoint is loaded into rax, in r14 our IRP:
WINDBG>!irp @r14
Irp is active with 4 stacks 4 is current (= 0xffffdf08c0c5b408)
>[IRP_MJ_CREATE(0), N/A(0)]
0 0 ffffdf08bf161920 ffffdf08c0eb5780 00000000-00000000
\Driver\AFD
Args: ffffa700e57f35d0 03000020 00030000 00000039
The call of this function is carried out. Checks of the structure transferred to rax are carried out, then for an object of ETHREAD ndis!NdisGetThreadObjectCompartmentId is caused, then vmbus!VmbusTlCreateEndpoint (the 2nd parameter - pointer EPROCESS object), further hvsocket!VmbusTlCreateObjectFromLookasideList in which nt! ExpInterlockedPopEntrySList is caused.
Then hvsocket!VmbusTlInitializeObject (causes nt! KeInitializeEvent and nt!KeInitializeSpinLock). There is a return from hvsocket! VmbusTlCreateObjectFromLookasideList, then the memory block with the size 200h and the Vnpi tag is allocated. Then the memory block is nullified by the size 38h, nt!KeEnterCriticalRegion are caused and nt!ExAcquireFastMutexUnsafe, registration of hvsocket!VmbusTlEndpointActionWorkQueueRoutine is also carried out through netio!NetioInitializeWorkQueue.
Later there is a return from vmbus!VmbusTlCreateEndpoint, then afd!AfdTLCreateEndpointComplete(PIRP IRP)is caused by, the 4th parameter – pointer to vmbus!VmbusTlProviderEndpointDispatch. Next hvsocket!VmbusTlCreateEndpoint address is written in _AFD_CONNECTION element. Then afd!ObDereferenceSecurityDescriptor is caused, also goes check of successful result - depending on it nt!iofCompleteRequest is carried out or not. Then there is a return from hvsocket.sys to afd.sys.
The value returned to vmbus! VmBusTlProviderEndpoint - 103h. Therefore AfdTLPendRequest is caused further if it will return not 103h, but is at once afd!AfdCompleteTLEndpCreate call, and only after that there is afd!AfdCompleteTLEndpCreate and nt!IofCompleteRequest. There is a return from afd!AfdCreate (in the rax - same 103h).
WINDBG>dc 0xf6f368 – after execution of NtCreateFile
00f6f368 00000000 00000000 00f6f498 00000003 ................
00f6f378 00000144 00000000 00000000 00000022 D..........."...
00f6f388 00000000 00000144 c0140000 00000020 ....D....... ...
00f6f398 80000000 00000001 00000039 00000022 ........9..."...
0x144 - it is that handle which will be returned finally by the socket function and which will be transferred to the bind function as the first parameter. This handle is created by nt!ObpCreateHandle function caused from the nt! ObOpenObjectByNameEx (earlier when performing NtCreateFile).
WINDBG>k
Child-SP RetAddr Call Site
ffffd201`d9b6e820 fffff801`a72630e9 nt!ObOpenObjectByNameEx+0x310
ffffd201`d9b6e960 fffff801`a7262cf9 nt!IopCreateFile+0x3d9
ffffd201`d9b6ea00 fffff801`a6fd4493 nt!NtCreateFile+0x79
ffffd201`d9b6ea90 00007ff9`e3f46b74 nt!KiSystemServiceCopyEnd+0x13
00000000`00e3e0a8 00000000`58f9ae28 ntdll!NtCreateFile+0x14
Before execution of function in the list of objects:
!handle 0 f ffffda8527121080
013c: Object: ffffca8986a9e3d0 GrantedAccess: 00020019 (Inherit) Entry: ffffca8985cb14f0
Object: ffffca8986a9e3d0 Type: (ffffda852717d0e0) Key
ObjectHeader: ffffca8986a9e3a0 (new version)
HandleCount: 1 PointerCount: 32759
Directory Object: 00000000 Name: \REGISTRY\MACHINE\SYSTEM\SYSTEM\CONTROLSET001\SERVICES\WINSOCK2\PARAMETERS\NAMESPACE_CATALOG5
0140: Object: ffffda85299be6f0 GrantedAccess: 001f0003 (Audit) Entry: ffffca8985cb1500
Object: ffffda85299be6f0 Type: (ffffda8527087650) Event
ObjectHeader: ffffda85299be6c0 (new version)
HandleCount: 1 PointerCount: 1
0144: free handle, Entry address ffffca8985cb1510, Next Entry ffffca8985cb1520
0148: free handle, Entry address ffffca8985cb1520, Next Entry ffffca8985cb1530
After execution of function new record appears:
!handle 0 f 0xffffe382ff528480 – ServerExample.exe process object
0144: Object: ffffda8529a9bcf0 GrantedAccess: 0016019f (Audit) Entry: ffffca8985cb1510
Object: ffffda8529a9bcf0 Type: (ffffda852718cb00) File
ObjectHeader: ffffda8529a9bcc0 (new version)
HandleCount: 1 PointerCount: 2
Directory Object: 00000000 Name: \Endpoint {Afd}
0148: free handle, Entry address ffffca8985cb1520, Next Entry ffffca8985cb1530
The instruction mov rax, [rpb-58h] puts in rax value of a the handle.
We return to usermode. Further there is mswsock!SockGetInformation call (from which there is ntdll!NtDeviceIoControlFile was called to which as handle of the file it is transferred earlier returned 0x144).
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 (\Device\Afd\Endpoint). On the client there will be \Device\Afd
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, F6F2E0
_In_ ULONG IoControlCode, 1207B – AfdDispatchImmediateIrp ()
_In_ PVOID InputBuffer, F6F2FC
_In_ ULONG InputBufferLength,10
_Out_ PVOID OutputBuffer, F6F2FC
_In_ ULONG OutputBufferLength 10
);
Input buffer:
WINDBG>dc F6F2FC
00f6f2fc 00000007 00f6f38c c0140000 00f6f348 ............H...
IOCTL code - 1207Bh (afd!AfdDispatchImmediateIrp), but it will not be executed since the FastIO mechanism is involved (details further on the example of Send), afd!AfdFastIoDeviceControl will be executed, at the same time an IRP package is not formed, the usermode-buffer and its length are transferred as the 3rd and 4th parameter of this function
WINDBG>r
rcx – \Endpoint {Afd} object
r8=0000000000f6f2fc
r9=0000000000000010
After function execution (the result registers in the same buffer):
WINDBG>dc F6F2FC
00f6f2fc 00000007 00000000 00010000 00000000 ................
Further ntdll!NtDeviceIoControlFile function is caused repeatedly, in Input buffer data which were returned after the previous execution. But after repetition the result did not change. If function returned 103h, then mswsock! SockWaitForSingleObject would be caused, then ws2_32! WahInsertHandleContext, further an exit from mswsock!SockSocket also we come back to mswsock!WSPSocket. Entrance to mswsock!SockSetHandleContext. Further wshhyperv! WSHGetSocketInformation is caused, then nt!NtDeviceIoControlFile with IOCTL 12047h (AfdDispatchImmediateIrp)
As the Input-buffer it is transferred (0xD4 – size of buffer):
WINDBG>dc 006FED58 006FED58+D4
00f6f358 00000000 00000022 00000001 00000001 ...."...........
00f6f368 00000024 00000024 00000000 00000000 $...$...........
00f6f378 00000000 00010000 00010000 00001000 ................
00f6f388 00000000 000003e9 00020026 00000008 ........&.......
00f6f398 00000000 00000000 00000000 00000000 ................
00f6f3a8 00000000 00000000 00000000 00000000 ................
00f6f3b8 00000000 00000000 1234191b 4ca74bf7 ..........4..K.L
00f6f3c8 d7dfe086 45542bc3 00000004 656b6361 .....+TE....acke
00f6f3d8 00000000 00000000 00000000 00000000 ................
00f6f3e8 00000000 00000000 00000000 00000000 ................
00f6f3f8 00000000 012247f0 00000000 00000000 .....G".........
00f6f408 00000000 00000000 00000000 00000000 ................
00f6f418 00000000 00000000 00000000 d2ffd7d3 ................
00f6f428 00000022
In output buffer (changes after execution mswsock!SockSetHandleContext did not happen)
WINDBG>dc 00F6F400 00F6F400+24
00f6f400 00000000 00000000 00000000 00000000 ................
00f6f410 00000000 00000000 00000000 00000000 ................
00f6f420 00000000 d2ffd7d3 ........
From there is ws2_32!WPUModifyIFSHandle call (ws2_32! WahInsertHandleContext, is caused, then goes work with the ws2_32!SockPrimes array). An exit from mswsock!SockSetHandleContext. We return to our application.
- Bind
The socket function finished successfully, bind is executed further
iResult = bind(ListenSocket, hints.ai_addr, (int)hints.ai_addrlen);
Further I will try not to get information for usermode as detailed as for socket, and I will note only those moments which are specific to Hyper-V sockets. Mswsock!WSPbind is caused, further mswsock!WahReferenceContextByHandle (HANDLE socket, PVOID SockContextTable). Comparison of the address of the procedure, contained in ebx, with the address of the beginning mswsock_Tcpip4_WSHGetSockaddrType, or mswsock_Tcpip6_WSHGetSockaddrType is carried out. If the address coincides, there is a call of the corresponding procedure, if is not - then call ebx is carried out (in our case wshhyperv! WSHGetSockaddrType)
The type of a socket is get on the basis of GUID set during execution of bind (we set HV_GUID_ZERO)
On the server in esi:
WINDBG>dc @esi
00f6f990 00000022 00000000 00000000 00000000 "............... – HV_GUID_ZERO
00f6f9a0 00000000 b1d00d3e 4570fe10 487662ad ....>.....pE.bvH
00f6f9b0 1b7a9d77
On the client:
WINDBG>dc esi – in esi virtual machine ID GUID -6a964317-1d87-4a74-abf9-46a69b048900 (GUID which was returned on the server when performing a cmdlet of Get-VM | select ID, Name)
00affb2c 00000022 6a964317 4a741d87 a646f9ab "....C.j..tJ..F.
00affb3c 0089049b b1d00d3e 4570fe10 487662ad ....>.....pE.bvH
00affb4c 1b7a9d77 6a964317 4a741d87 a646f9ab w.z..C.j..tJ..F. ????????????
00affb5c 0089049b b1d00d3e 4570fe10 487662ad ....>.....pE.bvH
00affb6c 1b7a9d77 00d90000 40000062 02020202 w.z.....b..@....
00affb7c 536e6957 206b636f 00302e32 00000000 WinSock 2.0.....
Comparison of GUID is made at first with HV_GUID_LOOPBACK, then with HV_GUID_BROADCAST and then HV_GUID_ZERO. Then 0 returned. Before it there was a check of the second argument on excess of the 24 and first dword of first argument on equality with 22. Otherwise error was returned (271E and 273F respectively).
Further there are various checks, then ntdll!NtDeviceIoControlFile is caused.
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 (\Device\Afd\Endpoint). On the client - \Device\Afd
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, F6F728
_In_ ULONG IoControlCode, 12003
_In_ PVOID InputBuffer, 1223088
_In_ ULONG InputBufferLength,28
_Out_ PVOID OutputBuffer, 1223088
_In_ ULONG OutputBufferLength 24
);
For descriptive reasons we will check this using handle.exe from Sysinternals Suite:
handle.exe -a -p ServerExample.exe
140: Event
144: File (---) \Device\Afd\Endpoint (on client - \Device\Afd)
WINDBG>dc 1223088 1223088+28
01223088 00000000 00000022 00000000 00000000 ...."...........
01223098 00000000 00000000 b1d00d3e 4570fe10 ........>.....pE – GUID of our service
012230a8 487662ad 1b7a9d77 abababab .bvHw.z.....
The afd.sys driver registers function for handle IOCTL codes
0: kd> !drvobj afd 2
Driver object (ffffa40eb952c0d0) is for:
\Driver\AFD
Dispatch routines:
[0e] IRP_MJ_DEVICE_CONTROL fffff8008ab7d460 afd!AfdDispatchDeviceControl
Fast I/O routines:
FastIoRead fffff80aa54fbed0 afd!AfdFastIoRead
FastIoWrite fffff80aa54fbfd0 afd!AfdFastIoWrite
FastIoUnlockAll fffff80aa55018f0 afd!AfdSanFastUnlockAll
FastIoDeviceControl fffff80aa54f1ab0 afd!AfdFastIoDeviceControl
Let's consider what occurs in a kernel at this stage. IOCTL code 12003 of the table AfdIoctlTable in the table AfdIrpCallDispatch corresponds to afd!AfdBind which is caused from afd!AfdDispatchDeviceControl.
Pointer to IRP is transferred as first parameter of Afd!AfdBind functions (in rcx)
WINDBG>!irp @rcx 1
Irp is active with 4 stacks 4 is current (= 0xffffda8527506d18)
UserIosb = 00e6e920
UserEvent = ffffda85299be6f0
UserBuffer = 01223088
cmd flg cl Device File Completion-Context
>[IRP_MJ_DEVICE_CONTROL(e), N/A(0)]
5 0 ffffda8527de29d0 ffffda8529a9bcf0 00000000-00000000
\Driver\AFD
Args: 00000024 00000028 00012003 01223088
WINDBG>!object @r8 – 3-й параметр – Driver Object
Object: ffffda8527de19c0 Type: (ffffda852718cf20) Driver
ObjectHeader: ffffda8527de1990 (new version)
HandleCount: 0 PointerCount: 4
Directory Object: ffffca897e8f7060 Name: AFD
WINDBG>dc @rdx – 2-й параметр
ffffda85`27506d18 0005000e 00000000 00000024 00000000 ........$.......
ffffda85`27506d28 00000028 00000000 00012003 00000000 (........ ...... - iOCTL
ffffda85`27506d38 01223088 00000000 27de29d0 ffffda85 .0"......).'.... UserBuffer , DeviceObject (Afd)
ffffda85`27506d48 29a9bcf0 ffffda85 00000000 00000000 ...)............ – FileObject (\Endpoint {Afd})
At first there is a check of the sizes of entering and proceeding buffers (24 and 28 bytes corresponding) and other parameters. Allocation of a pool of 0x24 in size, and moving to it of the transferred parameters is carried out from usermode.
WINDBG>dc @rdx – memmove parameter
00000000`0122308c 00000022 00000000 00000000 00000000 "...............
00000000`0122309c 00000000 b1d00d3e 4570fe10 487662ad ....>.....pE.bvH
00000000`012230ac 1b7a9d77
On the client side the buffer will look so:
WINDBG>dc @rdx
00000000`013d38fc 00000022 6a964317 4a741d87 a646f9ab "....C.j..tJ..F.
00000000`013d390c 0089049b b1d00d3e 4570fe10 487662ad ....>.....pE.bvH
00000000`013d391c 1b7a9d77
Further there is nt!IoAllocateMdl call (Length eq 0x24)
WINDBG>dt nt!_MDL @rax – result of execution of IoAllocateMdl
+0x000 Next : (null)
+0x008 Size : 0n56
+0x00a MdlFlags : 0n8
+0x00c AllocationProcessorNumber : 0
+0x00e Reserved : 0xffff
+0x010 Process : 0xffffda85`293c9840 _EPROCESS
+0x018 MappedSystemVa : 0xffff9080`27c4903c Void
+0x020 StartVa : 0x00000000`01223000 Void
+0x028 ByteCount : 0x24
+0x02c ByteOffset : 0x88
Address of UserBuffer from the DeviceIoControl parameters loaded in rcx, then loading of pages from the pagefile (if pages was swapped) and their blocking in memory by means of nt!MmProbeAndLockPages. For the allocated pool with size 0x24 caching type cached, options HighPagePriority and MdlMappingNoExecute was set by nt!MmMapLockedPagesSpecifyCache.
afd!AfdTLBindSecurity-> afd!AfdTLBind-> afd!AfdTLIoControl (we leave from afd!AfdTLIoControl to afd!AfdTLBindComplete after afd!AfdTLBindComplete2, and only then we return directly to an exit from afd! AfdTLIoControl)
afd!AfdTLBindSecurity:
The 1st parameter - the same IRP
The 2nd parameter - _AFD_CONNECTION object
Further afd!AfdTlBind (same parameters).
In afd!AfdTLBind
hvsocket!VmbusTlProviderEndpointDispatch was loaded into rcx before AfdTlBindComplete was load into rax:
WINDBG>dps @rcx L50– tableof hvsocket.sys handlers
fffff803`db95c048 fffff803`db952460 hvsocket!VmbusTlCommonProviderCloseEndpoint
fffff803`db95c050 fffff803`db963210 hvsocket!VmbusTlEndpointIoControlEndpoint
fffff803`db95c058 fffff803`db95a8c0 hvsocket!TlDefaultRequestQueryDispatchEndpoint
fffff803`db95c060 fffff803`db952460 hvsocket!VmbusTlCommonProviderCloseEndpoint
fffff803`db95c068 fffff803`db963be0 hvsocket!VmbusTlListenerIoControlEndpoint
fffff803`db95c070 fffff803`db95a8c0 hvsocket!TlDefaultRequestQueryDispatchEndpoint
fffff803`db95c078 fffff803`db95a8d0 hvsocket!TlDefaultRequestResume
fffff803`db95c080 fffff803`db954300 hvsocket!VmbusTlConnectionCloseEndpoint
fffff803`db95c088 fffff803`db965fc0 hvsocket!VmbusTlConnectionIoControlEndpoint
fffff803`db95c090 fffff803`db95a8c0 hvsocket!TlDefaultRequestQueryDispatchEndpoint
fffff803`db95c098 fffff803`db954c60 hvsocket!VmbusTlConnectionSend
fffff803`db95c0a0 fffff803`db954ea0 hvsocket!VmbusTlConnectionReceive
fffff803`db95c0a8 fffff803`db9585a0 hvsocket!VmbusTlConnectionDisconnect
fffff803`db95c0b0 00000001`00000000
fffff803`db95c0b8 00000000`00000000
fffff803`db95c0c0 fffff803`db969de0 hvsocket!VmbusTlXPartAcceptConnection
fffff803`db95c0c8 00000000`00000000
fffff803`db95c0d0 fffff803`db959830 hvsocket!VmbusTlXPartProcessIoRequest
fffff803`db95c0d8 00000000`00000000
fffff803`db95c0e0 fffff803`db959890 hvsocket!VmbusTlXPartIoRequestCompleted
fffff803`db95c0e8 fffff803`db959950 hvsocket!VmbusTlXPartReleaseReceiveIndications
fffff803`db95c0f0 fffff803`db969ee0 hvsocket!VmbusTlXPartDisconnect
fffff803`db95c0f8 fffff803`db959820 hvsocket!VmbusTlXPartSendConsumptionNotice
fffff803`db95c100 fffff803`db959940 hvsocket!VmbusTlXPartIsIncomingEmpty
fffff803`db95c108 00000000`00000000
fffff803`db95c110 00000001`00000000
fffff803`db95c118 fffff803`db96a430 hvsocket!VmbusTlLoopbackSetupConnection
fffff803`db95c120 fffff803`db96a6b0 hvsocket!VmbusTlLoopbackAcceptConnection
fffff803`db95c128 00000000`00000000
fffff803`db95c130 fffff803`db959ed0 hvsocket!VmbusTlLoopbackProcessIoRequest
fffff803`db95c138 fffff803`db96a850 hvsocket!VmbusTlLoopbackPostprocessIoRequest
fffff803`db95c140 fffff803`db959f10 hvsocket!VmbusTlLoopbackIoRequestCompleted
fffff803`db95c148 fffff803`db959f30 hvsocket!VmbusTlLoopbackReleaseReceiveIndications
fffff803`db95c150 fffff803`db95a0c0 hvsocket!VmbusTlLoopbackDisconnect
fffff803`db95c158 fffff803`db95a340 hvsocket!VmbusTlLoopbackNotifyReceiveConsumed
fffff803`db95c160 fffff803`db95a2f0 hvsocket!VmbusTlLoopbackIsIncomingEmpty
Then afd!AfdTLIoControl is caused (before a call rather big set of parameters is formed), the first parameter the pointer on the hvsocket!VmbusTlEndpointIoControlEndpoint function, which then is also carried out. Then hvsocket! VmbusTlHandleEndpointIoControl is caused. During execution we receive a recursion:
WINDBG>kc
Call Site
afd!AfdTLIoControl
afd!AfdTLBindComplete2
afd!AfdTLBindComplete
afd!AfdTLIoControl
afd!AfdTLBind
afd!AfdTLBindSecurity
afd!AfdBind
nt!IopSynchronousServiceTail
this time hvsocket!VmbusTlContainerGetVmId is caused from hvsocket!VmbusTlHandleEndpointIoControl. Next hvsocket! VmbusTlFindAndReferencePartitionByContainerId is caused. In the first parameter of function on +80h the coherent list:
WINDBG>!list @rcx+80h
ffffda85`27f6bb00 ffffda85`27f009e8 ffffda85`27f009e8
ffffda85`27f6bb10 00000000`00000001 ffffda85`27f6bb18
ffffda85`27f6bb20 ffffda85`27f6bb18 ffffda85`27f00910
ffffda85`27f6bb30 00000000`00000000 0000257a`d80d0eb8
ffffda85`27f6bb40 ffffda85`27f2de20 00000000`00000000
ffffda85`27f6bb50 00000000`00000000 00000000`00000000
ffffda85`27f6bb60 00000000`00000000 00000000`00000000
ffffda85`27f6bb70 00000000`00000000 00000000`00000000
ffffda85`27f009e8 ffffda85`27f6bb00 ffffda85`27f6bb00
ffffda85`27f009f8 4f790d35`90db8b89 cdb7c80a`ea49e98c – HV_CHILD_GUID
ffffda85`27f00a08 00000000`00000000 ffffda85`27f00a10
ffffda85`27f00a18 00000000`00000000 00000000`00000000
ffffda85`27f00a28 00000000`00000000 00000000`00000000
ffffda85`27f00a38 00000000`00000000 00000000`00000000
ffffda85`27f00a48 00000000`00000000 00000000`00000000
ffffda85`27f00a58 fffff803`db9516c0 fffff803`db951db0
If to turn on the guest virtual machine, then one more element will appear
ffffb08b`93e490e8 ffffb08b`923a9700 ffffb08b`924390e8
ffffb08b`93e490f8 4a741d87`6a964317 0089049b`a646f9ab – GUID of the switched-on VM in which a client part of our application will be started.
ffffb08b`93e49108 00000000`00000001 ffffb08b`93e49110
ffffb08b`93e49118 00000000`00000000 00000000`00000000
ffffb08b`93e49128 00000000`00000000 00000000`00000000
ffffb08b`93e49138 00000000`00000000 00000000`00000000
ffffb08b`93e49148 00000000`00000000 00000000`00000000
ffffb08b`93e49158 fffff805`2dfc16c0 fffff805`2dfc1db0
This function make search in this coherent list and returns structure which on offset +E8h contains the pointer on HV_GUID_CHILDREN, if the 2nd parameter transferred to the hvsocket! VmbusTlContainerGetVmId function coincides with the value placed on offset +18Ch this structure. Further afd!AfdTLBindGetAddrComplete.
Stack (we still in afd!AfdTLBindComplete2)
WINDBG>kc
Call Site
afd!AfdTLBindGetAddrComplete
afd!AfdTLIoControl
afd!AfdTLBindComplete2
afd!AfdTLBindComplete
afd!AfdTLIoControl
afd!AfdTLBind
afd!AfdTLBindSecurity
afd!AfdBind
in which tdi!TdiCopyBufferToMdl is caused
The copied buffer looks as follows:
WINDBG>dc @rcx
ffffda85`274bf180 00000022 90db8b89 4f790d35 ea49e98c ".......5.yO..I. - HV_GUID_CHILDREN
ffffda85`274bf190 cdb7c80a b1d00d3e 4570fe10 487662ad ....>.....pE.bvH
ffffda85`274bf1a0 1b7a9d77
After copying nt!MmUnlockPages and nt!ioFreeMdl are caused consistently, and then nt!IofCompleteRequest. Execution of afd! AfdTlBindSecurity was finished. In EAX again writes 103h. As we saw earlier, this value will be returned by the NtDeviceIoControlFile function. In this case mswsock!SockWaitForSingleObject is caused in addition.
WINDBG>dc 1223088 1223088+24 – UserBuffer contents after execution of NtDeviceIoControlFile
01223088 00000022 90db8b89 4f790d35 ea49e98c ".......5.yO..I.
01223098 cdb7c80a b1d00d3e 4570fe10 487662ad ....>.....pE.bvH
012230a8 1b7a9d77
In usermode, as well as when socket call, mswsock!SockSetHandleContext is caused, then NtDeviceIoControlFile is caused again:
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 (\Device\Afd\Endpoint). On client \Device\Afd
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, F6F600
_In_ ULONG IoControlCode, 12047 (afd!AfdDispatchImmediateIrp)
_In_ PVOID InputBuffer, F6F608
_In_ ULONG InputBufferLength,D4
_Out_ PVOID OutputBuffer, 0
_In_ ULONG OutputBufferLength 0
);
Inputbuffer:
WINDBG>dd 00F6F608 00F6F608+d4
00f6f608 00000001 00000022 00000001 00000001 – parameters from the socket provider section in registry (partition Hyper-V RAW)
00f6f618 00000024 00000024 00000000 00000000
00f6f628 00000000 00010000 00010000 00001000
00f6f638 00000000 000003e9 00020026 00000008
00f6f648 00000000 00000000 00000000 00000000
00f6f658 00000000 00000000 00000000 00000000
00f6f668 00000000 00000000 1234191b 4ca74bf7
00f6f678 d7dfe086 45542bc3 00000004 00000610
00f6f688 00000022 90db8b89 4f790d35 ea49e98c – the buffer created by afd!AfdBind
00f6f698 cdb7c80a b1d00d3e 4570fe10 487662ad
00f6f6a8 1b7a9d77 fffffffe 00000000 00000000
00f6f6b8 00000000 00000000 00000000 00000000
00f6f6c8 00000000 00000000 00000000 00000030
00f6f6d8 00000022 00000000
Data processing will pass through FastIo again. In this case search of the function, which handle IOCTL, is made in the table AfdImmediateCallDispatch. In our case Afd!AfdSetContext will become a case the handler. At the beginning, delivery of all APC to the current thread is turned off by means of nt!KeEnterGuardedRegion, contents of UserBuffer is copied to one of fields of structure _AFD_CONNECTION. Function finishes with nt!KeLeaveGuardedRegion call. We come back to the application.
- Listen
Далее серверное приложение вызывает Listen. UserMode рассматривать особо не будет, сразу обратимся к функции afd!AfdStartListen. Всё так же вызов идёт через ntdll!NtDeviceIoControlFile.
Further the server application causes Listen. UserMode will especially not consider, at once we will address to the function afd!AfdStartListen. The call goes through ntdll!sNtDeviceIoControlFile.
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 (\Device\Afd\Endpoint).
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, F6F798
_In_ ULONG IoControlCode, 1200B (afd!AfdStartListen)
_In_ PVOID InputBuffer, F6F7A0
_In_ ULONG InputBufferLength,C
_Out_ PVOID OutputBuffer, 0
_In_ ULONG OutputBufferLength 0
);
WINDBG>dc F6F7A0
00f6f7a0 00000100 7fffffff 00000000
In the kernel InputBuffer it is compared to AfdUserProbeAddress, then SLIST_ENTRY is formed. In _AFD_CONNECTION structure in the field "type" writes 0xAFD4. Next afd!AfdRefTLBaseEndpoint is caused, then afd!AfdTLListen, from where hvsocket! VmbusTlProviderListen is caused, then hvsocket!VmbusTlCreateEndpoint is carried out, where nt!PsChargeProcessPoolQuota is caused. POOL_TYPE set to 200h (NonPagedPoolNx), process is current, then hvsocket!VmbusTlCreateObjectFromLookasideList is caused, next - nt! ExAlocatePoolWithTag with buffer the size 130h, the Vnpi tag, then hvsocket!VmbusTlInitializeObject, then work queue initialization is carried out (Netio!NetioInitializeWorkQueue) with the hvsocket!VmbusTlEndpointActionWorkQueueRoutine parameter, then hvsocket! VmbusTlAssociateListenerToPartition from where hvsocket!VmbusTlFindAndReferencePartitionByContainerId is caused.
WINDBG>kc
Call Site
hvsocket!VmbusTlAssociateListenerToPartition
hvsocket!VmbusTlProviderListen
afd!AfdTLListen
afd!AfdStartListen
nt!IopSynchronousServiceTail
nt!IopXxxControlFile
nt!NtDeviceIoControlFile
Then hvsocket! VmbusTlResolvePartitionId, then hvsocket! VmbusTlGetPartitionListenerEndpoint, from where - hvsocket! VmbusTlFindOrCreateService, function comes to the end (in rax - 0), then there is a check to what partition we work - we receive HV_CHILDREN_GUID, again we cause hvsocket!VmbusTlFindOrCreateService, and once again hvsocket!VmbusTlFindOrCreateService, comes back to r14 0xC0000225.
Further hvsocket! VmbusTlFindAndReferencePartition, then hvsocket! VmbusTlEndpointIsPrivileged is caused, Then next sequence:
call cs:__imp_SeCaptureSubjectContextEx
lea rcx, [rbp+SubjectSecurityContext]
call cs:__imp_SeLockSubjectContext
call cs:__imp_IoGetFileObjectGenericMapping
Function returns _GENERIC_MAPPING struct:
WINDBG>dt _GENERIC_MAPPING @rax
combase!_GENERIC_MAPPING
+0x000 GenericRead : 0x120089
+0x004 GenericWrite : 0x120116
+0x008 GenericExecute : 0x1200a0
+0x00c GenericAll : 0x1f01ff
then nt!SeAccessCheck, and nt!ObDereferenceSecurityDescriptor. Result of the hvsocket!VmbusTlEndpointIsPrivileged function at one of traces:
WINDBG>!error @rax
Error code: (NTSTATUS) 0xc0000022 (3221225506) - {Access Denied} A process has requested access to an object, but has not been granted those access rights. (However, the last byte of result at the exit is nullified).
In hvsocket there is a VmbusTlEndpointSecurityDescriptor variable
WINDBG>!sd hvsocket!VmbusTlEndpointSecurityDescriptor
->Revision: 0x1
->Sbz1 : 0x0
->Control : 0x4
SE_DACL_PRESENT
->Owner : is NULL
->Group : is NULL
->Dacl :
->Dacl : ->AclRevision: 0x2
->Dacl : ->Sbz1 : 0x0
->Dacl : ->AclSize : 0x34
->Dacl : ->AceCount : 0x2
->Dacl : ->Sbz2 : 0x0
->Dacl : ->Ace[0]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
->Dacl : ->Ace[0]: ->AceFlags: 0x3
->Dacl : ->Ace[0]: OBJECT_INHERIT_ACE
->Dacl : ->Ace[0]: CONTAINER_INHERIT_ACE
->Dacl : ->Ace[0]: ->AceSize: 0x14
->Dacl : ->Ace[0]: ->Mask : 0x000f003f
->Dacl : ->Ace[0]: ->SID: S-1-5-18
->Dacl : ->Ace[1]: ->AceType: ACCESS_ALLOWED_ACE_TYPE
->Dacl : ->Ace[1]: ->AceFlags: 0x3
->Dacl : ->Ace[1]: OBJECT_INHERIT_ACE
->Dacl : ->Ace[1]: CONTAINER_INHERIT_ACE
->Dacl : ->Ace[1]: ->AceSize: 0x18
->Dacl : ->Ace[1]: ->Mask : 0x000f003f
->Dacl : ->Ace[1]: ->SID: S-1-5-95-0
->Sacl : is NULL
hvsocket!VmbusTlEndpointIsPrivileged returns c0000000, again hvsocket!VmbusTlFindOrCreateService is called.
WINDBG>kc
Call Site
hvsocket!VmbusTlFindOrCreateService
hvsocket!VmbusTlAssociateListenerToPartition
hvsocket!VmbusTlProviderListen
afd!AfdTLListen
afd!AfdStartListen
nt!IopSynchronousServiceTail
nt!IopXxxControlFile
nt!NtDeviceIoControlFile
And only now hvsocket!VmbusTlCreateService is called. It consists of two calls: hvsocket!VmbusTlCreateObject and hvsocket! VmbusTlInitializeObjectTable. The AVL table is created
VOID RtlInitializeGenericTableAvl(
_Out_ PRTL_AVL_TABLE Table,
_In_ PRTL_AVL_COMPARE_ROUTINE CompareRoutine,- hvsocket!VmbusTlCompareGuids
_In_ PRTL_AVL_ALLOCATE_ROUTINE AllocateRoutine,- hvsocket!VmbusTlAllocateForAvlTable
_In_ PRTL_AVL_FREE_ROUTINE FreeRoutine, - hvsocket!VmbusTlFreeForAvlTable
_In_opt_ PVOID TableContext
);
After initialization of the table we receive:
WINDBG>dt nt!_RTL_AVL_TABLE ffffb904c25c9f60
+0x000 BalancedRoot : _RTL_BALANCED_LINKS
+0x020 OrderedPointer : (null)
+0x028 WhichOrderedElement : 0
+0x02c NumberGenericTableElements : 0
+0x030 DepthOfTree : 0
+0x038 RestartKey : (null)
+0x040 DeleteCount : 0
+0x048 CompareRoutine : 0xfffff80a`6d6d16c0 _RTL_GENERIC_COMPARE_RESULTS hvsocket!VmbusTlCompareGuids+0
+0x050 AllocateRoutine : 0xfffff80a`6d6d1db0 void* hvsocket!VmbusTlAllocateForAvlTable+0
+0x058 FreeRoutine : 0xfffff80a`6d6d1dd0 void hvsocket!VmbusTlFreeForAvlTable+0
+0x060 TableContext : 0x00000000`694c6353 Void
hvsocket!VmbusTlFindOrCreateService call is complete, then hvsocket!VmbusTlAssociateListenerToService is caused in which hvsocket!VmbusTlInsertObjectToTable is caused, after that the table looks so (one element was added):
WINDBG>dt nt!_RTL_AVL_TABLE ffffb904c25c9f60
+0x000 BalancedRoot : _RTL_BALANCED_LINKS
+0x020 OrderedPointer : (null)
+0x028 WhichOrderedElement : 0
+0x02c NumberGenericTableElements : 1
+0x030 DepthOfTree : 1
+0x038 RestartKey : (null)
+0x040 DeleteCount : 0
hvsocket!VmbusTlAssociateListenerToPartition finishes, then afd!AfdTLListenComplete, hvsocket! VmbusTlCommonProviderCloseEndpoint and hvsocket! VmbusTlQueueEndpointAction are caused from which netio!NetioInsertWorkQueue is caused (handler proc - hvsocket!VmbusTlEndpointActionWorkQueueRoutine), we return to afd!AfdTlListenComplete where there is afd!AfdDerefTLBaseEndpoint call and then nt!IofCompleteRequest.
We return to hvsocket!VmbusTlProviderListen, next hvsocket!VmbusTlListenerProcessPendingIncomingConnection is caused further from where hvsocket!VmbusTlGetPendingConnection is caused, then hvsocket_VmbusTlpGetPendingConnection, returned value - 103h. Exit from afd.sys, we return to usermode.
Further mswsock!SockSetHandleContext is caused.
Userbuffer contents for ntdll!NtDeviceIoControlFile
WINDBG>dd 00F6F668 00F6F668+d4
00f6f668 00000000 00000000 00000001 00000022
00f6f678 00000001 00000001 00000024 00000024
00f6f688 00000000 00000000 00000000 00010000
00f6f698 00010000 00001001 00000000 000003e9
00f6f6a8 00020026 00000008 00000000 00000000
00f6f6b8 00000000 00000000 00000000 00000000
00f6f6c8 00000000 00000000 00000000 00000000
00f6f6d8 1234191b 4ca74bf7 d7dfe086 45542bc3
00f6f6e8 00000004 7231873c 00000022 90db8b89
00f6f6f8 4f790d35 ea49e98c cdb7c80a b1d00d3e
00f6f708 4570fe10 487662ad 1b7a9d77 01223088
00f6f718 00000000 00000000 00000000 00000000
00f6f728 00000000 00000000 00000000 00000000
00f6f738 00000000 00000144
We returned to our application, accept call is executed
- Accept
ws2_32!WSAAccept is caused, then mswsock!WSPAccept, then ntdll!NtDeviceIoControlFile:
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 (\Device\Afd\Endpoint).
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, F6F5D0
_In_ ULONG IoControlCode, 1200C (afd!AfdWaitForListen). Interesting, that IOCTL 12090 is handled by same function
_In_ PVOID InputBuffer, 0
_In_ ULONG InputBufferLength,0
_Out_ PVOID OutputBuffer, F6F730
_In_ ULONG OutputBufferLength 28
);
WINDBG>k
Child-SP RetAddr Call Site
afd!AfdWaitForListen
nt!IopSynchronousServiceTail+0x1a0
nt!IopXxxControlFile+0x674
nt!NtDeviceIoControlFile+0x56
nt!KiSystemServiceCopyEnd+0x13
The Input buffer, apparently, is absent, only output. Management passes to afd!AfdWaitForListen. afd!AfdGetUnacceptedConnection is caused. The first parameter - _AFD_CONNECTION, is carried out element comparison on offset +50h with the address of the structure, 0 is return. Further in IRP-> CancelRoutine writes afd!AfdCancelWaitForListen
Calls of other functions do not occur, execution in afd.sys comes to the end, return directly to the appendix does not come from NtDeviceIoControlFile. Let's look where the control is transferred. All calls of nt come to the end with the instruction of sysret which transfers control in usermode to the address specified in rcx.
WINDBG>u @rcx
ntdll!NtReadFile+0x14:
00007ff9`41586194 c3 ret
00007ff9`41586195 cd2e int 2Eh
00007ff9`41586197 c3 ret
00007ff9`41586198 0f1f840000000000 nop dword ptr [rax+rax]
ntdll!NtDeviceIoControlFile:
00007ff9`415861a0 4c8bd1 mov r10,rcx
00007ff9`415861a3 b807000000 mov eax,7
00007ff9`415861a8 f604250803fe7f01 test byte ptr [SharedUserData+0x308 (00000000`7ffe0308)],1
00007ff9`415861b0 7503 jne ntdll!NtDeviceIoControlFile+0x15 (00007ff9`415861b5)
There is only one instruction, therefore we will consider a stack:
WINDBG>dqs @r8
00000000`0b8fbb98 00007ff9`3dfc3d34 – we get to SysWoW.dll. The stack at the same time looks so:
WINDBG>dqs @r8 (r8 will be placed in rsp just before execution of swapgs, sysret)
00000000`0b8fbb98 00007ff9`3dfc3d34 KERNELBASE!ReadFile+0x74
00000000`0b8fbba0 00000000`00000000
Stack content is changed during execution of nt!KeWaitForSingleObject-> nt! KiCommitThreadWait-> nt! KiSwapThread caused from nt! IopSynchronousServiceTail
WINDBG>kcn
# Call Site
00 nt!IopSynchronousServiceTail
01 nt!NtReadFile
02 nt!KiSystemServiceCopyEnd
03 ntdll!NtReadFile
04 KERNELBASE!ReadFile
05 SHCORE!CFileStream::Read
06 windows_storage!CShellLink::_LoadFromStream
07 windows_storage!CShellLink::_LoadFromFile
08 windows_storage!CShellLink::Load
09 windows_storage!InitializeFileHandlerWithFile
0a windows_storage!CFileSysItemString::HandlerCreateInstance
0b windows_storage!CFSFolder::_BindHandler
0c windows_storage!CFSFolder::GetUIObjectOf
0d windows_storage!CShellItem::BindToHandler
0e SHELL32!CAppResolver::GetAppIDForShortcut
0f SHELL32!CAppResolver::GetAppIDForWindow
10 Explorer!CTaskBand::CResolveWindowTask::_ResolveWindowWorker
11 Explorer!CTaskBand::CResolveWindowTask::_ResolveWindow
12 Explorer!CTaskBand::CResolveWindowTask::InternalResumeRT
13 Explorer!CRunnableTask::Run
14 windows_storage!CShellTask::TT_Run
15 windows_storage!CShellTaskThread::ThreadProc
16 windows_storage!CShellTaskThread::s_ThreadProc
17 SHCORE!ExecuteWorkItemThreadProc
18 ntdll!RtlpTpWorkCallback
19 ntdll!TppWorkerThread
1a KERNEL32!BaseThreadInitThunk
1b ntdll!RtlUserThreadStart
Return happens to another thread. Probably, these are features of WoW64.
Execution goes without surprises for 64-bit process, and we return to the end of the nt!NtDeviceIoControlFile function after execution of sysret from which there is a return to mswsock!WSPAccept, and program execution stops inside mswsock! SockWaitForSingleObject function.
Function execution will continue when on the client party will bw connect call. Accept execution continues. Mswsock!SockSocket is called (early was called from socket). Next mswsock!SockGetTdiName (with Hyper-V RAW Guid). There is pointer on structure with size not below 0xCC in esi. Variable mswsock!SockTLNPIListenerCount eq 1, GetCurrentProcess is not called, \\Device\\Afd\\Endpoint is initialized again, and next NtCreateFile again:
NTSTATUS NtCreateFile(
_Out_ PHANDLE FileHandle,- 00F6F4A4
_In_ ACCESS_MASK DesiredAccess, C0140000
_In_ POBJECT_ATTRIBUTES ObjectAttributes, 00F6F460
_Out_ PIO_STATUS_BLOCK IoStatusBlock,- 00F6F480
_In_opt_ PLARGE_INTEGER AllocationSize, - 00000000
_In_ ULONG FileAttributes,- 00000000
_In_ ULONG ShareAccess,- 3
_In_ ULONG CreateDisposition,-3
_In_ ULONG CreateOptions,0x20
_In_ PVOID EaBuffer, 00F6F4DC
_In_ ULONG EaLength , 39
);
WINDBG>dtx _OBJECT_ATTRIBUTES 00F6F460
(*((_OBJECT_ATTRIBUTES *)0xf6f460)) [Type: _OBJECT_ATTRIBUTES]
[+0x000] Length : 0x18 [Type: unsigned long]
[+0x004] RootDirectory : 0x0 [Type: void *]
[+0x008] ObjectName : 0xf6f454 : "\Device\Afd\Endpoint" [Type: _UNICODE_STRING *]
[+0x00c] Attributes : 0x42 [Type: unsigned long]
[+0x010] SecurityDescriptor : 0x0 [Type: void *]
[+0x014] SecurityQualityOfService : 0x0 [Type: void *]
In windows kernel: afd!AfdCreateFile-> afd!AfdCheckTDIFilter is not called, afd!AfdAllocateEndpoint executes immediately. After afd!AfdTlFindAndReferenceTransport execution in rax:
WINDBG>!mex.foreachitem @rax -c
ffffda8527f6b8c0
fffff803db337530
ffffda8527dfeeb0
ffffda8527dfedd0
ffffda8527dfeca0
Processed 5 items.
Nt!IoGetCurrentProcess, afd!AfdEndpointsFreeing compates with 0xAh, if eq or above, then afd!AfdReuseEndpoint и ExReleaseResourceAndLeaveCriticalRegion are executed.
Next we see early mentioned afd!AfdTLCreateEndpoint, next afd!AfdTLPendRequest (result – 103h), then afd! AfdCompleteTLEndpCreate, we return from afd!AfdCreate, next nt!SeClearLearningModeObjectInformation, next nt! SeSetLearningModeObjectInformation and nt! ObpCreateHandle.
Descriptor number is 0x148h. Now we have two sockets (formally – file objects)
0144: Object: ffffda8529a9bcf0 GrantedAccess: 0016019f (Audit) Entry: ffffca8985cb1510
Object: ffffda8529a9bcf0 Type: (ffffda852718cb00) File
ObjectHeader: ffffda8529a9bcc0 (new version)
HandleCount: 1 PointerCount: 32761
Directory Object: 00000000 Name: \Endpoint {Afd}
0148: Object: ffffda8529acc6d0 GrantedAccess: 0016019f (Inherit) Entry: ffffca8985cb1520
Object: ffffda8529acc6d0 Type: (ffffda852718cb00) File
ObjectHeader: ffffda8529acc6a0 (new version)
HandleCount: 1 PointerCount: 2
Directory Object: 00000000 Name: \Endpoint {Afd}
Next NtDeviceIoControlFile
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 File (---) \Device\Afd)
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, 00F6F5D0
_In_ ULONG IoControlCode, 00012010 (afd!AfdAccept)
_In_ PVOID InputBuffer, 00F6F5D8
_In_ ULONG InputBufferLength,0xС
_Out_ PVOID OutputBuffer, 0
_In_ ULONG OutputBufferLength 0
);
WINDBG>dd 00F6F5D8
00f6f5d8 00000000 00000001 00000148 00000000
Go to afd!AfdAccept:
IoIs32bitProcess, next ObReferenceObjectByHandle (Handle – 0x148), then в rax загружается ссылка на AfdDeviceObject (0FFFFDA8527DE29D0h)
WINDBG>!devobj 0FFFFDA8527DE29D0h
Device object (ffffda8527de29d0) is for:
Afd \Driver\AFD DriverObject ffffda8527de19c0
Current Irp 00000000 RefCount 85 Type 00000011 Flags 00000050
Dacl ffffcb8a7e8ccd11 DevExt 00000000 DevObjExt ffffda8527de2b20
ExtensionFlags (0x00000800) DOE_DEFAULT_SD_PRESENT
Characteristics (0x00020000) FILE_DEVICE_ALLOW_APPCONTAINER_TRAVERSAL
Device queue is not busy.
fild in _AFD_CONNECTION struct with offset +8h is compated with AfdDeviceObject. Next nt!KeAcquireInStackQueuedSpinLock and afd! AfdGetReturnedConnection. Result (in rax):
WINDBG>dc @rax
ffffda85`298e7ac0 0002afd8 00061000 28bf3e40 ffffda85 ........@>.(....
ffffda85`298e7ad0 28fa1180 ffffda85 db95c080 fffff803 ...(............
ffffda85`298e7ae0 27121080 ffffda85 17d0e6d1 00000019 ...'............
then afd!AfdAcceptCore (1-й parameter - IRP)
WINDBG>dd @rdx
ffffda85`29b80860 0002afd0 01000000 00000100 00000000
ffffda85`29b80870 297cf380 ffffda85 db95c048 fffff803
WINDBG>dd @r8
ffffda85`298e7ac0 0002afd8 00061000 28bf3e40 ffffda85
ffffda85`298e7ad0 28fa1180 ffffda85 db95c080 fffff803
Inside of function: nt!KeAcquireInStackQueuedSpinLockAtDpcLevel, then afd! AfdSetupAcceptEndpoint (nt!ExFreePoolWithTag is executed only) next nt!KeReleaseInStackQueuedSpinLockFromDpcLevel, return to afd!AfdAccept: nt!KeReleaseInStackQueuedSpinLock, then afd!AfdDerefTLBaseEndpoint и afd!AfdTLCloseEndpoint, and hvsocket! VmbusTlCommonProviderCloseEndpoint is called, next is hvsocket!VmbusTlQueueEndpointAction which initialise work queue hvsocket!VmbusTlEndpointActionWorkQueueRoutine using netio! NetioInsertWorkQueue. Next hvsocket! VmbusTlCommonProviderCloseEndpoint resturns 103h. Next afd!AfdTlDereferenceTransport is called (WskTdiTransport was checked – I have zero value during tracing). netio!NmrClientDetachProviderComplete may be called. Return to afd!AfdAccept, next nt!ObfDereferenceObject and nt!IofCompleteRequest. Go away from afd.sys
After some time hvsocket!VmbusTlEndpointActionWorkQueueRoutine will be executed. hvsocket!VmbusTlCommonEndpointCleanup is called from it, next afd!AfdTLCloseEndpointComplete (we can see DbgPrint call with 'Failed to close TLI endpoint! Status=%lx, AFD endp=%p' message. We can see it rarely In windows kernel. WPP is used usually). 2nd paramaters is not zero and therefore go to afd! AfdDereferenceEndpointInline, return to hvsocket.sys. Next hvsocket!VmbusTlEndpointDestructor, then nt!PsReturnPoolQuota и nt!ObfDereferenceObject (form ServerExample.exe process object). We can see wotk with VmbusProviderContext variable:
WINDBG>dps VmbusProviderContext
fffff803`db95e0b8 ffffda85`27f6ba80
fffff803`db95e0c0 00000000`00000000
fffff803`db95e0c8 00000000`00000000
fffff803`db95e0d0 00000000`00000000
fffff803`db95e0d8 00000000`00000000
fffff803`db95e0e0 00000000`00040001
fffff803`db95e0e8 00000000`00000000
fffff803`db95e0f0 00000000`00000000
fffff803`db95e0f8 00000000`00000000
fffff803`db95e100 ffffda85`27f2c0b0
fffff803`db95e108 ffffda85`27f2c0b0
fffff803`db95e110 00000000`00000000
fffff803`db95e118 00000000`00000000
fffff803`db95e120 00000000`00000000
fffff803`db95e128 fffff803`db95c260 hvsocket!WPP_ThisDir_CTLGUID_HvSocketTraceGuid
fffff803`db95e130 00000000`00000000
WINDBG>dps 0FFFFDA8527F6BA80
ffffda85`27f6ba80 00000000`00000006
ffffda85`27f6ba88 00000000`00000009
ffffda85`27f6ba90 00000000`00000001
ffffda85`27f6ba98 00000000`00000000
ffffda85`27f6baa0 00000000`00000000
ffffda85`27f6baa8 00000000`00060001
ffffda85`27f6bab0 ffffda85`27f6bab0
ffffda85`27f6bab8 ffffda85`27f6bab0
ffffda85`27f6bac0 00000000`00000000
ffffda85`27f6bac8 00000000`00000000
ffffda85`27f6bad0 fffff803`db951e50 hvsocket!HvSocketProviderDestructor
ffffda85`27f6bad8 00000000`00000001
ffffda85`27f6bae0 ffffda85`27f6b9a0
ffffda85`27f6bae8 ffffda85`27f6b8c0
ffffda85`27f6baf0 fffff803`db32d160 afd!AfdTlClientDispatch
ffffda85`27f6baf8 00000000`00000001
hvsocket!VmbusTlEndpointWorkQueueDestructor is queued using netio!NetioInsertWorkQueue, then go from hvsocket!VmbusTlEndpointDestructor to netio!NetioInsertWorkQueue and then go to windows kernel…
Next we returnt to mswsock!WSAAccept, mswsock!SockNotifyHelperDll is called next, then go to mswsock!SockCoreAccept.
wshhyperv!WSHGetSocketInformation is called 3 times next wshhyperv!WSHGetSocketInformation and then mswsock!SockUpdateWindowSizes.
Mswsock!SockGetTdiHandles is called, which goes to ntdll!NtDeviceIoControlFile execution:
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 148 File (---) \Device\Afd)
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, 00F6F424
_In_ ULONG IoControlCode, 00012037 (AfdDispatchImmediateIrp)
_In_ PVOID InputBuffer, 00F6F5D8
_In_ ULONG InputBufferLength,0xС
_Out_ PVOID OutputBuffer, 0
_In_ ULONG OutputBufferLength 0
);
- Connect
The server is ready to accept incoming connections so now we will consider connect call which is carried out from the client.
From ClientExample!Connect ws2_32 call! Prolog_v2 goes, then ws2_32!WahReferenceContextByHandle (as a descriptor it is transferred 144 - handle \Device\Afd). Further is caused mswsock!WSPConnect, from which mswsock! SockDoConnect, further mswsock! SOCK_SQM_INFO_CAPTURE __ NonCore_WSAConnect, then wshhyperv! WSHGetWildcardSockaddr (returns wshhyperv! HV_GUID_ZERO), then mswsock! WSPBind from where is caused wshhyperv!WSHGetSockaddrType
WINDBG>k – bind from connect
ChildEBP RetAddr
0036f478 7203873c ntdll!NtDeviceIoControlFile
0036f530 7204dab6 mswsock!WSPBind+0x1cc
0036f5c0 7204e3af mswsock!SockDoConnect+0x2c0
0036f5dc 75de4d76 mswsock!WSPConnect+0x1f
0036f62c 002611df WS2_32!connect+0x86
В input buffer до вызова NtDeviceIOControlFile (с IOCTL 12003) из mswsock!WSPBind
Value of input buffer to NtDeviceIOControlFile call (with IOCTL 12003) from mswsock!WSPBind
WINDBG>dc 007D4930 007D4930+28
007d4930 00000002 baad0022 00000000 00000000
007d4940 00000000 00000000 00000000 00000000
007d4950 00000000 00000000
Input buffer after call:
WINDBG>dc 007D4930 007D4930+24
007d4930 00000022 90db8b89 4f790d35 ea49e98c - HV_GUID_CHILDREN
007d4940 cdb7c80a 00000000 00000000 00000000
007d4950 00000000 00000000
Returns from mswsock!WSPBind
Next mswsock!SockDoConnectReal, from which ntdll!NtDeviceIoControlFile is caused
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 File (---) \Device\Afd)
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, 0036F4C0
_In_ ULONG IoControlCode, 00012007 (afd!AfdConnect)
_In_ PVOID InputBuffer, 0036F4E8
_In_ ULONG InputBufferLength,30
_Out_ PVOID OutputBuffer, 0
_In_ ULONG OutputBufferLength 0
);
0:000> k
# ChildEBP RetAddr
00 00cff670 7232df2b ntdll!NtDeviceIoControlFile
01 00cff724 7232dc08 mswsock!SockDoConnectReal+0x2c6
02 00cff7b0 7232e3af mswsock!SockDoConnect+0x412
03 00cff7cc 75ae4d76 mswsock!WSPConnect+0x1f
04 00cff81c 009711df WS2_32!connect+0x86
WINDBG>dd 0036F4E8 0036F4E8+30
0036f4e8 007d4800 00000000 007e1270 00000022
0036f4f8 a42e7cda 480cd03f dea4c29c 78b8ab20
0036f508 b1d00d3e 4570fe10 487662ad 1b7a9d77
WINDBG>dtx _GUID 0036F4E8+10
(*((_GUID *)0x36f4f8)) : {A42E7CDA-D03F-480C-9CC2-A4DE20ABB878} [Type: _GUID] - HV_GUID_PARENT
[<Raw View>] [Type: _GUID]
WINDBG>dtx _GUID 0036F4E8+20
(*((_GUID *)0x36f508)) : {B1D00D3E-FE10-4570-AD62-7648779D7A1B} [Type: _GUID] – GUID of our service.
[<Raw View>] [Type: _GUID]
We pass into a kernel in the afd!AfdConnect function. At first there is a check whether service from 32-bit process, or 64-bit was caused (nt!IoIs32bitProcess). Then the size of transferred buffer (since 0x22) is compared to AfdStandardAddressLength (equal 1Ch), the size of this structure is equal 24h therefore there is an allocation of a pool (ExAllocatePoolWithTagPriority) in 24h byte and copying to it the transferred buffer. What is interesting, the upper bound of InputBufferLength is not controlled in any way, and from usermode we can give any size of the buffer, which behind a deduction 0xC, will be transferred to the nt!ExAllocatePoolWithTagPriority function.
Further SOCKADDR_SIZE is called - function which on the basis of number of the protocol receives size of socket of the address from the array:
Further after numerous checks afd!AfdCreateConnection is caused (in rcx pointer on element of structure afd! AfdTlTransportListHead). The quantity of elements of such structure can be seen through WinDBG mex expansion (if there is no desire to bother with team! list)
WINDBG>!mex.foreachitem afd!AfdTlTransportListHead -c
fffff80535407530
ffffc80e74a37bf0
ffffc80e74814b10
ffffc80e742a9400
ffffc80e742a9320
Processed 5 items.
From it nt!PsChargeProcessPoolQuota is caused in which there is a work with the PplConnectionPool variable. Further the 100h-sized area of memory is nullified, ExpInterlockedPopEntrySList is caused - the new structure of _AFD_CONNECTION begins to be formed, in the field "type" writes 0xAFD8. Further afd!AfdTimerWheelInitializeEntry, then nt!ObfReferenceObject, where as an object - ClientExample.exe process. Afd!AfdReceiveWindowSize and afd!AfdSendWindowSize are loaded respectively in +90h and +94h offsets of structure. There is a check of esi (the 3rd AfdCreateConnection parameter is loaded there) if it is equal 0, then nt!IoCreateFile is caused, however is equal in our case to 1. We return from afd!AfdCreateConnection. afd!AfdAddConnectedReference is caused (the 1st parameter pointer to _AFD_CONNECTION) – 16th bit of this structure set to 1 and increases on 1 48th bit, then afd!AfdEnableFailedConnectEvent is called (it is dumped 3ch+8 bit and DWORD on offset 18Ch the same structure is nullified). Further afd!AfdGetEndpointConnectDispatch (returns or the address of the afd!AfdTlClientConnectDispatch procedure, or if the 7th byte of _AFD_CONNECTION is equal 10h, then AfdRioTlClientConnectDispatch returns) and afd!AfdRefTLBaseEndpoint are carried out.
Then hvsocket!VmbusTIProviderConnect is caused. Inside hvsocket!VmbusTlEndpointIsPrivileged and hvsocket!VmbusTlValidateSockAddress are caused (the structure from two GUID, specified in socket parameters from the ClientExample appendix is transferred to rdx, check correctness of Address Family value - 0x22 and the transferred GUID is checked whether the hvsocket!VmbusTlIsServiceEnabled service is created). Hvsocket!VmbusTlFindOrCreateService is caused from hvsocket!VmbusTlIsServiceEnabled and make search of GUID in list, created by services values from registry
WINDBG>!mex.foreachitem @rdx -x "dt nt!_GUID @#Item+10"
Item #1 @ 0xffffaf89af0af750
nt!_GUID
{999e53d4-3d5c-4c3e-8779-bed06ec056e1}
Item #2 @ 0xffffaf89adac0200
nt!_GUID
{a5201c21-2770-4c11-a68e-f182edb29220}
Item #3 @ 0xffffaf89ade0c870
nt!_GUID
{acef5661-84a1-4e44-856b-6245e69f4620}
Item #4 @ 0xffffaf89adc6a960
nt!_GUID
{b1d00d3e-fe10-4570-ad62-7648779d7a1b} – Our service
Item #5 @ 0xffffaf89ad8ddc20
nt!_GUID
{b1d00d3e-fe10-4570-ad62-7648779d7a1c} – Testing service
Item #6 @ 0xffffaf89ad90bcd8
nt!_GUID
{00000000-0000-0000-0000-000000000000}
Item #7 @ 0xffffaf89adbc1590
nt!_GUID
{7fdfd0ea-cea8-4576-92d6-e072ddd2c422}
In function there is hvsocket!VmbusTlResolvePartitionId call, parameter is transmitted through xmm0 into which GUID of the virtual machine received through Get-VM - name of $VMName.Id is loaded.
As hvsocket!VmbusTlFindAndReferencePartition occurs comparison of the transferred GUID with the known GUID (child, zero or parent partition). At coincidence the same GUID comes back to rdi.
WINDBG>dc @rbx
ffff8907`0a21b5a0 00000004 00000000 00000002 00000000 ................
ffff8907`0a21b5b0 00000001 00000000 00000000 00000000 ................
ffff8907`0a21b5c0 00000000 00000000 00060001 00000000 ................
ffff8907`0a21b5d0 0a21b5d0 ffff8907 0a21b5d0 ffff8907 ..!.......!.....
ffff8907`0a21b5e0 00000000 00000000 00000000 00000000 ................
lea rbx, [rcx-0D8h]
mov rax, [rbx+0E8h]
cmp rax, [rdx]
WINDBG>dt _GUID @rbx+e8h
combase!_GUID
{90db8b89-0d35-4f79-8ce9-49ea0ac8b7cd} - HV_GUID_CHILDREN
WINDBG>dt _GUID @rbx+e8h
combase!_GUID
{a42e7cda-d03f-480c-9cc2-a4de20abb878} - HV_GUID_PARENT
Function does not find GUID and returns 0 that leads to return of error code 0x0C0000141.
WINDBG>!error 0C0000141
Error code: (NTSTATUS) 0xc0000141 (3221225793) - The address handle given to the transport was invalid.
Next are caused: hvsocket!VmbusTlCreateConnection->hvsocket!VmbusTlCreateEndpoint->hvsocket! VmbusTlCreateObjectFromLookasideList
Before function call:
WINDBG>!mex.foreachitem @r8 -c
Mex External 3.0.0.7172 Loaded!
ffffdc00c33555a8
Processed 1 items.
After function call:
WINDBG>!mex.foreachitem ffffdc00c33555a8 -c
ffffdc00c33555a8
ffffc80e75920830
Processed 2 items.
Then Work Queue hvsocket!VmbusTlEndpointActionWorkQueueRoutine is initialized (through netio! NetioInitializeWorkQueue). Return from hvsocket!VmbusTlCreateEndpoint. Further 2 DPC are initialized: hvsocket! VmbusTlConnectTimeoutDpc and hvsocket! VmbusTlOppositeEndpointDisconnectTimeoutDpc (also initialization of the timer is carried out – without KeSetTimer). We return from hvsocket! VmbusTlCreateConnection in hvsocket! VmbusTIProviderConnect
Further there is hvsocket!VmbusTlContainerGetVmId call (in detail was considered in the section Bind earlier). Then hvsocket!VmbusTlAssociateConnectionToPartition is caused. Parameters
WINDBG>r xmm1:ud
xmm1=78b8ab20 dea4c29c 480cd03f a42e7cda
WINDBG>r xmm0:ud
xmm0=cdb7c80a ea49e98c 4f790d35 90db8b89
then hvsocket! VmbusTlSetupConnection (the 2nd parameter is LIST_ENTRY element which was created by the hvsocket! VmbusTlCreateObjectFromLookasideList function) from which hvsocket!VmbusTlSetObjectCancellable is caused (1 element is added to earlier created AVL table, before that table is empty).
Further hvsocket!VmbusTlXPartChildSetupConnection from where hvsocket!VmbusTlSetupConnectionId is caused (in our case work went with HV_GUID_ZERO, but if the transferred GUID differs, then nt!ExUuidCreate is caused and then hvsocket!VmbusTlSetEndpointId), then vmbus!ChTlConnectRequest is caused, 3rd parameter is GUID of the our service:
WINDBG>dtx _GUID @r8 -r
(*((_GUID *)0xffffdc00c3355550)) : {B1D00D3E-FE10-4570-AD62-7648779D7A1B} [Type: _GUID]
(if function returns 0, then error code 0x0C000009A will be returned further)
Further vmbus!ChAllocateSendMessageSized -> vmbus! XPartAllocateSendMessage are caused (comes down to nt!ExAllocatePoolWithTag call and zeroing allocating memory) and already habitual work with vmbus described in the previous article also begins (Hyper-V Internals - http://hvinternals.blogspot.com/2015/10/hyper-v-internals.html). If vmbus! ChAllocateSendMessageSized was executed successfully
WINDBG>kc
Call Site
vmbus!ChAllocateSendMessageSized
vmbus!ChTlConnectRequest
hvsocket!VmbusTlXPartChildSetupConnection
hvsocket!VmbusTlSetupConnection
hvsocket!VmbusTlProviderConnect
afd!AfdConnect
and the pool was allocated, vmbus!ChSendMessage is carried out
WINDBG>dc @rdx - the 2nd vmbus!ChSendMessage parameter
ffffc80e`758d8ba8 00000015 00000000 00000000 00000000 ................
ffffc80e`758d8bb8 00000000 00000000 b1d00d3e 4570fe10 ........>.....pE
ffffc80e`758d8bc8 487662ad 1b7a9d77
WINDBG>kc
Call Site
winhv!WinHvPostMessage
vmbus!PncSendMessage
vmbus!XpartSendMessage (jmp from vmbus!ChSendMessage)
vmbus!ChTlConnectRequest
hvsocket!VmbusTlXPartChildSetupConnection
hvsocket!VmbusTlSetupConnection
hvsocket!VmbusTlProviderConnect
afd!AfdConnect
There is parameters of winhv!WinHvPostMessage
Rcx = 1
Rdx = 1
R9 = 0x28
WINDBG>dc @r8 @r8+28 – a message body
ffffa60d`d4eaebb8 00000015 00000000 00000000 00000000 ................
ffffa60d`d4eaebc8 00000000 00000000 b1d00d3e 4570fe10 ........>.....pE
ffffa60d`d4eaebd8 487662ad 1b7a9d77
Before winhv!WinHvpHypercallRoutine call (goes to vmcall)
Rcx = 0x5c
Rdx = 0
R8 = 23a9000
R9 = 0
WINDBG>!dd @rdx
# 62af000 00000001 3ba2286a 00000001 00000028
# 62af010 00000015 00000000 00000000 00000000
# 62af020 00000000 00000000 b1d00d3e 4570fe10
# 62af030 487662ad 1b7a9d77 00000000 00000000
0x15 - CHANNELMSG_TL_CONNECT_REQUEST
0x28 – the size of the transferred message
The message is sent, we return to hvsocket!VmbusTlXPartChildSetupConnection. hvsocket!VmbusTlPendConnect-> hvsocket! VmbusTlPendConnectLocked are caused. Return to hvsocket!vmbusTiSetupConnection, hvsocket! VmbusTlConnectQueueTimer is caused further (comes down to nt!KeSetTimer call, DPC is hvsocket! VmbusTlConnectTimeoutDpc). We come back to afd!AfdConnect, in eax - 103h therefore afd!AfdTLPendRequest is caused further and then afd! AfdTLConnectComplete2 (afd! AfdCloseConnection, afd! AfdFinishConnect+nt! IofCompleteRequest).
Exit from afd.sys
Guest OS sent the message. Let's separately consider how the message sent through vmbus is processed. As is well-known from Hyper-V internals article, all messages sent through hvix!HvPostMessage hypercall, are processing by vmbus! ChReceiveChannelMessage. We put bp on this function in root OS and start ServerExample.exe in root OS, ClientExample.exe in guest OS, we stop. In rdx our message:
WINDBG>dc @rdx
ffffbf01`7b2379f0 00000015 00000000 00000000 00000000 ................
ffffbf01`7b237a00 00000000 00000000 b1d00d3e 4570fe10 ........>.....pE
ffffbf01`7b237a10 487662ad 1b7a9d77
There takes place validation of the message, the pointer to hvsocket!HvSocketProviderConnectNotification is loaded into rax. The buffer is allocated with the size 48h (Vnpi tag), netio!NetioInsertWorkQueue is caused with the hvsocket! VmbusTlConnectRequestWorkQueueRoutine parameter. Processing comes to the end on it.
Set bp on hvsocket!VmbusTlConnectRequestWorkQueueRoutine in root OS and restart the client application, there is a stop, go to hvsocket!VmbusTlProcessConnectRequestWorkItem, there are pointer to 3 GUIDs in rcx.
WINDBG>dc @rcx
ffffae08`c1bbb880 6a964317 4a741d87 a646f9ab 0089049b – VM GUID
ffffae08`c1bbb890 b1d00d3e 4570fe10 487662ad 1b7a9d77 – Service GUID
ffffae08`c1bbb8a0 00000000 00000000 00000000 00000000 - HV_GUID_ZERO
Then hvsocket! VmbusTlFindAndReferencePartition is caused with the 2nd HV_GUID_CHILDREN parameter, then hvsocket VmbusTlFindOrCreateService, then hvsocket!VmbusTlIsServiceEnabled (returns 1).
Check of the 3rd GUID on equality of HV_GUID_ZERO, if it true, nt!ExUuidCreate is caused, we receive GUID
WINDBG>r xmm0:ud
xmm0=01cf5129 0c00cd83 11e745c8 da36f003
Then hvsocket!VmbusTlProcessNewConnection is caused. Next hvsocket!VmbusTlProcessNewConnectionForListener from which hvsocket!VmbusTlGetPartitionListenerEndpoint is caused (we receive an element from an AVL tree), then hvsocket!VmbusTlCreateConnection and hvsocket! VmbusTlAssociateConnectionToPartition, hvsocket! VmbusTlSetEndpointId.
Further hvsocket!VmbusTlInsertObjectToTable
WINDBG>dt nt!_RTL_AVL_TABLE ffffae08bfb058a0
+0x000 BalancedRoot : _RTL_BALANCED_LINKS
+0x020 OrderedPointer : (null)
+0x028 WhichOrderedElement : 0
+0x02c NumberGenericTableElements : 2
+0x030 DepthOfTree : 2
+0x038 RestartKey : (null)
+0x040 DeleteCount : 2
+0x048 CompareRoutine : 0xfffff804`602d16c0 _RTL_GENERIC_COMPARE_RESULTS hvsocket!VmbusTlCompareGuids+0
+0x050 AllocateRoutine : 0xfffff804`602d1db0 void* hvsocket!VmbusTlAllocateForAvlTable+0
+0x058 FreeRoutine : 0xfffff804`602d1dd0 void hvsocket!VmbusTlFreeForAvlTable+0
+0x060 TableContext : 0x00000000`6e6f4350 Void
After go to hvsocket!VmbusTlSetObjectCancellable and hvsocket!VmbusTlPendConnect, hvsocket! VmbusTlListenerProcessPendingIncomingConnection.
Return to NETIO! NetiopIoWorkItemRoutine.
In root OS at the same time:
kd> kcn
# Call Site
00 winhvr!WinHvPostMessage
01 vmbusr!PncSendMessage
02 vmbusr!XPartSendMessage
03 vmbusr!ChSendOfferMessageLocked
04 vmbusr!ChOfferChannel
05 vmbusr!RootIoctlChannelOffered
06 vmbusr!RootIoctlDispatch
07 vmbusr!RootDeviceControl
……………..WDF stuff
11 vmbusr!RootIoctlDeviceControlPreprocess
…………… WDF stuff
16 vmbkmclr!KmclpSynchronousIoControl
17 vmbkmclr!KmclpServerOfferChannel
18 vmbkmclr!VmbChannelEnable
19 vmbusr!PipeStartChannel
1a vmbusr!PipeOffer
1b hvsocket!VmbusTlXPartRootSetupConnection
1c hvsocket!VmbusTlSetupConnection
1d hvsocket!VmbusTlXPartAcceptConnection
1e hvsocket!VmbusTlListenerProcessPendingIncomingConnection
1f hvsocket!VmbusTlProcessNewConnectionForListener
20 hvsocket!VmbusTlProcessNewConnection
21 hvsocket!VmbusTlProcessConnectRequestWorkItem
22 hvsocket!VmbusTlConnectRequestWorkQueueRoutine
23 NETIO!NetiopIoWorkItemRoutine
24 nt!IopProcessWorkItem
25 nt!ExpWorkerThread
26 nt!PspSystemThreadStartup
27 nt!KiStartSystemThread
kd> dc @r8 @r8+@r9 – the message:
ffffda85`292c7f30 00000001 00000000 b1d00d3e 4570fe10 ........>.....pE
ffffda85`292c7f40 487662ad 1b7a9d77 0ec85988 11e74d2f .bvHw.z..Y../M..
ffffda85`292c7f50 0c00d483 01cf5129 00000000 00000000 ....)Q..........
ffffda85`292c7f60 00000000 00000000 00002011 00000000 ......... ......
ffffda85`292c7f70 00000000 00000000 00000000 00000000 ................
ffffda85`292c7f80 00000000 00000000 00000000 00000000 ................
ffffda85`292c7f90 00000000 00000000 00000000 00000000 ................
ffffda85`292c7fa0 00000000 00000000 00000000 00000000 ................
ffffda85`292c7fb0 00000000 00000000 00000000 00000000 ................
ffffda85`292c7fc0 00000000 00000000 00000000 00000000 ................
ffffda85`292c7fd0 00000000 00000000 00000000 00000000 ................
ffffda85`292c7fe0 00000000 00000000 0000000b 000100ff ................
ffffda85`292c7ff0 0001000b 004e0079
Also the port is created:
********** Bp winhvr!WinHvCreatePort ********
rcx=0000000000000001
rdx=0000000080000000
r8=000000000000002d
r9=0000000000000004
# Call Site
00 winhvr!WinHvCreatePort
01 vmbusr!ParentClaimInterruptResources
02 vmbusr!XPartCreateInterrupt
03 vmbusr!ChpInitializeServerChannelLocked
04 vmbusr!ChOfferChannel
05 vmbusr!RootIoctlChannelOffered
06 vmbusr!RootIoctlDispatch
07 vmbusr!RootDeviceControl
…….WDFstuff
11 vmbusr!RootIoctlDeviceControlPreprocess
…….WDFstuff
16 vmbkmclr!KmclpSynchronousIoControl
17 vmbkmclr!KmclpServerOfferChannel
18 vmbkmclr!VmbChannelEnable
19 vmbusr!PipeStartChannel
1a vmbusr!PipeOffer
1b hvsocket!VmbusTlXPartRootSetupConnection
1c hvsocket!VmbusTlSetupConnection
1d hvsocket!VmbusTlXPartAcceptConnection
1e hvsocket!VmbusTlListenerProcessPendingIncomingConnection
1f hvsocket!VmbusTlProcessNewConnectionForListener
20 hvsocket!VmbusTlProcessNewConnection
21 hvsocket!VmbusTlProcessConnectRequestWorkItem
22 hvsocket!VmbusTlConnectRequestWorkQueueRoutine
23 NETIO!NetiopIoWorkItemRoutine
24 nt!IopProcessWorkItem
25 nt!ExpWorkerThread
26 nt!PspSystemThreadStartup
27 nt!KiStartSystemThread
And connection to it is carried out:
********** Bp winhvr!WinHvConnectPort ********
rcx=0000000000000004
rdx=0000000080000000
r8=000000000001000c
r9=0000000000000001
# Call Site
00 winhvr!WinHvConnectPort
01 vmbusr!ParentConnectDedicatedInterrupt
02 vmbusr!ParentClaimInterruptResources
03 vmbusr!XPartCreateInterrupt
04 vmbusr!ChpInitializeServerChannelLocked
05 vmbusr!ChOfferChannel
06 vmbusr!RootIoctlChannelOffered
07 vmbusr!RootIoctlDispatch
08 vmbusr!RootDeviceControl
………….WDFStuff
12 vmbusr!RootIoctlDeviceControlPreprocess
………….WDFStuff
17 vmbkmclr!KmclpSynchronousIoControl
18 vmbkmclr!KmclpServerOfferChannel
19 vmbkmclr!VmbChannelEnable
1a vmbusr!PipeStartChannel
1b vmbusr!PipeOffer
1c hvsocket!VmbusTlXPartRootSetupConnection
1d hvsocket!VmbusTlSetupConnection
1e hvsocket!VmbusTlXPartAcceptConnection
1f hvsocket!VmbusTlListenerProcessPendingIncomingConnection
20 hvsocket!VmbusTlProcessNewConnectionForListener
21 hvsocket!VmbusTlProcessNewConnection
22 hvsocket!VmbusTlProcessConnectRequestWorkItem
23 hvsocket!VmbusTlConnectRequestWorkQueueRoutine
24 NETIO!NetiopIoWorkItemRoutine
25 nt!IopProcessWorkItem
26 nt!ExpWorkerThread
27 nt!PspSystemThreadStartup
28 nt!KiStartSystemThread
Then in guest OS nt!IoRegisterDeviceInterface is carried out:
# Call Site
00 nt!IoRegisterDeviceInterface
01 Wdf01000!Mx::MxRegisterDeviceInterface
02 Wdf01000!FxDeviceInterface::Register
03 Wdf01000!FxDeviceInterface::Register
04 Wdf01000!imp_WdfDeviceCreateDeviceInterface
05 vmbus!RootStartDeviceInterfaceByContext
06 hvsocket!VmbusTlXPartProcessNewConnection
07 vmbus!RootNotifyDeviceInterfaceArrival
08 Wdf01000!FxWorkItem::WorkItemHandler
09 Wdf01000!FxWorkItem::WorkItemThunk
0a nt!IopProcessWorkItem
0b nt!ExpWorkerThread
0c nt!PspSystemThreadStartup
0d nt!KiStartSystemThread
NTSTATUS IoRegisterDeviceInterface(
_In_ PDEVICE_OBJECT PhysicalDeviceObject,
_In_ const GUID *InterfaceClassGuid,
_In_opt_ PUNICODE_STRING ReferenceString,
_Out_ PUNICODE_STRING SymbolicLinkName
);
kd> !devobj @rcx
Device object (ffffe38bf77145b0) is for:
00000013 \Driver\ACPI DriverObject ffffe38bf79d5a00
Current Irp 00000000 RefCount 0 Type 00000032 Flags 00001040
SecurityDescriptor ffffa48e93c56dc0 DevExt ffffe38bf7a71c60 DevObjExt ffffe38bf7714700 DevNode ffffe38bf79d1c50
ExtensionFlags (0000000000)
Characteristics (0x00000180) FILE_AUTOGENERATED_DEVICE_NAME, FILE_DEVICE_SECURE_OPEN
AttachedDevice (Upper) ffffe38bf7af5970 \Driver\vmbus
Device queue is not busy.
kd> dx _GUID @rdx
(*((_GUID *)0xffffa48ea28888f0)) : {B1D00D3E-FE10-4570-AD62-7648779D7A1B} [Type: _GUID]
kd> dx UNICODE_STRING @r8
(*((UNICODE_STRING *)0xffffa48ea2888900)) : "{b1d00d3e-fe10-4570-ad62-7648779d7a1b}-{00000000-0000-0000-0000-000000000000}-0000" [Type: UNICODE_STRING]
Then the nt!PnpNotifyDeviceClassChange function is executed
kd> k
# Child-SP RetAddr Call Site
00 nt!PnpNotifyDeviceClassChange
01 nt!PnpDeviceEventWorker+0x263
02 nt!ExpWorkerThread+0xe9
03 nt!PspSystemThreadStartup+0x41
04 nt!KiStartSystemThread+0x16
kd> dc @rcx
ffffa48e`9f13f3f8 cb3a4004 11d046f0 60008fb0 3f051397 .@:..F.....`...?
kd> dx _GUID @rdx
(*((_GUID *)0xffffa48ea0dcd0a8)) : {B1D00D3E-FE10-4570-AD62-7648779D7A1B} [Type: _GUID]
kd> du @rdx+10
ffffa48e`a0dcd0b8 "\??\ACPI#VMBus#0#{b1d00d3e-fe10-"
ffffa48e`a0dcd0f8 "4570-ad62-7648779d7a1b}\{b1d00d3"
ffffa48e`a0dcd138 "e-fe10-4570-ad62-7648779d7a1b}-{"
ffffa48e`a0dcd178 "00000000-0000-0000-0000-00000000"
ffffa48e`a0dcd1b8 "0000}-0000"
ClientExample.exe stream stack at the same time:
Child-SP RetAddr Call Site
ffffd180`715e1070 fffff800`081690d5 nt!KxDispatchInterrupt+0x122
ffffd180`715e11b0 fffff800`08169d32 nt!KiDpcInterruptBypass+0x25
ffffd180`715e11c0 fffff800`07e6a003 nt!KiVmbusInterrupt2+0x212 (TrapFrame @ ffffd180`715e11c0)
ffffd180`715e1358 fffff809`c86a17ff 0xfffff800`07e6a003
ffffd180`715e1360 fffff809`c86a19b4 winhv!WinHvpHypercall+0x57
ffffd180`715e13a0 fffff809`c86a1f96 winhv!WinHvpSimplePoolHypercall+0x40
ffffd180`715e13e0 fffff809`c8628f92 winhv!WinHvPostMessage+0x8e
ffffd180`715e1470 fffff809`c8628664 vmbus!PncSendMessage+0x42
ffffd180`715e14a0 fffff809`c863c08d vmbus!XPartSendMessage+0x60
ffffd180`715e14f0 fffff809`c8669b24 vmbus!ChTlConnectRequest+0x4d
ffffd180`715e1530 fffff809`c8665e67 hvsocket!VmbusTlXPartChildSetupConnection+0xb4
ffffd180`715e1580 fffff809`c8662fe5 hvsocket!VmbusTlSetupConnection+0x18b
ffffd180`715e15d0 fffff809`c98970e2 hvsocket!VmbusTlProviderConnect+0x615
ffffd180`715e1680 fffff800`0842e180 afd!AfdConnect+0x6b2
ffffd180`715e1820 fffff800`0842d064 nt!IopSynchronousServiceTail+0x1a0
ffffd180`715e18e0 fffff800`0842c9e6 nt!IopXxxControlFile+0x674
ffffd180`715e1a20 fffff800`08170493 nt!NtDeviceIoControlFile+0x56
ffffd180`715e1a90 00000000`61e7222c nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ ffffd180`715e1b00)
- Send
The most functional part - data reception and transmission. Let's consider send. Recv, I think, essentially should not differ. Will consider the client application. In usermode mswsock!WPSend is caused to ntdll!NtDeviceIoControlFile
NTSTATUS WINAPI NtDeviceIoControlFile(
_In_ HANDLE FileHandle, - 144 (\Device\Afd)
_In_ HANDLE Event, - 140 (Event)
_In_ PIO_APC_ROUTINE ApcRoutine, 0
_In_ PVOID ApcContext, 0
_Out_ PIO_STATUS_BLOCK IoStatusBlock, 00DBF45C
_In_ ULONG IoControlCode, 1201F
_In_ PVOID InputBuffer, DBF44C
_In_ ULONG InputBufferLength,10
_Out_ PVOID OutputBuffer, 0
_In_ ULONG OutputBufferLength 0
);
However in the standard handler of afd!AfdDispatchDeviceControl we will not get, instead processing of this code will be carried out by afd! AfdFastIoDeviceControl. At initialization of the driver the corresponding handler is registered:
objDrv-> FastIoDispatch = &AfdFastIoDispatch
WINDBG>kc
Call Site
winhv!WinHvSignalEvent
vmbus!BusChSendInterrupt
vmbkmcl!KmclSendSignal
vmbus!PipeWrite
hvsocket!VmbusTlXPartProcessIoRequest
hvsocket!VmbusTlConnectProcessIoRequest
hvsocket!VmbusTlConnectionSend
afd!AfdFastConnectionSend
afd!AfdFastIoDeviceControl
nt!IopXxxControlFile
nt!NtDeviceIoControlFile
nt!KiSystemServiceCopyEnd
wow64cpu!CpupSyscallStub
wow64cpu!DeviceIoctlFileFaul
winhv!WinHvpHypercallRoutine has following parameters:
WINDBG>r
rcx=000000000001005d – hypercall code
rdx=000000000001000a – CONNECTION_ID
r8 = 0
In a research of process of data transmission the Hyper-V Data Exchange component (http://hvinternals.blogspot.com/2015/10/hyper-v-internals.html, the section Integration Services - Data Exchange) in case of data transmission via the general buffer, a signal for its reading is WinHvSignalEvent call. The general buffer for transfer was allocated earlier and represents area of the memory available to reading\writing to both guest OS, and root OS. To see this area it is necessary to put the breakpoint to vmbusr!PkGetReceiveBuffer and to see the buffer pointer of which is located in rcx+18h. The size of the buffer rather big - in 2012 r2 for it was allocated 10 physical pages, these are 40 KB.
WINDBG>dc ffffbf01`7c9a9000 L1000
ffffbf01`7c9a9000 00000028 00000000 00000001 00000000 (...............
ffffbf01`7c9a9010 00000000 00000000 00000000 00000000 ................
……………………………………………………………………………………………………………….
ffffbf01`7c9aa000 00020006 00000004 00000000 00000000 ................
ffffbf01`7c9aa010 00000001 00000008 74736554 74736554 ........TestTest
ffffbf01`7c9aa020 00000000 00000000 00000000 00000000 ................
WINDBG>kcn
# Call Site
00 vmbusr!PkGetReceiveBuffer
01 vmbusr!PipeValidateAndGetReceiveBuffer
02 vmbusr!PipeForwardToValidPacket
03 vmbusr!PipeTryReadOrPeekSingle
04 vmbusr!PipePeekMultiple
05 vmbusr!PipeProcessDeferredReadWrite
06 vmbusr!PipeProcessDeferredIosAndUnlock
07 vmbusr!PipeEvtChannelSignalArrived
08 vmbkmclr!KmclpVmbusManualIsr
09 vmbusr!ParentRingInterruptDpc
0a nt!KiExecuteAllDpcs
0b nt!KiRetireDpcList
0c nt!KiIdleLoop
In the same procedure Work Item is added to queue
kd> k
# Child-SP RetAddr Call Site
00 NETIO!NetioInsertWorkQueue
01 hvsocket!VmbusTlQueueEndpointAction+0x14e
02 hvsocket!VmbusTlDeliverDataIndications+0x6b
03 hvsocket!VmbusTlXPartIndicateReceive+0x9b
04 vmbusr!PipePeekMultiple+0xf3
05 vmbusr!PipeProcessDeferredReadWrite+0x1b6
06 vmbusr!PipeProcessDeferredIosAndUnlock+0x74
07 vmbusr!PipeEvtChannelSignalArrived+0x91
08 vmbkmclr!KmclpVmbusManualIsr+0x1d
09 vmbusr!ParentRingInterruptDpc+0x62
0a nt!KiExecuteAllDpcs+0x2b1
0b nt!KiRetireDpcList+0x5df
0c nt!KiIdleLoop+0x5a
Further it is carried out, data are copied in the buffer allocated by hvsocket.sys, then transmitted to application through the same Fast I/O.
WINDBG>kcn
# Call Site
00 hvsocket!VmbusTlIndicateReceive
01 hvsocket!VmbusTlConnectIoRequestCompleted
02 hvsocket!VmbusTlXPartIoRequestCompleted
03 nt!IopfCompleteRequest
04 hvsocket!VmbusTlFulfillReceiveRequest
05 hvsocket!VmbusTlDeliverSingleDataIndicationList
06 hvsocket!VmbusTlDeliverDataIndications
07 hvsocket!VmbusTlEndpointActionWorkQueueRoutine
08 NETIO!NetiopIoWorkItemRoutine
09 nt!IopProcessWorkItem
0a nt!ExpWorkerThread
0b nt!PspSystemThreadStartup
0c nt!KiStartSystemThread
- PowerShell Direct
Windows PowerShell for a long time supports the PowerShell Remoting protocol which allows to be connected to workstations and servers on network for remote management and executing of any commands. PowerShell Remoting is described by Microsoft in the document [MS-PSRP] which is uploaded publicly within the Open Specifications program. PowerShell Direct uses the same protocol for the work, however the environment of delivery of data is not the network on the basis of TCP\IP of a stack, but VMBUS.
PowerShell Direct uses the same cmdlets, as for PowerShell Remoting: Enter-PSSession, Invoke-PSSession and New-PSSession, only instead of a name of the computer enters a name of the virtual machine or its GUID.
Hyper-V PowerShell Direct Service was created for support of this technology in guest OS (a name of service - vmicvmsession), which functionality is realized in %SystemRoot library %\System32\ICSvc.dll. Start type - Manual (Trigger start). Under what conditions the service will be started?
PS C:\Users\Administrator> sc.exe qtriggerinfo vmicvmsession
[SC] QueryServiceConfig2 SUCCESS
SERVICE_NAME: vmicvmsession
START SERVICE
DEVICE INTERFACE ARRIVAL : 999e53d4-3d5c-4c3e-8779-bed06ec056e1 [INTERFACE CLASS GUID] - HV_GUID_VM_SESSION_SERVICE_ID
It will occur if device with INTERFACE CLASS GUID 999e53d4-3d5c-4c3e-8779-bed06ec056e1 will be connected to system. Apparently, GUID of this device coincides with GUID of service from the section of the registry in root-OS (HKLM:\SOFTWARE\Microsoft \Windows NT\CurrentVersion\Virtualization\GuestCommunicationServices) where for work with sockets 2 keys with GUID 999e53d4-3d5c-4c3e-8779-bed06ec056e1 and a5201c21-2770-4c11-a68e-f182edb29220 are by default created.
In guest OS devices with such GUIDs are present at
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses\{999e53d4-3d5c-4c3e-8779-bed06ec056e1}
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses\{a5201c21-2770-4c11-a68e-f182edb29220}\##?#ACPI#VMBus#0#{a5201c21-2770-4c11-a68e-f182edb29220}\#{a5201c21-2770-4c11-a68e-f182edb29220}-{00000000-0000-0000-0000-000000000000}-0000
The socket is initialized when one of mentioned cmdlets execution in root-OS, at the same time in guest OS the vmicvmsession service creates process of Powershell.exe and one Hyper-V a socket. The data transformed to a XML format are transferred by send socket calls encoded by Base64. The mechanism does not demand inclusion in options of the virtual machine and is available by default. Earlier to learn a work algorithm researchers decompiled the System.Management.Automation.dll module which is a part of Windows Powershell, now the situation became simpler, and source code can be found on github:
(https://github.com/PowerShell/PowerShell/blob/master/src/System.Management.Automation/engine/remoting/common/RemoteSessionHyperVSocket.cs).
Here it is possible to gather a lot of information, in particular, in what way to realize the own application for work with Hyper-V as sockets on C#.
Let's put the symbolical breakpoints: nt!PnpNotif * in guest OS and when executing Enter-PSSession we stop on nt!PnpNotifyDeviceClassChange.
kd> kcn
# Call Site
00 nt!PnpNotifyDeviceClassChange
01 nt!PnpDeviceEventWorker
02 nt!ExpWorkerThread
03 nt!PspSystemThreadStartup
VM Session Service 1 GUID is 2nd parameter:
kd> dd @rcx – 1st parameter some GUID
ffffa806`a2b5d6c8 cb3a4004 11d046f0 60008fb0 3f051397
ffffa806`a2b5d6d8 00000002 00000000 00000000 00000000
ffffa806`a2b5d6e8 00000000 00000164 00000000 00000000
kd> dd @rdx – 2nd parameter
ffffa806`a2b5d6f8 999e53d4 4c3e3d5c d0be7987 e156c06e
kd> du @rdx+10 – on offset +10h - the ID device which appears on VMBUS
ffffa806`a2b5d708 "\??\ACPI#VMBus#0#{999e53d4-3d5c-"
ffffa806`a2b5d748 "4c3e-8779-bed06ec056e1}\{999e53d"
ffffa806`a2b5d788 "4-3d5c-4c3e-8779-bed06ec056e1}-{"
ffffa806`a2b5d7c8 "00000000-0000-0000-0000-00000000"
ffffa806`a2b5d808 "0000}-0000"
Nevertheless this device is not displayed in the list of child vmbus devices (we see it using !devnode 0 1). In the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\VMBus, these GUID is also absent. However DeviceInstance parameter
in the section HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\DeviceClasses\{ 999e53d4-3d5c-4c3e-8779-bed06ec056e1 }\##?#ACPI#VMBus#0#{999e53d4-3d5c-4c3e-8779-bed06ec056e1} matters ACPI\VMBUS\0 that completely coincides with Device Instance value of the vmbus device.
Also when Enter-PSSession is executed the vmbus!RootAddDeviceInterface function works and nt!IoRegisterDeviceInterface:
kd> kcn
# Call Site
00 nt!IoRegisterDeviceInterface
01 Wdf01000!Mx::MxRegisterDeviceInterface
02 Wdf01000!FxDeviceInterface::Register
03 Wdf01000!FxDeviceInterface::Register
04 Wdf01000!imp_WdfDeviceCreateDeviceInterface
05 vmbus!RootStartDeviceInterfaceByContext
06 hvsocket!VmbusTlXPartProcessNewConnection
07 vmbus!RootNotifyDeviceInterfaceArrival
08 Wdf01000!FxWorkItem::WorkItemHandler
09 Wdf01000!FxWorkItem::WorkItemThunk
0a nt!IopProcessWorkItem
0b nt!ExpWorkerThread
0c nt!PspSystemThreadStartup
0d nt!KiStartSystemThread
The last function is documented on MSDN
NTSTATUS IoRegisterDeviceInterface(
_In_ PDEVICE_OBJECT PhysicalDeviceObject,
_In_ const GUID *InterfaceClassGuid,
_In_opt_ PUNICODE_STRING ReferenceString,
_Out_ PUNICODE_STRING SymbolicLinkName
);
kd> !devobj @rcx - PhysicalDeviceObject
Device object (ffffc9094ca24630) is for:
00000013 \Driver\ACPI DriverObject ffffc9094cdeea00
Current Irp 00000000 RefCount 0 Type 00000032 Flags 00001040
SecurityDescriptor ffffb90d7d06a630 DevExt ffffc9094ce89c60 DevObjExt ffffc9094ca24780 DevNode ffffc9094cda7c50
ExtensionFlags (0000000000)
Characteristics (0x00000180) FILE_AUTOGENERATED_DEVICE_NAME, FILE_DEVICE_SECURE_OPEN
AttachedDevice (Upper) ffffc9094ceb4970 \Driver\vmbus
Device queue is not busy
kd> dx _GUID @rdx - InterfaceClassGuid
(*((_GUID *)0xffffb90d7ed57eb0)) : {999E53D4-3D5C-4C3E-8779-BED06EC056E1} [Type: _GUID]
[<Raw View>] [Type: _GUID]
kd> dx _UNICODE_STRING @r8 - ReferenceString
(*((_UNICODE_STRING *)0xffffb90d7ed57ec0)) : "{999e53d4-3d5c-4c3e-8779-bed06ec056e1}-{00000000-0000-0000-0000-000000000000}-0000" [Type: _UNICODE_STRING]
When Enter-PSSession is executed in root-OS ws2_32!connect is carried out, then winhvr!WinHvPostMessage transfers the message to guest OS.
kd> !dc @rdx – just before vmcall call (a part of a body of the message)
#227b36000 00000001 00000000 00000001 000000c4 ................
#227b36010 00000001 00000000 999e53d4 4c3e3d5c .........S..\=>L
#227b36020 d0be7987 e156c06e
Further in guest OS as we saw, executing nt!IoRegisterDeviceInterface and nt! PnpNotifyDeviceClassChange then the trigger on start of service vmicvmsession works. But we can replace the transferred GUID on any other that will provoke start of other services which have a similar trigger. Now such services are:
All Hyper-V guest services
Bluetooth Support Service
Windows Camera Frame Server
Human Interface Device Service
Geolocation Service
Microsoft Passport
Portable Device Enumerator Service
Sensor Service
Sensor Monitoring Service
Storage Service
Touch Keyboard and Handwriting (probably)
From the technical point of view, we simply emulate connection of a certain device to system. Real devices are not presented and some services don’t work in such conditions and after start give the message:
On Shielded VM (it is checked in the Admin-Trusted mode) the effect is similar, however the service Hyper-V Powershell Direct service will not be started.
Its interesting, that value of credentials parameter for Enter-PSSession, which is used for start of powershell.exe in guest OS, is transferred as clear text.
if (emptyPassword)
{
HyperVSocket.Send(Encoding.ASCII.GetBytes("EMPTYPW"));
HyperVSocket.Receive(response);
responseString = Encoding.ASCII.GetString(response);
}
else
{
HyperVSocket.Send(Encoding.ASCII.GetBytes("NONEMPTYPW"));
HyperVSocket.Receive(response);
HyperVSocket.Send(password);
HyperVSocket.Receive(response);
responseString = Encoding.ASCII.GetString(response);
In root OS: bp ws2_32!recv in powershell process context.
In guest OS bp sspicli!LogonUserExExW
ri
Breakpoints on recv and send are set in guest and root OS for viewing of xml-messages. In a RAW format they look as follows:
If to load WINDBG extension for debugging of .NET (.cordll - ve - u - l), then it is possible to see a stack of powershell.exe:
As we see, sending is made by the SendOneItem function from the OutOfProcessTransportManager.cs module.
Command parameter (or code in ScriptBlock cmdlet option) is coded by the CreateDataPacket function from the same OutOfProcessTransportManager.cs module
internal static string CreateDataPacket(byte[] data, DataPriorityType streamType, Guid psGuid)
{
string result = string.Format(CultureInfo.InvariantCulture,
"<{0} {1}='{2}' {3}='{4}'>{5}</{0}>",
PS_OUT_OF_PROC_DATA_TAG, - Data
PS_OUT_OF_PROC_STREAM_ATTRIBUTE, - Stream
streamType.ToString(), - Default
PS_OUT_OF_PROC_PSGUID_ATTRIBUTE, - PSGUID
psGuid.ToString(), - 223adb3d-b639-4e84-aa83-6b193db87e1e
Convert.ToBase64String(data)); - BASE64 текст
return result;
}
Packet header variants:
For instance, if we open PowerShell Direct session and execute the command “mkdir С:\Tools\Test” in guest OS, then a text part of dialogue, will look as follows:
The dialogue scheme: root OS sends several XML messages one of which contains command
And its arguments:
Guest OS returns several messages which contain besides servicing information results of command execution (every line of withdrawal of the mkdir command is transferred as a separate object):
In principle, it is possible to receive decoded data, having put the breakpoint to System.Management.Automation.Remoting.Client.OutOfProcessClientCommandTransportManager.SendData (the address can be seen using !clrstack). When we stop on the necessary point, we execute the command !clrstack – a and receive address of data:
0:020> !clrstack -a
OS Thread Id: 0x3b8 (20)
………………………
PARAMETERS:
this (<CLR reg>) = 0x000001e0a8459060
data (<CLR reg>) = 0x000001e0a84964b0
priorityType (<CLR reg>) = 0x0000000000000000
Размер буфера можно узнать:
0:020> !DumpObj /d 000001e0a84964b0
Name: System.Byte[]
MethodTable: 00007ffc081993d0
EEClass: 00007ffc07bd4dc8
Size: 2219(0x8ab) bytes
When we executed dc in WINDBG it is possible to see message contents:
On PowerShell Direct description page is specified that privileges of Hyper-V Administrators are necessary for using it:
Actually, these privileges are necessary for start working with Enter-PSSession cmdlet. Hyper-V sockets do not require any privileges for their work so this functionality can use in the applications started by unprivileged user.
From root OS we can be connected to the vmicvmsession service. For that in ClientExample.exe application (https://github.com/gerhart01/HyperV-sockets) we will replace HV_PARENT_GUID with GUID of the virtual machine.
WSADATA wsaData;
SOCKADDR_HV clientService;
CLSID VmID, ServiceID;
// Initialize GUIDs
//wchar_t* clsid_str = L"{a42e7cda-d03f-480c-9cc2-a4de20abb878}"; // HV_PARENT_GUID
wchar_t* clsid_str = L"{6a964317-1d87-4a74-abf9-46a69b048900}";
CLSIDFromString(clsid_str, &VmID);
clsid_str = L"{999e53d4-3d5c-4c3e-8779-bed06ec056e1}";
CLSIDFromString(clsid_str, &ServiceID); //GUID of Powershell Direct Service
Let's compile, we will start application and then receive answer from guest OS:
We see how icsvc.dll module begins interaction with the client. The protocol of a client part is present in Powershell source codes in the OutOfProcTransportManager.cs and RemoteSessionHyperVSocket.cs therefore technically there are no obstacles to copy this code in the stand-alone program and to use it as full client for communications with PowerShell Direct service without any additional privileges. Certainly, there will be a need to specify registration data under powershell.exe process will be started in guest OS.
The table with ports created by root OS for communications with guest OS. Two additional ports are created for PowerShell Direct.
№
|
Call stack (or calling function)
|
Created Ports (r8)
|
PortID
|
Type
|
SINTx
|
vmbusr!ParentCreateParentMessagePort
vmbusr!XPartCreateVidPartition
vmbusvdev!VmbusVdev::PowerOnCold
|
1
|
2
|
HvPortTypeMessage
|
4
| |
vmbusr!ParentCreateChildMessagePort
vmbusr!XPartCreateVidPartition
vmbusvdev!VmbusVdev::PowerOnCold
|
21
|
1
|
HvPortTypeMessage
|
2
| |
vmbusr!ParentCreateChildEventPort
vmbusr!XPartCreateVidPartition
vmbusvdev!VmbusVdev::PowerOnCold
|
2
|
1
|
HvPortTypeEvent
|
2
| |
vmdynmem!DynMemDevice::PowerOnCold
|
d
|
10001
|
HvPortTypeEvent
|
5
| |
vmuidevices!SynthRdpDevice::PowerOnCold
|
e
|
10002
|
HvPortTypeEvent
|
5
| |
vmuidevices!SynthMouseDevice::PowerOn
|
12
|
10003
|
HvPortTypeEvent
|
5
| |
vmuidevices!SynthKeyboardDevice::PowerOnCold
|
13
|
10004
|
HvPortTypeEvent
|
5
| |
vmuidevices!VideoSynthDevice::PowerOn
|
14
|
10005
|
HvPortTypeEvent
|
5
| |
ActivationVDev!CSppActivationVDevT<CEmptyType>::PostReset
|
15
|
10006
|
HvPortTypeEvent
|
5
| |
vmicvdev!ICVdevDevice::PowerOnCold
|
16
|
10007
|
HvPortTypeEvent
|
5
| |
vmicrdv!ICRdvVdevDevice::PowerOnCold
|
17
|
10008
|
HvPortTypeEvent
|
5
| |
VmSynthNic!SynthNic::PowerOn
|
18
|
10009
|
HvPortTypeEvent
|
5
| |
vmsynthstor!SynthStor::PowerOn
|
19
|
1000a
|
HvPortTypeEvent
|
5
| |
winhvr!WinHvCreatePort
vmbusr!XPartParentCreateNewMessa
vmbusr!ChServerReserveChannelLoc
vmbusr!ChmOpenReservedChannel
|
41
|
1a
|
HvPortTypeMessage
|
4
| |
winhvr!WinHvCreatePort
vmbusr!ParentCreateMonitors
vmbusr!XPartChildConnect
vmbusr!ChpChildConnectLocked
vmbusr!ChpNegotiateVersionLocked
|
1b
|
1b
|
HvPortTypeMonitor
|
-
| |
winhvr!WinHvCreatePort
vmbusr!ParentCreateMonitors
vmbusr!XPartChildConnect
vmbusr!ChpChildConnectLocked
vmbusr!ChpNegotiateVersionLocked
|
3
|
3
|
HvPortTypeMonitor
|
-
| |
winhvr!WinHvConnectPort
vmbusr!ParentConnectDedicated
vmbusr!ParentClaimInterruptRe
vmbusr!XPartCreateInterrupt
WS2_32!WSAConnect
|
1c
|
1000b
|
HvPortTypeEvent
|
5
| |
winhvr!WinHvCreatePort
vmbusr!ParentClaimInterruptR
vmbusr!XPartCreateInterrupt
WS2_32!WSAConnect
|
1d
|
1000c
|
HvPortTypeEvent
|
5
|
After that in general the picture becomes more clear: when Enter-PSSession is executed in guest OS registration of the new interface for the vmbus driver is initiated. That operation leads to start of service vmicsession which starts powershell process with credentials transferred from root OS, this process opens Hyper-V sockets and starts to communicate with powershell.exe working in root OS using send and recv calls.
Additional, I think, it should be noted that some logic of drivers execution (hvsocket.sys in our case) can be learned, using Windows software trace preprocessor (WPP).
In many drivers developed by Microsoft WPP is used, and traceview utility from WDK allows to see it. For this purpose it is necessary to know trace GUID. If to look in IDA, then the name of a variable which contains that GUID will be look as WPP_ThisDir_CTLGUID_HvSocketTraceGuid or WPP_ThisDir_CTLGUID_VMBusDriverTraceGuid (for vmbus driver). GUID will be in a binary look: 0B8A5B44354C0BBA849083340689010E5h. It needs to be transformed to a usual format (689010e5-3340-4908-a8bb-c05443b4a5b8), for instance, made dt _GUID team <the variable address> in WinDBG.
The received GUID can be entered into TraceView, when new session is started: File-> Create Ne Log Session-> Add Provider->
Then to specify a path to the file in which data will be wrote. The picture will turn out not too informative (TMF file for decoding Microsoft does not offer):
But if we load saved etl-file into Windows Message Analyzer, then we will be able to see PID and TID, and also the WPP message in a RAW look:
We can see what contains in that addresses using WinDBG (Similar binary values met at a stage of initialization of a socket)
kd> dc FFFF9D0C351266C0
ffff9d0c`351266c0 00000001 00000000 00000004 00000000 ................
ffff9d0c`351266d0 00000001 00000000 00000000 00000000 ................
ffff9d0c`351266e0 00000000 00000000 00060001 00000000 ................
kd> dc FFFF9D0C34A9C340
ffff9d0c`34a9c340 00000001 00000000 00000003 00000000 ................
ffff9d0c`34a9c350 00000001 00000000 00000000 00000000 ................
ffff9d0c`34a9c360 00000000 00000000 00060001 00000000 ................
In general, it is possible to draw a conclusion that PowerShell Direct was created for such cases when Hyper-V administrator and the administrator of the virtual server - the same person. Versatile dataflow goes from guest OS in a host OS, and is processed by powershell.exe started with Hyper-V Administrator privileges (the requirement for Powershell Direct work). Of course, having it is unlikely sent messages with the <cmd> tag to root OS, we can execute any commands, but the fact that vulnerabilities find in the dependent Windows components (.net, xml) more often than in the hypervisor module, is visually visible, for example, in the NVD database. However, specific conditions of use of vulnerabilities is necessary, in case of their existence, can make them unusable (the configuration - the terminal XenApp\RDS server on which user can get local administrator privileges and on which the Hyper-V administrator uses Powershell Direct).
- Conclusion
In article, we considered some aspects of Hyper-V sockets work. By results it is possible to understand that from the architectural point of view their work a little differs from work of usual network sockets, however for their support certain changes were made to Windows network stack components. In my opinion, it is the first documented channel for data exchange between guest and host operating systems in Hyper-V virtualization environment. It can be used, for example, for data exchange between the USB device connected to root OS and guest OS that is quite useful for hardware locks of protection or two-factor authentication at an entrance to an operating system. Certainly, in the example given above it is used only for demonstration of their work. As, for example, how this mechanism of data transmission will be work for multithreaded applications with several channels or by transfer of large volume of data, it is difficult to tell now. Let's look whether developers will be use the opportunity of communications between guest and OS host given by the Microsoft.
P.S. PDF version
