Great thanks to ERNW for the translation of the article!
This is translation of article published on securitylab.ru - http://www.securitylab.ru/contest/444112.php with some fixes for latest Hyper-V version (Windows Server 2016 TP2).
For the study was used the VMware Workstation 12, WinDBG 10, IDA PRO and different versions of Windows. To create a VMware virtual machine, set the type of the guest OS to Hyper-V and put the number of processors and cores to 1. Activate the Virtualize Intel VT-x / EPT, install Windows Server 2016 TP2 to activate the role of Hyper-V (gui you can install too) and install a guest in relation to the Hyper-V on Windows 10 x64.
1. Terms and definitions
− The hypervisor – component of Hyper-V, depending on the manufacturer of the processor (hvix64.exe for Intel and hvax64.exe for AMD). The article discusses the Intel hypervisor processor.
− Hypercall – call a given function in the hypervisor using the instructions vmcall.
− Root-partition – Windows Server 2016 TP2 with the included component of Hyper-V.
− VMCS (virtual-machine control structure) – a structure that defines the logic of the hypervisor.
− VMX root – mode, which is running a hypervisor.
− VMX non-root – mode in which the running operating system and its client application software.
− VM exit – the transition of the VMX non-root into VMX root. Occurs when the execution of instructions or conditions specified in the VMCS incorporated directly into the logic of the processor.
2. Debugging
Hyper-V consists of several components, a brief description can be found in (1). For debugging all components except the hypervisor you can use the standard methods, however, to connect to the hypervisor you have to perform a few extra steps to configure root-partition.
For debugging the hypervisor, Microsoft developed a special extension to WinDBG hvexts.dll, which, unfortunately, is not included in the distribution debugger and is available only to partners (probably, because that extension needs symbols for hvix64.exe which is not present). Also in the catalog winxp, located in a folder with WinDBG, is an extension of nvkd.dll, which is intended for debugging extensions virtual switch Hyper-V.
The MSDN (2) and (3) is a description of debugging hypervisor via cable through the com-port, implying the presence of two physical machines. However, the hypervisor can be debugged, if you run it in VMware and use the com-port emulator Free Virtual Serial Ports utility from the HHD-software (4). To do this:
− create com-port for a virtual machine (Hardware->Add->Serial port->Output to a named pipe)
− to perform root-partition commands to configure debugging hypervisor and the OS:
bcdedit /hypervisorsettings serial DEBUGPORT:1 BAUDRATE:115200
bcdedit /set hypervisordebug on
bcdedit /set hypervisorlaunchtype auto
bcdedit /set dbgtransport kdhvcom.dll
bcdedit /dbgsettings serial DEBUGPORT:1 BAUDRATE:115200
bcdedit /debug on
Bcdedit /set bootdebug on (needed to study the process for loading the hypervisor)
− restart Windows Server 2016 TP2. pending connections will stop Loading the debugger.
− run Free Virtual Serial Ports Select Pipe and press Create. In the field of Pipe name specify the same value for a virtual machine- \\.\pipe\com_1. Press Create.
In the case of a successful connection to the named pipe it will create a virtual com–port
- Run vmdemux (located in the Setup directory of WinDBG), specifying the name of the port as one of the parameters:
vmdemux.exe -src com:port=com2,baud=115200
In case of a successful connection we get:
You created a named pipe \\.\pipe\Vm1 must be used to attach the debugger:
WinDBG.exe -b -k com:port=\\.\pipe\Vm1,pipe,reconnect,resets=0
At the same time the debugger connects to the root-partition. Then you need to execute g command several times, then vmdemux shall issue:
After that, with the help of IDA PRO, you can connect directly to the hypervisor via a named pipe \\.\Pipe\Vm0, choosing as WinDBG debugger and specifying process options in the connection string:com:port=\\.\pipe\Vm0,pipe,resets = 0
In case the following message appears choose Same.
The debugger will stop within the hypervisor:
As we see in comparing with Windows Server 2012 (R2) there is new module – kdstub.dll. Early version of hypervisor have static link with debug library and huge size of file (2-3Mb), size of current version of hvix64.exe – 932 Kb.
On next stages kdstub.dll will be changed on appropriate debug module, f.e. kd_02_8086.dll (for network debugging):
In Windows Server 2012 hypervisor and higher an opportunity to debug the network, and even on MSDN at the time the article was no description of this method, however, a little digging in to help utility bcdedit, you can choose the options you want.
To do this in Windows Server 2016 TP2, it is necessary to write
Bcdedit /set dbgtransport kdnet.dll
Bcdedit /debug yes
Bcdedit /dbgsettings net hostip:192.168.2.1 port:50002
in response, the command will display the connection string of the root - partition
bcdedit /set hypervisordebug on
bcdedit /hypervisorsettings NET HOSTIP:192.168.2.1 PORT:50000
in response, the command will display the connection string of the hypervisor.
Inside the VMware virtual machine configuration for installing the Host Only adapter, go into the virtual network settings to configure DHCP for the adapter and make sure that Windows Server 2016 TP2 is normally assigned to this address, for example, by running the command ipconfig / renew.
Then run 2 instances of IDA PRO, set the debug type to KernelMode and specify the Process Option->Connection string to the following line from the command above:
net:port=50002,Key=pv1l8rzwhxhz.2vpkq86oc8zwg.tly17iosgzm8.1h7r18svpji4p - the root partition
net:port=50000,Key=3fkmf6l8a1tnd.3tsxig92rw4cc.2l0dhyq3p24qj.rgbz64xkofc0 - hypervisor
thereby acquiring the ability to simultaneously debug root-partition and the hypervisor.
Option bcdedit /dbgsettings nodhcp allows the debugger to use network mode, use the ip-address of the root partition. In this case, configuring the DHCP in VMware is not necessary.
You can use kdnet.exe (10), which was included in Debugging Tools for Windows 10, for configuring root-partition for debugging:
kdnet.exe 192.168.2.1 50002
You can see message “Microsoft hypervisor supports using KDNET in guest VMs”
For checking this feature kdnet.exe:
- Detecting the presence of a Hypervisor – cpuid(1), ecx[31] must be equivalent 1.
- Check Hypervisor Vendor ID Signature – cpuid (0x40000000) must return “Microsoft Hv” in ebx,ecx,edx registers.
- Check Hypervisor Interface Signature – cpuid(0x40000001), eax must contains Hv#1
- Check Build Number – cpuid(0x40000002), it must be equivalent or above 0x23F0. If it below caption appears:
The Microsoft hypervisor running this VM does not support KDNET. Please upgrade to the hypervisor shipped in Windows 8 or WS2012 or later.
- Finally execute cpuid (0x40000003) and check that ebx[12] must be 1 (it means that partition was created with CpuManagement flag)
If all checks was passed kdnet prints message about KDNET supporting.
If you configure guest virtual machine, kdnet.exe prints additional message:
C:\Program Files\Windows Kits\10\Debuggers\x86>kdnet 10.0.0.1 50020
Enabling network debugging on Microsoft Hypervisor Virtual Machine.
Key=3d1uhidq70zko.3uge1t58fhyaa.2ybt7ue2dbmou.1ehlagdp52dvg
To finish setting up KDNET for this VM, run the following command from an
elevated command prompt running on the Windows hyper-v host. (NOT this VM!)
9EE98E4 -port 50020
Then make sure to SHUTDOWN (not restart) the VM so that the new settings will
take effect. Run shutdown -s -t 0 from this command prompt.
To debug this vm, run the following command on your debugger host machine.
windbg -k net:port=50020,key=3d1uhidq70zko.3uge1t58fhyaa.2ybt7ue2dbmou.1ehlagdp5
2dvg,target=SRV2016
It mentions private kdnetdebugvm.ps1 script, which doesn’t present in debugger tools. KDNET.exe read vmguid from HKLM\Software\Microsoft\Virtual Machine\Guest\Parameters\VirtualMachineId key and debugging key from HKLM\BCD00000000\Objects\{4636856e-540f-4170-a130-a84776f4c654}\Elements\1200001d\Element
Debugging the guest against Hyper-V OS can be made either by the standard method via a virtual com-port or by using the debugging capabilities of the hypervisor. An example of second variant was mentioned on OSR Online (5), and this is how you can set it up:
- copy the file kdvm.dll from the Windows 8 directory C:\Windows\system32\kdvm.dll same goes for Windows 7 (of course, the file must be identical to the 64-bit operating system). For Windows 8.1 \ Windows Server 2012 R2 kdvm.dll must be taken from preview-build, since the RTM versions of the file has been removed. It looks like you cannot use that method on Windows Server 2016\Windows 10 OS - winload ignores “dbgtransport” with kdvm.dll parameter and load kd.dll. Yes, use standard COM-debugging!
winload.exe Windows 8.1 x86
|
winload.exe Windows 10 x86 (kdvm.dll is absent)
|
 |  |
- in Windows 8.1 run following commands
Bcdedit /set dbgtransport kdvm.dll
bcdedit /set {default} loadoptions host_ip="1.2.3.4",host_port=50011,encryption_key="1.2.3.4"
bcdedit /set debug on
- restart the OS.
- specify the parameters of the script hyperv-dbg.ps1 (the script in the archive has been adapted for Windows Server 2012 R2)"
- run the script hyperv-dbg.ps1 (run through the „Run as Administrator“, or disable UAC, run gpedit.msc and set Computer configuration \ Windows Settings \ Security Settings \ Local Policies \ Security Options \ User Account Control: Run All administrators in Admin Approval Mode to Disable) in the root-section
- start WinDBG:
WinDBG -k net:port=50011,target=127.0.0.1,key=1.2.3.4
− execute the command break, then the debugger will stop inside the guest OS:
Also, for the virtual machine VMware, where Windows Server 2016 TP2 is installed on, the gdb-debugger must be enabled. To do this, vmx-file of this machine, you have to add the line
debugStub.listen.guest64 = "TRUE"
debugStub.hideBreakpoints= "TRUE"
3. Loading the hypervisor
The research used hvloader.efi (10.0.10011.0) and hvix64.exe (10.0.10074.0). Before debugging load winload.exe into IDA PRO, choose Debugger -> Select Debugger -> GDB, in the Process Options to specify the Host name 127.0.0.1 and port 8864.
- Thanks to the previously installed boot loader options bootdebug on an early connection to download winload.efi, which produces the hypervisor launch this after the start of the OS, you need to:
- run WinDBG:
WinDBG.exe -b -k net:port=50002,key= pv1l8rzwhxhz.2vpkq86oc8zwg.tly17iosgzm8.1h7r18svpji4p
These circumstances must occur within the function winload! DebugService2
- find download address of winload.efi
kd> lm
start end module name
00000000`00939000 00000000`00aae000 winload (pdb symbols)
- run IDA PRO and load the previously analyzed module winload.efi, choose Debugger -> attach to process -> attach to process started on target, and after stopping run Edit -> Segments -> Rebase program, specified in the Image base load address winload.efi (0x00939000) and save it in IDA PRO. When loading winload.exe ASLR is not used, so the load address will not change, when you restart the operating system and downloading to the IDA PRO winload.efi will be immediately posted to the correct address.
- put in IDA PRO a breakpoint on winload!OslArchHypervisorSetup and continue debugging (F9). Also continue debugging in WinDBG:
kd> g
Winload checks whether the given parameter loader hypervisorlaunchtype (0x250000f0) is.
If the parameter is specified and its value is 1 (Auto), the function call HvlpLaunchHvLoader-> BlImgStartBootApplication->ImgArchEfiStartBootApplication that loads and passes the control module hvloader.efi which will have to download the file of the hypervisor hvix64.exe and prepare it for future work.
Function BlBdStop shuts off the WinDBG, but you debug through gdb in VMware, which cannot be prevented.
The function Archpx64TransferTo64BitApplicationAsm is used to give control to the hvlMain from hvloader.efi(the address of the function hvlMain is in ArchpChildAppEntryRoutine).
For properly debugging hvloader.efi, you need replace the first instruction of HvlMain to EB FE 90, that fix the code and will provide an opportunity to restart IDA PRO, download hvloader.efi and reconnect gdb-debugger to VMware. Then you must return the changed bytes in place and perform rebase module. To improve the speed of operations you can apply changes to code with simple scripts written in python (PatchHvLoader.py and RestoreHvLoader.py). Base load hvloader.efi does not change and always has been 0x4BA000, so that, by analogy with winload.efi once performed rebase, the base remains, and on subsequent connections debugger module is located to the right address without performing additional operations.
In hvloader.efi you should pay attention to the function BtPrepareHypervisorLaunch (called from HvlMain->HvlpPrepareHypervisorForLaunch), which does basic operations for loading the hypervisor. Shortly before calling this function, you can see that the function BtLoadUpdateDll, which loads the library processor microcode updates mcupdate_GenuineIntel.dll. The functions BtLoadUpdateDll and BtPrepareHypervisorLaunch first performing BtpIdentifyPlatform, which is determined by the manufacturer of the processor
and returns a pointer to a structure BtpPlatformTable and the names of uploaded files.
Pointers to function VmxDetect and SvmDetect need only BtPrepareHypervisorLaunch. These functions are called immediately after BtpIdentityPlatform depending on the platform (VmxDetect for Intel and SvmDetect for AMD):
VmxDetect, for example, determines the capabilities of the processor
and returns a pointer to the next platform specific function VmxValidate (SvmDetect returns SvmValidate), etc.
Additionally, attention may be drawn to the calculation of the random offset for the load address of the hypervisor 0xFFFFF800 00000000 and its subsequent displacement by calling BtpLayoutHvImage.
Also BtPrepareHypervisorLaunch call BtpLayoutKdExtension which load kdstub.dll
The structure BtpAllocateAndBuildLoaderBlock is filled with BtpLoaderBlockPages (HvlpLoaderBlock in winload), which later will be used to transfer control to the start of the procedure hvix64.exe.
The Rebase Messages
Rebase Hv by: 6c25000
Rebase Kdnet extension by: 7000
show the boot offset hypervisor and kdstub.dll on address 0xFFFFF800 00000000. This shift will be needed at the moment we switch to IDA PRO debug with winload.exe on hvix64.exe
Back in the winload.efi
The function HvlpTransferToHypervisor made the transition to the start feature of hvix64.exe.
The Instruction jmp r8 transfers execution to the code located at the address specified in HvlpBelow1MbPage (0x2000)
In a previous rdx the structure was placed by hvLoaderBlock (offset +18h) address to the start of hvix64.exe
Later in IDA PRO you have to download hvix64.idb (similar to hvloader. efi), which works as follows:
- insert statement jmp $ (EB FE) at the start of the procedure start in hvix64. exe;
- completion debugging of winload.efi through the Debugger->the Detach from process;
- file download hvix64.efi in IDA PRO;
- connection to the gdb debugger vmware;
- restore the changed bytes to the original;
- performing the operation Edit -> Segment -> Rebase program indicating an Image Base 0xFFFFF800 00000000 + value, which was issued by the debugger in the Rebase Hv by: 6c25000.
Next quite a number of different operations as to be done in preparation for the execution of vmxon instruction:
Then vmptrld, subsequent filling VMCS with necessary values and in the last instance it will start vmlaunch.
After vmlaunch gets into HvlpReturnFromHypervisor while debugging via GDB we will see that after the first instruction cpuid, calling VM exit, the transition is made directly to the HOST_RIP.
After returning from the procedure, HvlpReturnFromHypervisor passes control to the next instruction after HvlpTransferToHypervisor.
at the end of the function HvlpLaunchHypervisor starts the kernel Windows through OslArchTransferToKernel.
If the Debugger is connected to the hypervisor, we can observe the following output (for the virtual system Windows Server 2012 checked build with two processors, each consists of two cores).
[0] Hypervisor initialized.
[0] Root Vp created.
MTRR map: number of ranges = 6 (default=UC)
Base=0x0000000000000000, Size=0x00000000000a0000, Type=WB, Synth=0
Base=0x00000000000a0000, Size=0x0000000000020000, Type=UC, Synth=0
Base=0x00000000000c0000, Size=0x000000000000c000, Type=WP, Synth=0
Base=0x00000000000cc000, Size=0x0000000000024000, Type=UC, Synth=0
Base=0x00000000000f0000, Size=0x0000000000010000, Type=WP, Synth=0
Base=0x0000000000100000, Size=0x00000000bff00000, Type=WB, Synth=0
----------------------
[0] Root Vp started.
[1] Root Vp created.
[1] Root Vp started.
[2] Root Vp created.
[2] Root Vp started.
[3] Root Vp created.
[3] Root Vp started.
MTRR map: number of ranges = 6 (default=UC)
Base=0x0000000000000000, Size=0x00000000000a0000, Type=WB, Synth=0
Base=0x00000000000a0000, Size=0x0000000000020000, Type=UC, Synth=0
Base=0x00000000000c0000, Size=0x000000000000c000, Type=WP, Synth=0
Base=0x00000000000cc000, Size=0x0000000000024000, Type=UC, Synth=0
Base=0x00000000000f0000, Size=0x0000000000010000, Type=WP, Synth=0
Base=0x0000000000100000, Size=0x00000000bff00000, Type=WB, Synth=0
It is worth mentioning that the process of loading a hypervisor in Windows Server 2012 (and higher) differs significantly from Windows Server 2008 R2, where the preparation and launch of the hypervisor directly produced by the hvboot.sys that run after loading the kernel Windows. This activation of the hypervisor instruction vmlaunch performed in the driver hvboot.sys and the next VM exit was processed in the hvix64.exe.
Find symbol information
When loading hvix64.exe in IDA PRO we get about three thousand functions with names like sub_FFFFF8000XXXXX because Microsoft, unfortunately, does not provide the symbol information for the hypervisor. Facilitate the research of the hypervisor can first try to identify some of the functions without detailed study.
In the first place it is worth using bindiff (or diaphora) to compare the files hvix64, hvloader and winload where symbol information are provided. Comparison shows that the networking function (e1000), USB, cryptography and some other features are exactly the same as the ones that are present in winload.exe (in Windows Server 2016 debugging functions have been moved to a separate module). This will help set the appointment of 500 functions. The same bindiff allows you to move the names of matching functions from one database to another idb. However, this method should be taken with caution and do not move all fully matched functions. At least the result should be analyzed by Visual comparison graph matching functions (Ctrl + E).
Next, let's define exception/interrupt functions, which are standard for processor architecture x86. A little script is written in python (ParseIDT.py) to parse the IDT, which must be run in IDA PRO, beeing connected through a debugging module of WinDBG to the hypervisor.
In the case of ISR was not found, check the tab List of problems in IDA PRO, since these procedures can not be found in the automatic analysis code that IDA performs.
Next, you can define the exit procedure in VM after reading field values VMCS. This can be done after the procedure fill the VMCS at hvix64.exe or use this script display-vmcs.py, which in the context of the hypervisor reads all fields VMCS and prints their values.
Hypercall
Microsoft released document Hypervisor Top-Level Functional Specification: Windows Server 2012 R2 (6), describes the architecture of Hyper-V 4.0. Hypervisor Top-Level Functional Specification for Windows Server 2016 has not yet been published.
Each virtual machine, as well as directly with the OS component installed Hyper-V is presented in terms of the partition (partition). Each section has its own identifier that must be unique to the host server.
For each section are given privileges to create (structure HV_PARTITION_PRIVILEGE_MASK), which determine the ability to perform specific hypercall.
Learn privileges by executing in the root-partition the following code in ring0:
WinHvGetPartitionId(&PartID);//PartID – ID section
WinHvGetPartitionProperty(PartID,HvPartitionPropertyPrivilegeFlags,&HvProp);// the result is returned in HvProp.
HvPartitionPropertyPrivilegeFlags – one of the enumeration values
HV_PARTITION_PROPERTY_CODE, which operate functions exported driver winhv.sys.
HV_STATUS
WinHvGetPartitionProperty(
__in HV_PARTITION_ID PartitionId,
__in HV_PARTITION_PROPERTY_CODE PropertyCode,
__out PHV_PARTITION_PROPERTY PropertyValue
);
Also, if necessary, these privileges can be changed, causing root-partition in the following function:
HV_STATUS
WinHvSetPartitionProperty(
__in HV_PARTITION_ID PartitionId,
__in HV_PARTITION_PROPERTY_CODE PropertyCode,
__in HV_PARTITION_PROPERTY PropertyValue
);
The value of HvPartitionPropertyPrivilegeFlags for the root partition: 000039FF00001FFF
AccessVpRunTimeMsr
AccessPartitionReferenceCounter
AccessSynicMsrs
AccessSyntheticTimerMsrs
AccessApicMsrs
AccessHypercallMsrs
AccessVpIndex
AccessResetMsr
AccessStatsMsr
AccessPartitionReferenceTsc
|
AccessGuestIdleMsr
AccessFrequencyMsrs
AccessDebugMsrs
CreatePartitions
AccessPartitionId
AccessMemoryPool
AdjustMessageBuffers
PostMessages
SignalEvents
CreatePort
|
ConnectPort
AccessStats
Debugging
CpuManagement
ConfigureProfiler
|
The value of HvPartitionPropertyPrivilegeFlags for child partition 000008B000000E7F:
AccessVpRunTimeMsr
AccessPartitionReferenceCounter
AccessSynicMsrs
AccessSyntheticTimerMsrs
AccessApicMsrs
AccessHypercallMsrs
AccessVpIndex
|
AccessPartitionReferenceTsc
AccessGuestIdleMsr
AccessFrequencyMsrs
PostMessages
SignalEvents
ConnectPort
Debugging
|
In a Windows guest OS, privileges can be obtained by placing EAX 0x40000003 and following the instructions CPUID (in document Hypervisor Functional Specification top-level 4.0 a given interpretation of the results of the cpuid).
CPUID 40000003 called
EAX = 00000E7F (00001110 01111111)
Bit 0: VP Runtime (HV_X64_MSR_VP_RUNTIME)
Bit 1: Partition Reference Counter (HV_X64_MSR_TIME_REF_COUNT)
Bit 2: Basic SynIC MSRs (HV_X64_MSR_SCONTROL through HV_X64_MSR_EOM and HV_X64_MSR_SINT0 through HV_X64_MSR_SINT15)
Bit 3: Synthetic Timer MSRs (HV_X64_MSR_STIMER0_CONFIG through HV_X64_MSR_STIMER3_COUNT)
Bit 4: APIC access MSRs (HV_X64_MSR_EOI, HV_X64_MSR_ICR and HV_X64_MSR_TPR)
Bit 5: Hypercall MSRs (HV_X64_MSR_GUEST_OS_ID and HV_X64_MSR_HYPERCALL)
Bit 6: Access virtual processor index MSR (HV_X64_MSR_VP_INDEX)
EBX = 000008B0 (00001000 10110000)
Bit 4: PostMessages
Bit 5: SignalEvents
Bit 7: ConnectPort
Bit 11: Debugging
ECX = 00000002 (00000000 00000010)
Maximum Processor Power State is C2
EDX = 000007B2 (00000111 10110010)
Bit 1: Guest debugging support is available
Bit 4: Support for passing hypercall input parameter block via XMM registers is available
Bit 5: Support for a virtual guest idle state is available
In Windows 10 x86 as guest in Windows Server 2016 TP2 privileges in EBX were extended:
EBX = 003880B0 (1110001000000010110000)
Bit 4: PostMessages
Bit 5: SignalEvents
Bit 7: ConnectPort
Bit 15: Unknown
Bit 19: Unknown
Bit 20: Unknown
Bit 21: Unknown
Unknown privileges is not mentioned in TLFS 4.0
The hypervisor privileges section, which carried out the operation that caused the VM exit, can be obtained by calculating the value of gs: 0, read the value of the field in the VMCS HOST_GS_BASE or IA32_GS_BASE MSR:
WINDBG>rdmsr 0xc0000101
msr[c0000101] = fffff800`05464000
then get the value pointed to gs: 82e8, and go to the offset 0xd8.
WINDBG>dc poi(fffff800`05464000+82e8)+0xd8
00000080`04dd70d8 00001fff 000039ff 00000000 ffffe800 .....9..........
00000080`04dd70e8 00000001 00000000 00000000 00000000 ................
In this case, the VM exit was made from root-partition.
The hypervisor in each section forms a special page to run hypercall. Its address can be obtained by reading MSR 0x40000001 (HV_X64_MSR_HYPERCALL):
Windows 7 x86 on Windows Server 2012:
|
Windows 8 x86 on Windows Server 2016 TP2:
|
kd> rdmsr 0x40000001
msr[40000001] = 00000000`1ffb1001
kd> !dc 00000000`1ffb1001
#1ffb1000 c3c1010f 90909090 90909090 90909090 ................
#1ffb1010 90909090 90909090 90909090 90909090 ................
As you can see, 0xc3c1010f - instructs opcodes to vmcall; ret
|
kd> rdmsr 0x40000001
msr[40000001] = 00000000`00004001
kd> !dc 4000
# 4000 c3c1010f 11b8c88b 0f000000 48c3c101
# 4010 c748c18b 000011c1 c1010f00 b8c88bc3
# 4020 00000012 c3c1010f 48c18b48 0012c1c7
# 4030 010f0000 9090c3c1 90909090 90909090
kd> up 4000 L50
00004000 0f01c1 vmcall
00004003 c3 ret
00004004 8bc8 mov ecx,eax
00004006 b811000000 mov eax,11h
0000400b 0f01c1 vmcall
0000400e c3 ret
0000400f 48 dec eax
00004010 8bc1 mov eax,ecx
00004012 48 dec eax
00004013 c7c111000000 mov ecx,11h
00004019 0f01c1 vmcall
0000401c c3 ret
0000401d 8bc8 mov ecx,eax
0000401f b812000000 mov eax,12h
00004024 0f01c1 vmcall
00004027 c3 ret
00004028 48 dec eax
00004029 8bc1 mov eax,ecx
0000402b 48 dec eax
0000402c c7c112000000 mov ecx,12h
00004032 0f01c1 vmcall
00004035 c3 ret
00004036 90 nop
|
Windows Server 2012 following changes took place in the export of the driver winhv.sys in comparison with the Windows Server 2008 R2:
Added
|
Removed
|
WinHvAddLogicalProcessor
WinHvAttachDevice
WinHvDetachDevice
WinHvGetLogicalProcessorProperty
WinHvGetLogicalProcessorRegisters
WinHvGetNextQueuedPort
WinHvGetSystemInformation
WinHvInjectSyntheticMachineCheckEvent
WinHvMapDeviceInterrupt
WinHvPrepareForSleep
WinHvProcessorIndexToLpIndex
WinHvProcessorNumberToVpIndex
WinHvRemoveLogicalProcessor
WinHvSetLogicalProcessorProperty
WinHvSetLogicalProcessorRegisters
WinHvUnmapDeviceInterrupt
|
WinHvOnInterrupt
WinHvReclaimInterruptVector
WinHvSupplyInterruptVector
|
Export Winhv.sys in client versions of Windows (as you see – many function were removed from Windows 10 winhv.sys):
Windows 8.1 x86
|
Windows 10 x86
| |
WinHvAddLogicalProcessor
WinHvAllocateOverlayPages
WinHvAllocatePartitionSintIndex
WinHvAllocatePortId
WinHvAllocateSingleSintIndex
WinHvAssertVirtualInterrupt
WinHvAttachDevice
WinHvCancelTimer
WinHvClearVirtualInterrupt
WinHvConfigureProfiler
WinHvConnectPort
WinHvCreateEventLogBuffer
WinHvCreatePartition
WinHvCreatePort
WinHvCreateTimer
WinHvCreateVp
WinHvDeleteEventLogBuffer
WinHvDeletePartition
WinHvDeletePort
WinHvDeleteTimer
WinHvDeleteVp
WinHvDepositMemory
WinHvDetachDevice
WinHvDisconnectPort
WinHvFinalizeEventLogBufferGroup
WinHvFlushEventLogBuffer
WinHvFreeOverlayPages
WinHvFreePartitionSintIndex
WinHvFreePortId
WinHvFreeSingleSintIndex
WinHvGetCurrentVpIndex
WinHvGetLogicalProcessorProperty
WinHvGetLogicalProcessorRegisters
WinHvGetLogicalProcessorRunTime
WinHvGetMemoryBalance
WinHvGetNextChildPartition
WinHvGetNextQueuedPort
WinHvGetPartitionId
WinHvGetPartitionProperty
WinHvGetPortProperty
WinHvGetSintEventFlags
WinHvGetSintMessage
WinHvGetSystemInformation
WinHvGetVpRegisters
WinHvInitializeEventLogBufferGroup
WinHvInjectSyntheticMachineCheckEvent
WinHvInstallIntercept
WinHvLookupPortId
WinHvLowMemoryPolicyAutoDeposit
|
WinHvLowMemoryPolicyReturnStatus
WinHvMapDeviceInterrupt
WinHvMapEventLogBuffer
WinHvMapGpaPages
WinHvMapSparseGpaPages
WinHvMapStatsPage
WinHvModifySparseGpaPages
WinHvNtProcessorToVpIndex
WinHvPostMessage
WinHvPrepareForSleep
WinHvProcessorIndexToLpIndex
WinHvProcessorNumberToVpIndex
WinHvQueryInterceptIrql
WinHvQueryReferenceCounter
WinHvReadGpa
WinHvReleaseEventLogBuffer
WinHvRemoveLogicalProcessor
WinHvReportPresentHypervisor
WinHvRestorePartitionState
WinHvSavePartitionState
WinHvScrubPartition
WinHvSetAbsoluteTimer
WinHvSetEndOfMessage
WinHvSetEventLogCompletedNotificationRoutine
WinHvSetEventLogGroupSources
WinHvSetLogicalProcessorProperty
WinHvSetLogicalProcessorRegisters
WinHvSetPartitionProperty
WinHvSetPortProperty
WinHvSetSint
WinHvSetSintOnCurrentProcessor
WinHvSetStackwalkEvents
WinHvSetVpRegisters
WinHvSignalEvent
WinHvTranslateVirtualAddress
WinHvUnmapDeviceInterrupt
WinHvUnmapEventLogBuffer
WinHvUnmapGpaPages
WinHvUnmapStatsPage
WinHvWithdrawAllMemory
WinHvWithdrawMemory
WinHvWriteGpa
|
WinHvAllocateOverlayPages
WinHvDisablePartitionVtl
WinHvDisableVpVtl
WinHvEnablePartitionVtl
WinHvEnableVpVtl
WinHvFreeOverlayPages
WinHvGetCurrentVpIndex
WinHvGetSintEventFlags
WinHvGetSintMessage
WinHvGetVpRegisters
WinHvNtProcessorToVpIndex
WinHvPostMessage
WinHvProcessorNumberToVpIndex
WinHvSetEndOfMessage
WinHvSetSint
WinHvSetSintOnCurrentProcessor
WinHvSetVpRegisters
WinHvSignalEvent
|
In order to be able to use the export function winhv.sys can either dynamically calculate the addresses of the functions (7), or to create a lib-file (8). Consider the second option.
When you declare functions like stdcall (32-bit version of the driver) in the def-file, you must specify the ordinals of the functions or when loading the driver the imported functions will not be found (for some reason, the table import function hyperv3.sys driver gets a postfix @ number, even if the def-file register WinHvGetPartitionProperty @ 16 = WinHvGetPartitionProperty):
WinHvGetPartitionProperty@16 @42
To create a def-file using the output of dumpbin:
dumpbin /exports winhv.sys
(The Windows Server 2016 TP2 is using a winhvr.sys driver root-section, so the def-file for the driver in the OS is necessary to form it).
To build a 64-bit driver you do not need to make any changes.
After editing the def-file it must be re-form the lib-file with the command (for x86):
lib.exe /def:winhv.def /OUT:winhv.lib /machine:x86
For x64 (performed 1 time for a specific version winhv.sys):
lib.exe /def:winhv64.def /OUT:winhv64.lib /machine:x64
(of course, run native tools command prompt before execute this)
Let's try it in a loop from 0 to 0x100 consistently meet Hypercall 0x41 (HvInitializePartition), with the PartitionID in ECX, equal to the value of the loop iterator, with Fast bit (to pass parameters through the registers.) with EAX returns the output of the hypervisor.
for (i = 0x0; i <=0x100; i++)
{
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DBG_PRINT_LEVEL,"i %x VMCALL_EAX %x",i,ARCH_VMCALL_REG_MOD(i));
}
ARCH_VMCALL_REG_MOD PROC param1:DWORD
push esi
push edi
push ebx
xor edx,edx
mov ecx, param1
xor ebx,ebx
xor esi,esi
xor edi,edi
mov eax, 10041h
vmcall
pop ebx
pop edi
pop esi
ret
ARCH_VMCALL_REG_MOD ENDP
As a result (for Windows Server 2012), we obtain
In case if in the ecx was transferred to the active virtual machine PartitionID, the hypervisor returns 6 (HV_STATUS_ACCESS_DENIED), in other cases - d (HV_STATUS_INVALID_PARTITION_ID). Taking advantage of this fact, and the fact that the ID of each new section is calculated by simple adding 1 to the ID of the previous section, and the ID root-partition is always equal to 1, you can set the number of active virtual machines on the host. To do this, slightly modify the code for the driver:
for (i = 0x2; i <=0x10000; i++)
{
res = ARCH_VMCALL_REG_MOD(i);
if (res == HV_STATUS_INVALID_PARTITION_ID){
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DBG_PRINT_LEVEL,"PartitionID %x VMCALL_EAX %x \n",i,res);
}
}
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DBG_PRINT_LEVEL,"Number of active virtual machines: %x \n",counter);
and get a list of active sections ID and number:
The number of loop iterations must be greater than the number of running VMs + number of overloaded since the start VM hypervisor. After restarting the hypervisor numbering of all sections begins again.
These data are available for the following two reasons:
- The section PartitionID generated by simply adding 1 to the last used PartitionID.
- When processing a hypercall the hypervisor first checks the validity of the transferred PartitionID and just in case whats the referred PartitionID active partition, it checks the rights to perform hypercall.
This feature hypervisor can be used to determine the number of virtual machines running on a given host server. For the name of the host server, you can peek in the registry of the guest OS under HKLM \ Software \ Microsoft \ Virtual Machine \ Guest \ Parameter, which contains data on the host operating system, transmitted by Key Value Pair Integration Component, which is normally enabled by default. Also controlled restarting the virtual machine on the second Monday of the month and secure it PartitionID (there is quite a high probability that he will be the last in the list of active VM), you can determine whether a virtual neighbors on their servers coming out every second Tuesday security fixes. However, the reality is quite difficult to imagine that someone will need this information ...
This hypervisor behavior could be observed in the assembly 6.3.9431.0 (Windows Server 2012 R2 Preview), but Microsoft recognized this behavior as "unexpected behavior" and eliminated him in the assembly 6.3.9600.16384 ". The TLFS changes were made to allow for the enforcement of such hypercall behavior only from root-partition.
The Statement which is processing vmcall in the hypervisor runs roughly as follows:
- check ring protection in which the statement has been issued, if the statement was executed in ring 3, then processing stops;
- if the instruction is executed in ring0, it checks, whether at the same processor LongMode.
- depending on the operating mode of the processor to perform two different procedures, the logic is quite similar;
- each procedure loads a pointer to an array of structures that contain the parameters necessary for processing each of hypercall 0 to 8C (decryption codes listed in hypercall Hypervisor Top-Level Functional Specification: Windows Server 2012 R2. Appendix B: Hypercall Code Reference). One of the elements of each structure is a pointer to a procedure for processing hypercall:
- then there is a check which way the hypervisor have been transferred parameters through memory or through the registers (in this case, the fast call bit in EAX before hypercall should equal 1).
- then call the corresponding function.
For comparison, some of the important fields VMCS were obtained using the script display_vmcs.py after VM exit:
Root partition
|
Child partition
|
CPU_BASED_VM_EXEC_CONTROL = 0xb6206dfa
Use TSC offsetting
HLT exiting
MWAIT exiting
RDPMC exiting
Use TPR shadow
Use I/O bitmaps
Use MSR bitmaps
MONITOR exiting
Activate secondary controls
IO_BITMAP_A = 0x4e06000
IO_BITMAP_A_HIGH = 0x0
IO_BITMAP_B = 0x4e07000
IO_BITMAP_B_HIGH = 0x0
EXCEPTION_BITMAP = 0x40000
MSR_BITMAP = 0x4e08000
MSR_BITMAP_HIGH = 0x0
PIN_BASED_VM_EXEC_CONTROL = 0x1f
External-interrupt exiting
NMI exiting
SECONDARY_VM_EXEC_CONTROL = 0x2a
Enable EPT
Enable RDTSCP
Enable VPID
VM_ENTRY_CONTROLS = 0x13ff
Load debug controls
IA-32e mode guest
VM_EXIT_CONTROLS = 0x3efff
Save debug controls
Host address space size
Acknowledge interrupt on exit
|
CPU_BASED_VM_EXEC_CONTROL = 0xb5a06dfa
Use TSC offsetting
HLT exiting
MWAIT exiting
RDPMC exiting
Use TPR shadow
MOV-DR exiting
Unconditional I/O exiting
Use MSR bitmaps
MONITOR exiting
Activate secondary controls
CR0_GUEST_HOST_MASK = 0xffffffe1
CR0_READ_SHADOW = 0x8001003b
CR4_GUEST_HOST_MASK = 0xfffff874
CR4_READ_SHADOW = 0x406f8
EXCEPTION_BITMAP = 0x40000
GUEST_CR0 = 0x8001003b
GUEST_CR3 = 0x185000
GUEST_CR4 = 0x426f9
GUEST_RIP = 0x839b1000
GUEST_RSP = 0x8870f8a4
HOST_CR0 = 0x80010031
PIN_BASED_VM_EXEC_CONTROL = 0x1f
External-interrupt exiting
NMI exiting
SECONDARY_VM_EXEC_CONTROL = 0x62
Enable EPT
Enable VPID
WBINVD exiting
VM_ENTRY_CONTROLS = 0x11ff
Load debug controls
VM_EXIT_CONTROLS = 0x3efff
Save debug controls
Host address space size
Acknowledge interrupt on exit
|
For instance, you can see, that for guest-partition the hypervisor handles all input/output (I/O exiting Unconditional), and for the root partition monitors only certain ports (Use I/O bitmaps).
WINDBG>!dc 0x4e06000 L250 - IO_BITMAP_A
# 4e06000 00000000 00000003 00000000 00000010 ................
# 4e06010 00000000 00000003 00000000 00000000 ................
# 4e06020 00000000 00000000 00000000 00000000 ................
…………………………………………………………………………………………………
# 4e06190 00000000 00000000 00000000 f1000000 ................
If I am not mistaken in the calculations, then the root-partition monitored ports are 20h, 21h, 44h, A0h, A1h, 1D5Fh, 1D64h, 1D65h, 1D66h, 1D67h.
closing
The article describes the steps that must be done to create a stand for the research of Hyper-V, and very briefly describes some aspects of the work of the hypervisor. I hope this information is useful for beginners in hypervisor security researcher at Microsoft.
Files: https://drive.google.com/file/d/0B8WEjIxRncDRX1RHc0FQMmozOTgSources:
- The NT Insider (July-Aug 2015)
