The title is shorten from "How to use MkLink to solve the problem that the EXE file built by MASM611 cannot be workable at the directory named with 8.3 format in Windows environment."
Sometimes we use MASM611 to write Assembly code in our Windows 7+ environment. If your name of source code directory is long (non 8.3 format), it will be a problem. For example,
Below is the content of Hellow.asm.
Before compiling the ASM with MASM611, please set the MASM611 path as follows.
>set path=%path%;c:\MASM611\BIN;c:\MASM611\BINR
Then please run ml to compile Hellow.asm. (Sometimes the system hangs because the you run it in the long directory.)
The HELLO.EXE is generated. Please run it.
Unfortunately, it cannot be workable. I guess that it is caused by the EXE file built by MASM611 cannot be workable at the long directory. How do you keep the long directory and make it workable? You can 1) copy the EXE file to the root directory and run it, or 2) use MkLink to create a link directory, Hello, with 8.3 format.
>mklink /d "D:\Test\Hello" "D:\My Personal Data\範例程式\Hello World"
The link Hello is created in the Test directory.
Then you can run HELLO.exe in the link directory.
Finally you can edit the Hello.asm, compile it, and run it again in the link directory. All of your changes will impact the original directory.
-Count
A blog of computer science that covers cryptography, UEFI BIOS, programming language, software applications, and network communication.
Friday, September 19, 2014
Use MkLink to Solve the MASM611 EXE at 8.3 Directory Name
Tuesday, September 16, 2014
Install XBMC in Raspberry Pi
"Raspberry Pi is a credit-card sized computer that plugs into you TV and a keyboard."
http://www.raspberrypi.org
"XBMC is an award-winning free and open source (GPL) software media player, ...."
http://xbmc.org
Yesterday I installed XBMC in my Raspberry Pi to make a portable media player. It was interesting. Below was my Raspberry Pi with SD card. The device was connected to Internet, a USB hub, a USB HDD, and a monitor via HDMI. The USB hub was connected to a keyboard and a wireless mouse. I selected wireless mouse to be a remote control of the media player.
Here was a USB HDD with 2TB capability that stores movies.
I installed Raspbmc in the SD card (SDHC 8GB Class2). "Raspbmc is a minimal Linux distribution based on Debian that brings XBMC to your Raspberry Pi."
http://www.raspbmc.com
The following steps were installation of Raspbmc.
Step 1. Format the SD card otherwise the Raspbmc cannot be installed successfully in step 3.
Step 2. Download Windows Installation of Raspbmc.
http://www.raspbmc.com/wiki/user/windows-installation
Step 3. After decompressing the downloaded file, please run installer.exe.
http://www.raspberrypi.org
"XBMC is an award-winning free and open source (GPL) software media player, ...."
http://xbmc.org
Yesterday I installed XBMC in my Raspberry Pi to make a portable media player. It was interesting. Below was my Raspberry Pi with SD card. The device was connected to Internet, a USB hub, a USB HDD, and a monitor via HDMI. The USB hub was connected to a keyboard and a wireless mouse. I selected wireless mouse to be a remote control of the media player.
Here was a USB HDD with 2TB capability that stores movies.
I installed Raspbmc in the SD card (SDHC 8GB Class2). "Raspbmc is a minimal Linux distribution based on Debian that brings XBMC to your Raspberry Pi."
http://www.raspbmc.com
The following steps were installation of Raspbmc.
Step 1. Format the SD card otherwise the Raspbmc cannot be installed successfully in step 3.
Step 2. Download Windows Installation of Raspbmc.
http://www.raspbmc.com/wiki/user/windows-installation
Step 3. After decompressing the downloaded file, please run installer.exe.
Step 4. Check if Raspbmc installer is installed. Please open the SD card that should contain the following files.
Step 5. Connect Raspberry Pi with Internet because the installer will load files from Internet. Insert the SD card with the installer into the Raspberry Pi and turn on the device. The following pictures in the screen display in progress of installation.
Step 6. Now you have XBMC in Raspberry Pi. You can add movie information from web (e.g., IMDB) into your movies in USB HDD.
Saturday, September 13, 2014
Use Bochs to Lean BIOS Entry Point
It's Sunday in autumn. Really nice weather with wind, not hot. I can see sun, white cloud, and a small far island on the sea out the window. I want to be outdoor to enjoy the weather, but there is a more interesting thing I cannot bear to skip it. That is to use Bochs to trace BIOS entry point. You should be interested in the task if you are/will be/was a BIOS engineer.
Bochs is a PC emulator. I've introduced it in my blog page.
http://countchu.blogspot.tw/2014/09/bochs-pc-emulator.html
Bochs reads a file, BIOS-bochs-latest file, to be a BIOS of the PC emulator. We know that CPU considers the address f000:fff0 as a BIOS entry point when the PC is power on. So we open the BIOS file to get the last line.
We can find that there are machine codes and a date signature, 08/02/13, that I guess as the released date of the BIOS. What do the machine codes mean? Le's use debug command in prompt.
> copy BIOS-bochs-latest test.bin
> debug test.bin
Bochs is a PC emulator. I've introduced it in my blog page.
http://countchu.blogspot.tw/2014/09/bochs-pc-emulator.html
Bochs reads a file, BIOS-bochs-latest file, to be a BIOS of the PC emulator. We know that CPU considers the address f000:fff0 as a BIOS entry point when the PC is power on. So we open the BIOS file to get the last line.
We can find that there are machine codes and a date signature, 08/02/13, that I guess as the released date of the BIOS. What do the machine codes mean? Le's use debug command in prompt.
> copy BIOS-bochs-latest test.bin
> debug test.bin
So the the assembly language of the machine codes is JMP F000:E05B. Le's show the machine codes at F000:E05B.
The code started at XOR is POST Entry Point. Please refer the link to know what it is.
Above is a way to manually trace the BIOS entry point. How to use Bochs to easily trace it? Just run bochsdbg.exe, a debug mode of Bochs.
> bochsdbg.exe
It is just like the DOS debug command, it is broken at the line of <bochs:1> to wait for user command. I send the help command to display how to use it. Then I select the n command to trace BIOS entry point step by step. It displays in assembly language and teaches us how BIOS entry point works. Is it interesting?
-Count
Bochs - A PC Emulator
Bochs is a PC emulator. What is my expectation to Bochs? I want to write a simplest BIOS and burn it in the emulator without breaking my real PC. I'm too excited to read Bochs document. Let's download it, run it, and try to change the default BIOS of the emulator.
Here is the Bochs official web page.
http://bochs.sourceforge.net/
Please download Bochs-2.6.6.exe in the web page in your Windows environment and install it. Then, you can find Bochs 2.6.6 in your Windows start menu.
No doubt, just click the Start button in the dialog to turn on the PC. I'll explain the Load button later. You can see another window that displays as a monitor of the PC. Another dialog is also popped up because a fatal error, no-bootable-device, occurs.
We can see a default BIOS, Bochs BIOS - build: 08/02/13, in the window. The question is how to replace the default BIOS? Do you remember the Load button? If you click it, a dialog, Load Bochs Config File, displays. It loads a *.bxrc file that you specify. You should be interested in the file. Please search them.
C:\Program Files\Bochs-2.6.6>dir *.bxrc /s
Directory of C:\Program Files\Bochs-2.6.6\dlxlinux
2014/09/13 ?? 10:08 1,533 bochsrc.bxrc
You can find the bochsrc.bxrc. Please open it and search BIOS or rom keyword.
# filename of ROM images
romimage: file=../BIOS-bochs-latest
vgaromimage: file=../VGABIOS-lgpl-latest
You can change the romimage config to specify your BIOS.
Another way is to run the following command in the prompt.
C:\Program Files\Bochs-2.6.6>bochs -f dlxlinux\bochsrc.bxrc -q
What does the command mean? Please see the help text by running "bochs -h".
C:\Program Files\Bochs-2.6.6>bochs -h
The below page shows you how to write the simplest BIOS and burn it with Bochs.
http://countchu.blogspot.tw/2014/10/write-your-half-bios.html
-Count
Here is the Bochs official web page.
http://bochs.sourceforge.net/
Please download Bochs-2.6.6.exe in the web page in your Windows environment and install it. Then, you can find Bochs 2.6.6 in your Windows start menu.
Just click Bochs 2.6.6 icon to start the PC emulator. You can see a window and a dialog.
No doubt, just click the Start button in the dialog to turn on the PC. I'll explain the Load button later. You can see another window that displays as a monitor of the PC. Another dialog is also popped up because a fatal error, no-bootable-device, occurs.
We can see a default BIOS, Bochs BIOS - build: 08/02/13, in the window. The question is how to replace the default BIOS? Do you remember the Load button? If you click it, a dialog, Load Bochs Config File, displays. It loads a *.bxrc file that you specify. You should be interested in the file. Please search them.
C:\Program Files\Bochs-2.6.6>dir *.bxrc /s
Directory of C:\Program Files\Bochs-2.6.6\dlxlinux
2014/09/13 ?? 10:08 1,533 bochsrc.bxrc
You can find the bochsrc.bxrc. Please open it and search BIOS or rom keyword.
# filename of ROM images
romimage: file=../BIOS-bochs-latest
vgaromimage: file=../VGABIOS-lgpl-latest
You can change the romimage config to specify your BIOS.
Another way is to run the following command in the prompt.
C:\Program Files\Bochs-2.6.6>bochs -f dlxlinux\bochsrc.bxrc -q
What does the command mean? Please see the help text by running "bochs -h".
C:\Program Files\Bochs-2.6.6>bochs -h
The below page shows you how to write the simplest BIOS and burn it with Bochs.
http://countchu.blogspot.tw/2014/10/write-your-half-bios.html
-Count
Thursday, September 11, 2014
Write Python Program to Use DBGHELP.DLL to Access a PDB File
The DBGHELP.DLL is a library provided by Microsoft to access PDB file. This page teaches us,
- How to use Python to load DLL.
- How to read PDB file to report file path and line number by a given IP register value.
- How to use python ctypes.
I select Python to try DBGHELP.DLL because Python is easy for use. Below is the source code. You can run it with the command.
> python DbgHelpTest.py
You can see the results.
DbgHelpTest.py
#
# This source code is free software; you can redistribute it and/or
# modify it under the terms of version 2.1 of the GNU Lesser General Public
# License as published by the Free Software Foundation.
#
# This Python program demonstrates the usage of Windows DBGHELP.DLL by the
# below steps.
#
# 1. Locate functions of DBGHELP.DLL.
# 2. Load a PDB file at base address.
# 3. Input IP register value.
# 4. Output file path and line number of source code.
#
# Copyright (c) 2014 Count Chu.
#
import os
import ctypes
#
# Please set parameters here.
#
PdbFileName = "VcApp.pdb" # A PDB file that was created by you.
BaseAddr = ctypes.c_uint64 (0x400000) # Base address of the PDB file that will be loaded.
IpAddr = ctypes.c_uint64 (0x4113B0) # IP register value that will be input.
#
# Locate functions of DBGHELP.DLL that are used in the program.
#
DbgHelp = ctypes.WinDLL (r"dbghelp.dll")
print DbgHelp # for debug.
SymInitialize = DbgHelp ["SymInitialize"]
SymSetOptions = DbgHelp ["SymSetOptions"]
SymLoadModule64 = DbgHelp ["SymLoadModule64"]
SymGetLineFromAddr64 = DbgHelp ["SymGetLineFromAddr64"]
SymUnloadModule64 = DbgHelp ["SymUnloadModule64"]
#
# Define enumeration that is used in SymSetOptions.
#
class SymOpt:
LOAD_LINES = 0x00000010
UNDNAME = 0x00000002
DEFERRED_LOADS = 0x00000004
#
# Call SymSetOptions().
#
print ("Call SymSetOptions()") # for debug.
SymSetOptions (SymOpt.LOAD_LINES | SymOpt.UNDNAME | SymOpt.DEFERRED_LOADS)
#
# Get the current process handle. It should be -1.
#
ProcessHandle = ctypes.windll.kernel32.GetCurrentProcess ()
print ("ProcessHandle = %xh" % ProcessHandle) # for debug.
#
# Call SymInitialize().
#
print ("Call SymInitialize()") # for debug.
Status = SymInitialize (ProcessHandle, None, False)
if not Status:
print "Error. Calling SymInitialize() failed."
#
# Call SymLoadModule64().
#
FileSize = os.path.getsize (PdbFileName)
print ("FileSize = %d" % FileSize) # for debug.
print ("Call SymLoadModule64()") # for debug.
OutBaseAddr = SymLoadModule64 ( # DWORD64 WINAPI SymLoadModule64 (
ProcessHandle, # _In_ HANDLE hProcess,
0, # _In_opt_ HANDLE hFile,
PdbFileName, # _In_opt_ PCSTR ImageName,
None, # _In_opt_ PCSTR ModuleName,
BaseAddr, # _In_ DWORD64 BaseOfDll,
FileSize) # _In_ DWORD SizeOfDll
# );
print ("OutBaseAddr = %xh" % OutBaseAddr) # for debug.
#
# Declare IMAGEHLP_LINE64 structure and create a instance of it.
#
class IMAGEHLP_LINE64 (ctypes.Structure): # typedef struct _IMAGEHLP_LINE64 {
_fields_ = [ #
('SizeOfStruct', ctypes.c_uint32), # DWORD SizeOfStruct;
('Key', ctypes.c_void_p), # PVOID Key;
('LineNumber', ctypes.c_uint32), # DWORD LineNumber;
('FileName', ctypes.c_char_p), # PTSTR FileName;
('Address', ctypes.c_uint64) # DWORD64 Address;
] # } IMAGEHLP_LINE64, *PIMAGEHLP_LINE64;
Line = IMAGEHLP_LINE64 () # Create a instance of IMAGEHLP_LINE64.
#
# Call SymGetLineFromAddr64()
#
print ("Call SymGetLineFromAddr64()") # for debug.
Displacement = ctypes.c_uint32 (0)
Status = SymGetLineFromAddr64 ( # BOOL WINAPI SymGetLineFromAddr64 (
ProcessHandle, # _In_ HANDLE hProcess,
IpAddr, # _In_ DWORD64 dwAddr,
ctypes.byref (Displacement), # _Out_ PDWORD pdwDisplacement,
ctypes.byref (Line)) # _Out_ PIMAGEHLP_LINE64 Line
# );
print ("Status = %xh" % Status) # for debug.
if not Status:
print "Error. Calling SymGetLineFromAddr64() failed."
#
# Dump Displacement and IMAGEHLP_LINE64.
#
print ("Displacement = %d" % Displacement.value)
print ("Dump Line:")
print (" SizeOfStruct = %d" % Line.SizeOfStruct)
print (" Key = %s" % Line.Key)
print (" LineNumber = %d" % Line.LineNumber)
print (" FileName = %s" % Line.FileName)
print (" Address = %xh" % Line.Address)
#
# Call SymUnloadModule64()
#
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)
if not Status:
print "Error. Calling SymUnloadModule64() failed."
The Life Cycle of a UEFI Variable in Flash Part
Below is the data structure of one UEFI Variable stored in variable store.
We want to know the life cycle of the UEFI Variable in flash part.
In UEFI spec, We can do the following operations for one non-volatile Variable by calling SetVariable().
0xfe = 1111-1110 // Variable is in obsolete transition
0xfd = 1111-1100 // Variable is obsolete.
0x7f = 0111-1111 // Variable header has been valid.
0x3f = 0011-1111 // Variable has been completely added.
Add Operation
The variable state is changed as follows.
0111-1111 (After Step 1)
0011-1111 (After Step 3)
Update Operation
The old variable state is changed as follows.
0011-1111 (Initial state)
0011-1110 (After Step 1)
0011-1100 (After Step 5)
The new variable state is changed as follows.
0111-1111 (After Step 2)
0011-1111 (After Step 4)
Delete Operation
0011-1111 (Initial State)
0011-1100 (After Step 1)
In Conclusion.
The states of the variable in flash part are as follows.
0111-1111 = 0x7f (Variable is in creating.)
0011-1111 = 0x3f (Variable is created.)
0011-1110 = 0x3e (Variable is in deleting.)
0011-1100 = 0x3d (Variable is in deleted.)
The state diagram (life cycle) of a variable in flash part are.
0x7f ==> 0x3f ==> 0x3e ==> 0x3d.
The diagram is on-way and no circle.
Please note that the value in the bit field is changed from 1 to 0. It cannot be changed from 0 to 1 in flash-write operation. The reason is the hardware characters of flash part. I'll try to explain it in another page.
-Count
We want to know the life cycle of the UEFI Variable in flash part.
In UEFI spec, We can do the following operations for one non-volatile Variable by calling SetVariable().
- Add Operation
- Update Operation
- Delete Operation
We use state diagram to explain the life cycle of a UEFI variable.
Added ==> Updated ==> Deleted.
There is a circle on the Updated state. It means the variable can be updated many times.
Physically, if the variable exists in flash part, we cannot directly update variable on the same region in variable store. Variable driver must create a new variable with the same Name and GUID in the available region and set an invalid flag in the old variable. If the variable store is about to full, Variable driver reclaims the variable store to cleanup invalid variables.
Therefore the life cycle of a variable in flash part should be Added ==> Deleted. How is EDKII Variable designed to support Add/Update/Delete operations on variables in flash part and to assure fault tolerant? Let's focus on the State field of VARIABLE_HEADER. We expand them as binary format.
0xfe = 1111-1110 // Variable is in obsolete transition
0xfd = 1111-1100 // Variable is obsolete.
0x7f = 0111-1111 // Variable header has been valid.
0x3f = 0011-1111 // Variable has been completely added.
Add Operation
- State = VAR_HEADER_VALID_ONLY (0x7f)
- Write Data
- State = VAR_ADDED (0x3f)
The variable state is changed as follows.
0111-1111 (After Step 1)
0011-1111 (After Step 3)
Update Operation
- Old State &= VAR_IN_DELETED_TRANSITION (0xfe)
- State = VAR_HEADER_VALID_ONLY (0x7f)
- Write Data
- State = VAR_ADDED (0x3f)
- Old State &= VAR_DELETED (0xfd)
The old variable state is changed as follows.
0011-1111 (Initial state)
0011-1110 (After Step 1)
0011-1100 (After Step 5)
The new variable state is changed as follows.
0111-1111 (After Step 2)
0011-1111 (After Step 4)
Delete Operation
- State &= VAR_DELETED (0xfd)
The state is directly to add VAR_DELETED flag. The variable state is changed as follows.
0011-1111 (Initial State)
0011-1100 (After Step 1)
In Conclusion.
The states of the variable in flash part are as follows.
0111-1111 = 0x7f (Variable is in creating.)
0011-1111 = 0x3f (Variable is created.)
0011-1110 = 0x3e (Variable is in deleting.)
0011-1100 = 0x3d (Variable is in deleted.)
The state diagram (life cycle) of a variable in flash part are.
0x7f ==> 0x3f ==> 0x3e ==> 0x3d.
The diagram is on-way and no circle.
Please note that the value in the bit field is changed from 1 to 0. It cannot be changed from 0 to 1 in flash-write operation. The reason is the hardware characters of flash part. I'll try to explain it in another page.
-Count
Wednesday, September 10, 2014
How to Tweak EDKII BaseTools
Sometimes BIOS engineers need to tweak EDKII BaseTools to add some enhancements. Most of EDKII build tools are developed in Python scripts that are frozen into executables (EXE files) by cx_freeze. The BaseTools directory contains Python source code. The path is, for example.
D:\EDKII\BaseTools\Source\Python
We can modify them and rebuilt them into executables. This page teach us how to do it. If you have modified Python scripts files in the BaseTools directory, please follow the following steps to build executables.
Step 1. Install Python 2.7.
Please make sure that if the Python 2.7 is installed. If not, please go to python website, download it, and install it. The installed path is, for example.
C:\Python27
Step 2. Install cx_freeze.
Please check if cx_freeze installs in Python 2.7. How do we check it? Just run dir.
C:\Python27\Scripts> dir *freeze*.*
If the cx_freeze has been installed. The dir command displays as follows.
68 cxfreeze
1,256 cxfreeze-postinstall
78 cxfreeze-quickstart
78 cxfreeze-quickstart.bat
67 cxfreeze.bat
If the cx_freeze has not been installed, please go to the page. http://cx-freeze.sourceforge.net
Please click the PyPI link in the page, download cx_Freeze-4.3.3.win32-py2.7.exe, and install it.
Step 3. Check if the cxfreeze.bat exists.
Sometimes the cxfreeze.bat is missed after we install cx_freeze. If the file doesn't exist in python script path, (e.g., C:\Python27\Scripts), please manually create it as follows.
cxfreeze.bat
@echo off
C:\Python27\python.exe C:\Python27\Scripts\cxfreeze %*
Step 4. Go to BaseTools directory.
For example,
D:\EDKII\BaseTools\Source\Python
Step 5. Check the Makefile file and change it if necessary.
Sometimes we use old BaseTools where Makefile uses FreezePython.exe not cxfreeze.bat. Please modify it as follows.
Makefile
# FREEZE=$(PYTHON_FREEZER_PATH)\FreezePython.exe
FREEZE = $(PYTHON_FREEZER_PATH)\cxfreeze.bat
Step 6. Set environment variables before building.
> set PYTHON_FREEZER_PATH=C:\Python27\Scripts
> set BASE_TOOLS_PATH=D:\EDKII\BaseTools
> set EDK_TOOLS_PATH=D:\EDKII\BaseTools
Step 7. Run "nmake all"
D:\EDKII\BaseTools\Source\Python> nmake all
You can find that the executables are generated in BaseTools Win32 path. For example,
D:\EDKII\BaseTools\Bin\Win32
D:\EDKII\BaseTools\Source\Python
We can modify them and rebuilt them into executables. This page teach us how to do it. If you have modified Python scripts files in the BaseTools directory, please follow the following steps to build executables.
Step 1. Install Python 2.7.
Please make sure that if the Python 2.7 is installed. If not, please go to python website, download it, and install it. The installed path is, for example.
C:\Python27
Step 2. Install cx_freeze.
Please check if cx_freeze installs in Python 2.7. How do we check it? Just run dir.
C:\Python27\Scripts> dir *freeze*.*
If the cx_freeze has been installed. The dir command displays as follows.
68 cxfreeze
1,256 cxfreeze-postinstall
78 cxfreeze-quickstart
78 cxfreeze-quickstart.bat
67 cxfreeze.bat
If the cx_freeze has not been installed, please go to the page. http://cx-freeze.sourceforge.net
Please click the PyPI link in the page, download cx_Freeze-4.3.3.win32-py2.7.exe, and install it.
Step 3. Check if the cxfreeze.bat exists.
Sometimes the cxfreeze.bat is missed after we install cx_freeze. If the file doesn't exist in python script path, (e.g., C:\Python27\Scripts), please manually create it as follows.
cxfreeze.bat
@echo off
C:\Python27\python.exe C:\Python27\Scripts\cxfreeze %*
Step 4. Go to BaseTools directory.
For example,
D:\EDKII\BaseTools\Source\Python
Step 5. Check the Makefile file and change it if necessary.
Sometimes we use old BaseTools where Makefile uses FreezePython.exe not cxfreeze.bat. Please modify it as follows.
Makefile
# FREEZE=$(PYTHON_FREEZER_PATH)\FreezePython.exe
FREEZE = $(PYTHON_FREEZER_PATH)\cxfreeze.bat
Step 6. Set environment variables before building.
> set PYTHON_FREEZER_PATH=C:\Python27\Scripts
> set BASE_TOOLS_PATH=D:\EDKII\BaseTools
> set EDK_TOOLS_PATH=D:\EDKII\BaseTools
Step 7. Run "nmake all"
D:\EDKII\BaseTools\Source\Python> nmake all
You can find that the executables are generated in BaseTools Win32 path. For example,
D:\EDKII\BaseTools\Bin\Win32
Wednesday, September 3, 2014
How does debugger work with PDB?
Debugger is a big topic but this page only describes how debugger works with PDB file.
As WIKI description, PDB is developed by Microsoft for storing debugging information about a program. Debugger consumes PDB files to know the line number and file path for a given symbol or memory address where the program is broken. Unfortunately, the PDB format is undocumented. We only access PDB via DIA (Debug Interface Access) or DBGHELP DLL library (dbghelp.dll). We just imagine that PDB stores the following information at least.
The most basic question is how to find file path and line number by a given memory address. The answer is very simple. Debugger just queries PDB by memory address to get the information.
Let's write a test program with VC to verify my assumption. The following is a simple program to print "Hello World!"
We build the program in debug mode. When debugging it, Visual Studio displays that the VcApp.exe is loaded at the Base Address, 00400000h.
When we move the cursor to the function decorations of main() and PrintMessage(), it displays their memory address are as follows.
PrintMessage = 004113B0h
main = 00411410h
We have the second question. How does debugger know the file path and line number of the PrintMessage. For the question, I suppose that there are two tables in PDB.
Table1 = (Symbol Name, Offset)
Table2 = (Offset, Line Number, File Path)
Let's manually create Table1.
PrintMessage - Base Address = 000113B0h
main - Base Address = 00011410h
Symbol Name Offset
------------ ---------
PrintMessage 000113B0h
main 00011410h
When we run pdb_print_gvars.py provided by PDBparse that was describe in another page, it reads PDB to display offsets of symbols as follows.
> python pdb_print_gvars.py VcApp.pdb 0
__enc$textbss$begin,0x1000,0,.textbss
__enc$textbss$end,0x11000,0,.textbss
_PrintMessage,0x113b0,2,.text
__imp__printf,0x182bc,0,.idata
__RTC_CheckEsp,0x114e0,2,.text
__RTC_Shutdown,0x11720,2,.text
__RTC_InitBase,0x116e0,2,.text
_main,0x11410,2,.text
??_C@_0M@FKNCOEJD@Hello?5Word?6?$AA@,0x1573c,0,.rdata
___security_cookie,0x17000,0,.data
It displays that the offset of PrintMessage is 000113B0h and main is 00011410h.
It proofs that Table1 really exists. How about Table2? Unfortunately, there are not examples of PDBparse to display file path and line number. I have not dug the information by reading PDB file yet. I decided to use dbghelp.dll to get file path and line number in PDB. I wrote a Python program to consume the DLL. The following are part of the program.
The whole source code was published in the page.
Write Python Program to Use DBGHELP.DLL to Access a PDB File.
...
BaseAddr = ctypes.c_uint64 (0x400000)
Status = SymLoadModule64 (ProcessHandle, 0, "VcApp.pdb", None, BaseAddr, SizeOfDll)
...
dwAddr = ctypes.c_uint64 (0x4113B0)
Status = SymGetLineFromAddr64 (ProcessHandle, dwAddr, ctypes.byref (pdwDisplacement), ctypes.byref (Line))
print ("Dump Line:")
print (" SizeOfStruct = %d" % Line.SizeOfStruct)
print (" Key = %s" % Line.Key)
print (" LineNumber = %d" % Line.LineNumber)
print (" FileName = %s" % Line.FileName)
print (" Address = %xh" % Line.Address)
...
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)
...
The result displays:
Dump Line:
SizeOfStruct = 0
Key = None
LineNumber = 7
FileName = d:\vcapp\vcapp.c
Address = 4113b0h
As the result, this program reports file path (d:\vcapp\vcapp.c) and line number (7) by the given memory address (0x4113B0) where the VcApp.exe is broken at the PrintMessage(). This program proofs that Table2 exists.
This program also help to solve the first question, how to find file path and line number by an given memory address? I consider that Table1 and Table2 or the merged table fulfil the solution of the first question.
I've explains how debugger works with PDB file.
As WIKI description, PDB is developed by Microsoft for storing debugging information about a program. Debugger consumes PDB files to know the line number and file path for a given symbol or memory address where the program is broken. Unfortunately, the PDB format is undocumented. We only access PDB via DIA (Debug Interface Access) or DBGHELP DLL library (dbghelp.dll). We just imagine that PDB stores the following information at least.
- Line Number
- File Path
- Offset (of Base Address)
- Symbol Name
The most basic question is how to find file path and line number by a given memory address. The answer is very simple. Debugger just queries PDB by memory address to get the information.
Let's write a test program with VC to verify my assumption. The following is a simple program to print "Hello World!"
We build the program in debug mode. When debugging it, Visual Studio displays that the VcApp.exe is loaded at the Base Address, 00400000h.
When we move the cursor to the function decorations of main() and PrintMessage(), it displays their memory address are as follows.
PrintMessage = 004113B0h
main = 00411410h
We have the second question. How does debugger know the file path and line number of the PrintMessage. For the question, I suppose that there are two tables in PDB.
Table1 = (Symbol Name, Offset)
Table2 = (Offset, Line Number, File Path)
Let's manually create Table1.
PrintMessage - Base Address = 000113B0h
main - Base Address = 00011410h
Symbol Name Offset
------------ ---------
PrintMessage 000113B0h
main 00011410h
When we run pdb_print_gvars.py provided by PDBparse that was describe in another page, it reads PDB to display offsets of symbols as follows.
> python pdb_print_gvars.py VcApp.pdb 0
__enc$textbss$begin,0x1000,0,.textbss
__enc$textbss$end,0x11000,0,.textbss
_PrintMessage,0x113b0,2,.text
__imp__printf,0x182bc,0,.idata
__RTC_CheckEsp,0x114e0,2,.text
__RTC_Shutdown,0x11720,2,.text
__RTC_InitBase,0x116e0,2,.text
_main,0x11410,2,.text
??_C@_0M@FKNCOEJD@Hello?5Word?6?$AA@,0x1573c,0,.rdata
___security_cookie,0x17000,0,.data
It displays that the offset of PrintMessage is 000113B0h and main is 00011410h.
It proofs that Table1 really exists. How about Table2? Unfortunately, there are not examples of PDBparse to display file path and line number. I have not dug the information by reading PDB file yet. I decided to use dbghelp.dll to get file path and line number in PDB. I wrote a Python program to consume the DLL. The following are part of the program.
The whole source code was published in the page.
Write Python Program to Use DBGHELP.DLL to Access a PDB File.
...
BaseAddr = ctypes.c_uint64 (0x400000)
Status = SymLoadModule64 (ProcessHandle, 0, "VcApp.pdb", None, BaseAddr, SizeOfDll)
...
dwAddr = ctypes.c_uint64 (0x4113B0)
Status = SymGetLineFromAddr64 (ProcessHandle, dwAddr, ctypes.byref (pdwDisplacement), ctypes.byref (Line))
print ("Dump Line:")
print (" SizeOfStruct = %d" % Line.SizeOfStruct)
print (" Key = %s" % Line.Key)
print (" LineNumber = %d" % Line.LineNumber)
print (" FileName = %s" % Line.FileName)
print (" Address = %xh" % Line.Address)
...
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)
...
The result displays:
Dump Line:
SizeOfStruct = 0
Key = None
LineNumber = 7
FileName = d:\vcapp\vcapp.c
Address = 4113b0h
As the result, this program reports file path (d:\vcapp\vcapp.c) and line number (7) by the given memory address (0x4113B0) where the VcApp.exe is broken at the PrintMessage(). This program proofs that Table2 exists.
This program also help to solve the first question, how to find file path and line number by an given memory address? I consider that Table1 and Table2 or the merged table fulfil the solution of the first question.
I've explains how debugger works with PDB file.
Tuesday, September 2, 2014
Use Python PDBparse to parse PDB files.
PDBparse is a Python package for parsing Microsoft PDB files. I like Python solution for parsing PDB files because it is open source. By running its examples, I easily understand PDB format without reading PDB spec initially if there is really a PDB spec.
This page just describes how to install the package and how to run its examples.
Below is the website of PDBparse.
https://code.google.com/p/pdbparse/
We can download the source code here.
http://pdbparse.googlecode.com/svn/trunk
So please check out it and put it in a directory. For example, D:\pdbparse. Let's build it and install it.
D:\pdbparse>python setup.py build
running build
running build_py
running build_ext
building 'pdbparse._undname' extension
error: Unable to find vcvarsall.bat
If the error occurs, please don't give up. When running setup.py, Python 2.7 searches for Visual Studio 2008. If we don't have the version or another version of Visual Studio, the error occurs. We can trick Python by setting an environment variable. For example, environment has Visual Studio 2005.
>set VS90COMNTOOLS=%VS80COMNTOOLS%
Let's build the package again.
D:\pdbparse>python setup.py build
LINK : error LNK2001: unresolved external symbol init_undname
build\temp.win32-2.7\Release\src\_undname.lib : fatal error LNK1120: 1 unresolved externals
error: command '"C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe"' failed with exit status 1120
If the error occurs, we should check if the init_undname is defined in C files. There is only one C file, undname.c, in the source files of package. The file doesn't define init_undname but has undname() routine.
char *undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
{
return __unDName(buffer, mangled, buflen, malloc, free, flags);
}
Therefore my solution is to create init_undname() to wrap undname(). Please add the following init_undname() in undname.c.
char *init_undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
{
return undname (buffer, mangled, buflen, flags);
}
Let's try it again. It should be successful.
D:\pdbparse>python setup.py build
Let's install the package in Python environment.
D:\pdbparse>python setup.py install
Now we can run the examples of the package. It should be run successfully
D:\pdbparse\examples\python pdb_dump.py Test.pdb
Please run another example.
python pdb_get_syscall_table.py Test.pdb
File "C:\Python27\lib\site-packages\pdbparse\info.py", line 1, in <module>
from construct import *
ImportError: No module named construct
If the ImportError occurs, please download the contruct package here.
http://construct.readthedocs.org/en/latest/
Download the source code of construct package and put it in a directory. For example, D:\construct-2.5.2. Please build it and install it.
D:\construct>python setup.py build
D:\construct>python setup.py install
Please run the example again
python pdb_get_syscall_table.py Test.pdb
File "C:\Python27\lib\site-packages\construct\lib\binary.py", line 1, in <module>
import six
ImportError: No module named six
The error occurs because construct package depends on six package and we miss it. Please download it, build it, and install it.
https://pypi.python.org/pypi/six/1.7.3
D:\six-1.7.3>python setup.py build
D:\six-1.7.3>python setup.py install
This page just describes how to install the package and how to run its examples.
Below is the website of PDBparse.
https://code.google.com/p/pdbparse/
We can download the source code here.
http://pdbparse.googlecode.com/svn/trunk
So please check out it and put it in a directory. For example, D:\pdbparse. Let's build it and install it.
D:\pdbparse>python setup.py build
running build
running build_py
running build_ext
building 'pdbparse._undname' extension
error: Unable to find vcvarsall.bat
If the error occurs, please don't give up. When running setup.py, Python 2.7 searches for Visual Studio 2008. If we don't have the version or another version of Visual Studio, the error occurs. We can trick Python by setting an environment variable. For example, environment has Visual Studio 2005.
>set VS90COMNTOOLS=%VS80COMNTOOLS%
Let's build the package again.
D:\pdbparse>python setup.py build
LINK : error LNK2001: unresolved external symbol init_undname
build\temp.win32-2.7\Release\src\_undname.lib : fatal error LNK1120: 1 unresolved externals
error: command '"C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe"' failed with exit status 1120
If the error occurs, we should check if the init_undname is defined in C files. There is only one C file, undname.c, in the source files of package. The file doesn't define init_undname but has undname() routine.
char *undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
{
return __unDName(buffer, mangled, buflen, malloc, free, flags);
}
Therefore my solution is to create init_undname() to wrap undname(). Please add the following init_undname() in undname.c.
char *init_undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
{
return undname (buffer, mangled, buflen, flags);
}
Let's try it again. It should be successful.
D:\pdbparse>python setup.py build
Let's install the package in Python environment.
D:\pdbparse>python setup.py install
Now we can run the examples of the package. It should be run successfully
D:\pdbparse\examples\python pdb_dump.py Test.pdb
Please run another example.
python pdb_get_syscall_table.py Test.pdb
File "C:\Python27\lib\site-packages\pdbparse\info.py", line 1, in <module>
from construct import *
ImportError: No module named construct
If the ImportError occurs, please download the contruct package here.
http://construct.readthedocs.org/en/latest/
Download the source code of construct package and put it in a directory. For example, D:\construct-2.5.2. Please build it and install it.
D:\construct>python setup.py build
D:\construct>python setup.py install
Please run the example again
python pdb_get_syscall_table.py Test.pdb
File "C:\Python27\lib\site-packages\construct\lib\binary.py", line 1, in <module>
import six
ImportError: No module named six
The error occurs because construct package depends on six package and we miss it. Please download it, build it, and install it.
https://pypi.python.org/pypi/six/1.7.3
D:\six-1.7.3>python setup.py build
D:\six-1.7.3>python setup.py install
Please run the example again
python pdb_get_syscall_table.py Test.pdb
Traceback (most recent call last):
File "pdb_get_syscall_table.py", line 9, in <module>
from pefile import PE
ImportError: No module named pefile
python pdb_get_syscall_table.py Test.pdb
Traceback (most recent call last):
File "pdb_get_syscall_table.py", line 9, in <module>
from pefile import PE
ImportError: No module named pefile
So the example need pefile package to parse EXE file and we miss the package. Please download it, build it and install it.
D:\pefile-1.2.10-139>python setup.py build
D:\pefile-1.2.10-139>python setup.py install
Now the example can be run successfully. There are many others examples help us to understand the PDB file format. W can try them.
pdb_dump.py
pdb_get_syscall_table.py
pdb_lookup.py
pdb_print_ctypes.py
pdb_print_gvars.py
pdb_print_tpi.py
pdb_tpi_vtypes.py
symchk.py
tpi_closure.py
tpi_print_construct.py
tpi_size.py
Monday, September 1, 2014
The Design Pattern of UEFI Report Status Code
Observer is a kind of design pattern that is described in the book, Design Pattern - Elements of Reusable Object-Oriented Software, 1994. I recommend the book as a bible.
This blog page describes that the design patterns are not only used in the implementation with Object Oriented Program (e.g., C++, JAVA), but also used in the implementation (e.g., UEFI) with non OOP (e.g., C). Mostly EDKII are implemented in C language. We can find there are many Observer patterns in it. ReportStatusCode is an example.
If you have the book on your hand, you can compare the package diagram of the blog page with the structure diagram in Observer chapter of the book.
In OOP implementation, the Notify() calls all Observer->Update() to update all registered Observer objects.
Notify ()
{
for o in Observers {
o->Update()
}
}
But for non-OOP implementation, ReportStatusCode(), how does it work? Below is the protocol definition of EFI_RSC_HANDLER_PROTOCOL.
typedef struct {
EFI_RSC_HANDLER_REGISTER Register;
EFI_RSC_HANDLER_UNREGISTER Unregister;
} EFI_RSC_HANDLER_PROTOCOL;
Let's focus on Register() in which we only be interested. For Unregister(), it seems that it is seldom used.
typedef
EFI_STATUS
(EFIAPI *EFI_RSC_HANDLER_REGISTER)(
IN EFI_RSC_HANDLER_CALLBACK Callback,
IN EFI_TPL Tpl
);
The most important parameter of the Register() is Callback. Below is the prototype of the Callback, EFI_RSC_HANDLER_CALLBACK.
typedef
EFI_STATUS
(EFIAPI *EFI_RSC_HANDLER_CALLBACK)(
IN EFI_STATUS_CODE_TYPE CodeType,
IN EFI_STATUS_CODE_VALUE Value,
IN UINT32 Instance,
IN EFI_GUID *CallerId,
IN EFI_STATUS_CODE_DATA *Data
);
Below is the prototype of the EFI_STATUS_CODE_PROTOCOL.ReportStatusCode().
typedef
EFI_STATUS
(EFIAPI *EFI_REPORT_STATUS_CODE) (
IN EFI_STATUS_CODE_TYPE Type,
IN EFI_STATUS_CODE_VALUE Value,
IN UINT32 Instance,
IN EFI_GUID *CallerId OPTIONAL,
IN EFI_STATUS_CODE_DATA *Data OPTIONAL
)
We found that the prototype of Register() and ReportStatusCode() are same. Therefore I guess that the implementation of the ReportStatusCode() should be as follows. Forgive me that I use python-style pseudo code to explain it.
ReportStatusCode (params):
for CallBack in mCallBackList:
CallBack (params)
Is pseudo code really true? You can trace the EDKII source code by yourself to verify if my assumption is true. This page just describes that there are many design patterns in EDK and EDKII. Furthermore, the ReportStatusCodeRouterRuntimeDxe.inf should be a design pattern of Singleton. What is Singleton? Please read book or GOOGLE it.
This blog page describes that the design patterns are not only used in the implementation with Object Oriented Program (e.g., C++, JAVA), but also used in the implementation (e.g., UEFI) with non OOP (e.g., C). Mostly EDKII are implemented in C language. We can find there are many Observer patterns in it. ReportStatusCode is an example.
If you have the book on your hand, you can compare the package diagram of the blog page with the structure diagram in Observer chapter of the book.
- The ReportStatusCodeRouterRuntimeDxe.inf is corresponding to the Subject object.
- The FirmwarePerformanceDxe.inf and StatusCodeHandlerRuntimeDxe.inf are corresponding to the Observer objects.
- The Register () is corresponding to the Subject.Attach()
- The Unregister () is corresponding to the Subject.Detach()
- The ReportStatusCode() is corresponding to the Subject.Notify()
In OOP implementation, the Notify() calls all Observer->Update() to update all registered Observer objects.
Notify ()
{
for o in Observers {
o->Update()
}
}
But for non-OOP implementation, ReportStatusCode(), how does it work? Below is the protocol definition of EFI_RSC_HANDLER_PROTOCOL.
typedef struct {
EFI_RSC_HANDLER_REGISTER Register;
EFI_RSC_HANDLER_UNREGISTER Unregister;
} EFI_RSC_HANDLER_PROTOCOL;
Let's focus on Register() in which we only be interested. For Unregister(), it seems that it is seldom used.
typedef
EFI_STATUS
(EFIAPI *EFI_RSC_HANDLER_REGISTER)(
IN EFI_RSC_HANDLER_CALLBACK Callback,
IN EFI_TPL Tpl
);
The most important parameter of the Register() is Callback. Below is the prototype of the Callback, EFI_RSC_HANDLER_CALLBACK.
typedef
EFI_STATUS
(EFIAPI *EFI_RSC_HANDLER_CALLBACK)(
IN EFI_STATUS_CODE_TYPE CodeType,
IN EFI_STATUS_CODE_VALUE Value,
IN UINT32 Instance,
IN EFI_GUID *CallerId,
IN EFI_STATUS_CODE_DATA *Data
);
Below is the prototype of the EFI_STATUS_CODE_PROTOCOL.ReportStatusCode().
typedef
EFI_STATUS
(EFIAPI *EFI_REPORT_STATUS_CODE) (
IN EFI_STATUS_CODE_TYPE Type,
IN EFI_STATUS_CODE_VALUE Value,
IN UINT32 Instance,
IN EFI_GUID *CallerId OPTIONAL,
IN EFI_STATUS_CODE_DATA *Data OPTIONAL
)
We found that the prototype of Register() and ReportStatusCode() are same. Therefore I guess that the implementation of the ReportStatusCode() should be as follows. Forgive me that I use python-style pseudo code to explain it.
ReportStatusCode (params):
for CallBack in mCallBackList:
CallBack (params)
Is pseudo code really true? You can trace the EDKII source code by yourself to verify if my assumption is true. This page just describes that there are many design patterns in EDK and EDKII. Furthermore, the ReportStatusCodeRouterRuntimeDxe.inf should be a design pattern of Singleton. What is Singleton? Please read book or GOOGLE it.
Subscribe to:
Posts (Atom)