Monday, October 13, 2014

The GUI of iOS 8

The iOS 8 has many things that we can learn when we design a GUI of an application. For example, the Cellular is defaulted to be turned off in Settings.

The Cellular has > icon so that we know it has many sub settings. We swipe it to expand the sub settings. We know that the Cellular Data is the main switch of the Cellular feature.

If we swipe it to turn on, the another setting, Enable 4G, appears.

We go back the Settings and find that the Cellular has > icon instead of "Off >".

We enters Cellular again. We know that the Enable 4G depends on Cellular Data. It means that the Enable 4G can be turned on only if Cellular Data is turned on. 

If Cellular Data is turned off, the Enable 4G is hidden.

If we turn on Cellular Data, Enable 4G appears again and it is turned off. It seems that iOS 8 remembers the state of Enable 4G.

We can proof the assumption by turning on Enable 4G.

Then we turn off Cellular Data. The Enable 4G is hidden.

Finally we turn on Cellular Data and the Enable 4G displays as be turned on. It makes sense. iOS 8 remembers the state of Enable 4G and shows it only when Cellular Data is turned on.

I consider the GUI is very user-friendly. We can compare it with iOS 7 that Cellular Data and Enable 4G are independent. We are always confused when we turn on/off Cellular Data or Enable 4G in iOS 7, but iOS 8 eliminates our confusion.


Circular Dependency

When I used the EDKII Package, I found that the package is similar to the UML package. We know that we should avoid the circular dependency in software design although the circular dependency always happens in natural domain.

Circular dependency is one of AntiPatterns but we cannot consider it as bad. What is benefit of it? When we want to public our source code of package that depends on a framework package (e.g.,  MdeModulePkg in EDKII.), if we don't want that our ideas in source code are easily stolen by competitors, we can modify the framework package to depend on our package to make circular dependency. Therefore the more circular dependencies we build, the more ideas we protect.


Tuesday, October 7, 2014

Write Your Half BIOS

What is BIOS? BIOS is shorten from Basic Input Output System. Don't doubt it. That is really meaning of BIOS. Someone says that the purpose of BIOS is to boot OS. I don't think so. OS should be optional. What is Input? The keyboard and mouse are input devices. What is Output? The monitor and port 80 are output devices. If you write a program that enables input and output devices and your program is performed by CPU at reset vector, your program is BIOS.

I'm glad to see the page has the same idea of writing your BIOS. The page was written by Harrison Hsieh who is my former colleague.

My page just describes how to write a Half BIOS, a program that is performed by CPU at reset vector but it doesn't enable input and output devices. I also demonstrate how to run the Half BIOS with Bochs.

The following are source code of Half BIOS.


Please follow the steps to build it by MASM and run it with Bochs.

Step 1. Build it with MASM611 or above.

> masm HalfBios.asm

The HalfBios.obj is generated.

Step 2. Update bochsrc.bxrc to change romimage file to HalfBios.bin

# romimage: file=../BIOS-bochs-latest
romimage: file=../HalfBios.bin

Step 3. Manually generate HalfBios.bin

> copy BIOS-bochs-latest HalfBios.bin

Please manually copy the 32 bytes with the signature "########" from HalfBios.obj to the offset 1FFE0h in HalfBios.bin. The content of 32 bytes is the program of Half BIOS.

Please make sure that there are EB F3 machine code at the offset 1FFF0h where is reset vector.

Step 4. Run Half BIOS with Bochs in debug mode.

C:\Program Files\Bochs-2.6.6\dlxlinux>..\bochsdbg.exe -f bochsrc.bxrc

We see that CPU performs the Half BIOS at the reset vector, f000:fff0, and then jump POST entry point.


Monday, October 6, 2014

Why I Like Bochs?

What is computer? I consider computer is a machine that quickly and automatically runs algorithm. The computer of Chinese text is 電 (electrical) 腦 (brain). Our brains are computers but not quick and correct. For example, I am an BIOS engineer. How do I proof it? I have blurred BIOS algorithm in my mind but it cannot be correctly run in my brain. I can tell you the algorithm but not the whole. I need to write BIOS program and run it in a machine to make sure that I know something in BIOS.

The machine, my brain, is not smart. I admire mathematicians who can present their theories in boards with very long formulas without computers. For me, a computer is necessary to help me thinking. I need burn BIOS in PC and see the process so that I can learn more.

But I don't want to use my PC to update BIOS. That has risk than using my brain. The idea is to use PC to emulate PC. I use a PC emulator to run a simple BIOS without breaking my PC at home. That is why I use Bochs, a PC emulator.


Sunday, October 5, 2014

Auto Run Debug Command with Script

The DOS debug.exe is a good utility to learn basic x68 Assembly language because we don't need to compile and link an ASM file and we just run it directly by entering commands. Sometimes we want to keep our interesting commands in a script file and automatically run it again with debug. How do we do it?

We can use a redirection (<) to read script file instead of the console. For example,

Please create script.txt that has the following lines:

db 0a,0b,0c,0d,0e,0f
mov al,4
lea bx,[100]
(Enter to make a blank line)
(please press Enter to make a blank line)

Please run the command.

debug < script.txt

Below is the result:


Thursday, October 2, 2014

Observe Reclaiming Process of EDKII Variable Services

This page provides a way to observe reclaiming process of EDKII Variable Services with UEFI Fault Tolerant Write Protocol.

EDKII Variable Services provides reclaim process with the recovery mechanism. When the platform restarts after power-off happens on NV reclaiming process, the driver checks the flag NV_FTW_WORKING that the NV reclaiming is not completed, it restores variables from NV_FTW_SPARE into NV_VARIABLE_STORE. Below files which are generated in build-time provide the regions information.


#define _PCD_VALUE_PcdFlashNvStorageVariableBase    0xFFF90000U
#define _PCD_VALUE_PcdFlashNvStorageVariableSize    0x0001D000U
#define _PCD_VALUE_PcdFlashNvStorageFtwWorkingBase  0xFFFAE000U
#define _PCD_VALUE_PcdFlashNvStorageFtwWorkingSize  0x00002000U
#define _PCD_VALUE_PcdFlashNvStorageFtwSpareBase    0xFFFB0000U
#define _PCD_VALUE_PcdFlashNvStorageFtwSpareSize    0x00020000U

Please follow the steps to observe the reclaiming process.

Step 1. Burn BIOS in ROM and dump it with DediProg to observe the region of NV_FTW_WORKING.

We see the header of NV_FTW_WORKING. The header is used to check if data is successfully written by FTW protocol. If the writing data is completed, FTW driver set a flag in NV_FTW_WORKING to indicator the completion.

Step 2. Boot UEFI shell and observe NV_FTW_WORKING.

We use a tool to observe the region of NV_FTW_WORKING.

> ToolA.efi -read -id 3 -lba 1 -offset e000 -size 70

We see that the region only has header.

Step 3. Make variable reclaiming.

We use a tool to make reclaiming process happen.

> ToolB.efi -tc1:rt=1,size=1000


We use a tool to observe the region of NV_FTW_WORKING.

> ToolA.efi -read -id 3 -lba 1 -offset e000 -size 70

We a tool to observe the region of NV_FTW_SPARE.

> ToolA.efi -read -id 3 -lba 2 -offset 0000 -size 70

We find that the NV_FTW_SPARE is empty because FTW erased it after the large data were written successfully in NV variable store. If we disable the following code in FaultTolerantWrite.c,

#if 0
  Status  = FtwEraseSpareBlock (FtwDevice);
  Ptr     = SpareBuffer;
  for (Index = 0; Index < FtwDevice->NumberOfSpareBlock; Index += 1) {
    MyLength = FtwDevice->BlockSize;
    Status = FtwDevice->FtwBackupFvb->Write (
                                        FtwDevice->FtwSpareLba + Index,
    if (EFI_ERROR (Status)) {
      FreePool (SpareBuffer);
      DPRINTF_INST ("}..19\n");
      return EFI_ABORTED;

    Ptr += MyLength;

and repeat the step 1, 2, and 3 again, we can find that the the NV_FTW_SPARE stores the backup data.

> ToolA.efi -read -id 3 -lba 2 -offset 0000 -size 70

It proofs that the region of NV_FTW_SPARE is built by UEFI Fault Tolerant Write Protocol. The region is a backup of NV_VARIABLE_STORE that is reclaimed.

Friday, September 19, 2014

Use MkLink to Solve the MASM611 EXE at 8.3 Directory Name

The title is shorten from "How to use MkLink to solve the problem that the EXE file built by MASM611 cannot be workable at the directory named with 8.3 format in Windows environment."

Sometimes we use MASM611 to write Assembly code in our Windows 7+ environment. If your name of source code directory is long (non 8.3 format), it will be a problem. For example,

Below is the content of Hellow.asm.

Before compiling the ASM with MASM611, please set the MASM611 path as follows.
>set path=%path%;c:\MASM611\BIN;c:\MASM611\BINR

Then please run ml to compile Hellow.asm. (Sometimes the system hangs because the you run it in the long directory.)

The HELLO.EXE is generated. Please run it.

Unfortunately, it cannot be workable. I guess that it is caused by the EXE file built by MASM611 cannot be workable at the long directory. How do you keep the long directory and make it workable?  You can 1) copy the EXE file to the root directory and run it, or 2) use MkLink to create a link directory, Hello, with 8.3 format.
>mklink /d "D:\Test\Hello" "D:\My Personal Data\範例程式\Hello World"

The link Hello is created in the Test directory.

Then you can run HELLO.exe in the link directory.

Finally you can edit the Hello.asm, compile it, and run it again in the link directory. All of your changes will impact the original directory.


Tuesday, September 16, 2014

Install XBMC in Raspberry Pi

"Raspberry Pi is a credit-card sized computer that plugs into you TV and a keyboard."

"XBMC is an award-winning free and open source (GPL) software media player, ...."

Yesterday I installed XBMC in my Raspberry Pi to make a portable media player. It was interesting. Below was my Raspberry Pi with SD card. The device was connected to Internet, a USB hub, a USB HDD, and a monitor via HDMI. The USB hub was connected to a keyboard and a wireless mouse. I selected wireless mouse to be a remote control of the media player.

Here was a USB HDD with 2TB capability that stores movies.

I installed Raspbmc in the SD card (SDHC 8GB Class2). "Raspbmc is a minimal Linux distribution based on Debian that brings XBMC to your Raspberry Pi."

The following steps were installation of Raspbmc.

Step 1. Format the SD card otherwise the Raspbmc cannot be installed successfully in step 3.

Step 2. Download Windows Installation of Raspbmc.

Step 3. After decompressing the downloaded file, please run installer.exe.


Step 4. Check if Raspbmc installer is installed. Please open the SD card that should contain the following files.

Step 5. Connect Raspberry Pi with Internet because the installer will load files from Internet. Insert the SD card with the installer into the Raspberry Pi and turn on the device. The following pictures in the screen display in progress of installation.

Step 6. Now you have XBMC in Raspberry Pi. You can add movie information from web (e.g., IMDB) into your movies in USB HDD.

Saturday, September 13, 2014

Use Bochs to Lean BIOS Entry Point

It's Sunday in autumn. Really nice weather with wind, not hot. I can see sun, white cloud, and a small far island on the sea out the window. I want to be outdoor to enjoy the weather, but there is a more interesting thing I cannot bear to skip it. That is to use Bochs to trace BIOS entry point. You should be interested in the task if you are/will be/was a BIOS engineer.

Bochs is a PC emulator. I've introduced it in my blog page.

Bochs reads a file, BIOS-bochs-latest file, to be a BIOS of the PC emulator. We know that CPU considers the address f000:fff0 as a BIOS entry point when the PC is power on. So we open the  BIOS file to get the last line.

We can find that there are machine codes and a date signature, 08/02/13, that I guess as the released date of the BIOS. What do the machine codes mean? Le's use debug command in prompt.

> copy BIOS-bochs-latest test.bin
> debug test.bin

So the the assembly language of the machine codes is JMP F000:E05B. Le's show the machine codes at F000:E05B.

The code started at XOR is POST Entry Point. Please refer the link to know what it is.

Above is a way to manually trace the BIOS entry point. How to use Bochs to easily trace it? Just run bochsdbg.exe, a debug mode of Bochs.

>  bochsdbg.exe

It is just like the DOS debug command, it is broken at the line of <bochs:1> to wait for user command. I send the help command to display how to use it. Then I select the n command to trace BIOS entry point step by step. It displays in assembly language and teaches us how BIOS entry point works. Is it interesting?


Bochs - A PC Emulator

Bochs is a PC emulator. What is my expectation to Bochs? I want to write a simplest BIOS and burn it in the emulator without breaking my real PC. I'm too excited to read Bochs document. Let's download it, run it, and try to change the default BIOS of the emulator.

Here is the Bochs official web page.

Please download Bochs-2.6.6.exe in the web page in your Windows environment and install it. Then, you can find Bochs 2.6.6 in your Windows start menu.

Just click Bochs 2.6.6 icon to start the PC emulator. You can see a window and a dialog.

No doubt, just click the Start button in the dialog to turn on the PC. I'll explain the Load button later. You can see another window that displays as a monitor of the PC. Another dialog is also popped up because a fatal error, no-bootable-device, occurs.

We can see a default BIOS, Bochs BIOS - build: 08/02/13, in the window. The question is how to replace the default BIOS? Do you remember the Load button? If you click it, a dialog, Load Bochs Config File, displays. It loads a *.bxrc file that you specify. You should be interested in the file. Please search them.

C:\Program Files\Bochs-2.6.6>dir *.bxrc /s
Directory of C:\Program Files\Bochs-2.6.6\dlxlinux
2014/09/13  ?? 10:08             1,533 bochsrc.bxrc

You can find the bochsrc.bxrc. Please open it and search BIOS or rom keyword.

# filename of ROM images
romimage: file=../BIOS-bochs-latest
vgaromimage: file=../VGABIOS-lgpl-latest

You can change the romimage config to specify your BIOS.

Another way is to run the following command in the prompt.
C:\Program Files\Bochs-2.6.6>bochs -f dlxlinux\bochsrc.bxrc -q

What does the command mean? Please see the help text by running "bochs -h".
C:\Program Files\Bochs-2.6.6>bochs -h

The below page shows you how to write the simplest BIOS and burn it with Bochs.


Thursday, September 11, 2014

Write Python Program to Use DBGHELP.DLL to Access a PDB File

The DBGHELP.DLL is a library provided by Microsoft to access PDB file. This page teaches us,
  1. How to use Python to load DLL.
  2. How to read PDB file to report file path and line number by a given IP register value.
  3. How to use python ctypes.
I select Python to try DBGHELP.DLL because Python is easy for use. Below is the source code. You can run it with the command.
> python

You can see the results.

# This source code is free software; you can redistribute it and/or
# modify it under the terms of version 2.1 of the GNU Lesser General Public
# License as published by the Free Software Foundation.
# This Python program demonstrates the usage of Windows DBGHELP.DLL by the
# below steps.
# 1. Locate functions of DBGHELP.DLL.
# 2. Load a PDB file at base address.
# 3. Input IP register value.
# 4. Output file path and line number of source code.
# Copyright (c) 2014 Count Chu.

import os
import ctypes

# Please set parameters here.

PdbFileName = "VcApp.pdb"               # A PDB file that was created by you.
BaseAddr = ctypes.c_uint64 (0x400000)   # Base address of the PDB file that will be loaded.
IpAddr = ctypes.c_uint64 (0x4113B0)     # IP register value that will be input.

# Locate functions of DBGHELP.DLL that are used in the program.

DbgHelp = ctypes.WinDLL (r"dbghelp.dll")
print DbgHelp                           # for debug.
SymInitialize = DbgHelp ["SymInitialize"]    
SymSetOptions = DbgHelp ["SymSetOptions"]
SymLoadModule64 = DbgHelp ["SymLoadModule64"]
SymGetLineFromAddr64 = DbgHelp ["SymGetLineFromAddr64"]
SymUnloadModule64 = DbgHelp ["SymUnloadModule64"]

# Define enumeration that is used in SymSetOptions.

class SymOpt:
    LOAD_LINES = 0x00000010
    UNDNAME = 0x00000002
    DEFERRED_LOADS = 0x00000004
# Call SymSetOptions().

print ("Call SymSetOptions()")          # for debug.

# Get the current process handle. It should be -1.

ProcessHandle = ctypes.windll.kernel32.GetCurrentProcess ()
print ("ProcessHandle = %xh" % ProcessHandle) # for debug.

# Call SymInitialize().

print ("Call SymInitialize()")          # for debug.
Status = SymInitialize (ProcessHandle, None, False)
if not Status:
    print "Error. Calling SymInitialize() failed."
# Call SymLoadModule64().

FileSize = os.path.getsize (PdbFileName)
print ("FileSize = %d" % FileSize)      # for debug.
print ("Call SymLoadModule64()")        # for debug.
OutBaseAddr = SymLoadModule64 (         # DWORD64 WINAPI SymLoadModule64 (
             ProcessHandle,             #   _In_      HANDLE hProcess,
             0,                         #   _In_opt_  HANDLE hFile,
             PdbFileName,               #   _In_opt_  PCSTR ImageName,
             None,                      #   _In_opt_  PCSTR ModuleName,
             BaseAddr,                  #   _In_      DWORD64 BaseOfDll,
             FileSize)                  #   _In_      DWORD SizeOfDll
                                        #   );
print ("OutBaseAddr = %xh" % OutBaseAddr) # for debug.

# Declare IMAGEHLP_LINE64 structure and create a instance of it.

class IMAGEHLP_LINE64 (ctypes.Structure):   # typedef struct _IMAGEHLP_LINE64 {
    _fields_ = [                            #   
        ('SizeOfStruct', ctypes.c_uint32),  #   DWORD   SizeOfStruct;    
        ('Key', ctypes.c_void_p),           #   PVOID   Key;
        ('LineNumber', ctypes.c_uint32),    #   DWORD   LineNumber;
        ('FileName', ctypes.c_char_p),      #   PTSTR   FileName;
        ('Address', ctypes.c_uint64)        #   DWORD64 Address;
        ]                                   # } IMAGEHLP_LINE64, *PIMAGEHLP_LINE64;

Line = IMAGEHLP_LINE64 ()                # Create a instance of IMAGEHLP_LINE64.

# Call SymGetLineFromAddr64()

print ("Call SymGetLineFromAddr64()")   # for debug.
Displacement = ctypes.c_uint32 (0)
Status = SymGetLineFromAddr64 (             # BOOL WINAPI SymGetLineFromAddr64 (
            ProcessHandle,                  #   _In_   HANDLE hProcess,
            IpAddr,                         #   _In_   DWORD64 dwAddr,
            ctypes.byref (Displacement),    #   _Out_  PDWORD pdwDisplacement,
            ctypes.byref (Line))            #   _Out_  PIMAGEHLP_LINE64 Line
                                            # );
print ("Status = %xh" % Status)         # for debug.
if not Status:
    print "Error. Calling SymGetLineFromAddr64() failed."
# Dump Displacement and IMAGEHLP_LINE64.
print ("Displacement = %d" % Displacement.value)
print ("Dump Line:")
print ("  SizeOfStruct = %d" % Line.SizeOfStruct)
print ("  Key          = %s" % Line.Key)
print ("  LineNumber   = %d" % Line.LineNumber)
print ("  FileName     = %s" % Line.FileName)
print ("  Address      = %xh" % Line.Address)

# Call SymUnloadModule64()
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)
if not Status:
    print "Error. Calling SymUnloadModule64() failed."

The Life Cycle of a UEFI Variable in Flash Part

Below is the data structure of one UEFI Variable stored in variable store.

We want to know the life cycle of the UEFI Variable in flash part.

In UEFI spec, We can do the following operations for one non-volatile Variable by calling SetVariable().
  • Add Operation
  • Update Operation
  • Delete Operation
We use state diagram to explain the life cycle of a UEFI variable.
Added ==> Updated ==> Deleted.

There is a circle on the Updated state. It means the variable can be updated many times.

Physically, if the variable exists in flash part, we cannot directly update variable on the same region in variable store. Variable driver must create a new variable with the same Name and GUID in the available region and set an invalid flag in the old variable. If the variable store is about to full, Variable driver reclaims the variable store to cleanup invalid variables.

Therefore the life cycle of a variable in flash part should be Added ==> Deleted. How is EDKII Variable designed to support Add/Update/Delete operations on variables in flash part and to assure fault tolerant? Let's focus on the State field of VARIABLE_HEADER. We expand them as binary format.

    0xfe = 1111-1110  // Variable is in obsolete transition
    0xfd = 1111-1100  // Variable is obsolete.
    0x7f = 0111-1111  // Variable header has been valid.
    0x3f = 0011-1111  // Variable has been completely added.

Add Operation

  1. State = VAR_HEADER_VALID_ONLY (0x7f)
  2. Write Data
  3. State = VAR_ADDED (0x3f)

The variable state is changed as follows.
0111-1111 (After Step 1)
0011-1111 (After Step 3)

Update Operation

  1. Old State &= VAR_IN_DELETED_TRANSITION (0xfe)
  2. State = VAR_HEADER_VALID_ONLY (0x7f)
  3. Write Data
  4. State = VAR_ADDED (0x3f)
  5. Old State &= VAR_DELETED (0xfd)

The old variable state is changed as follows.
0011-1111 (Initial state)
0011-1110 (After Step 1)
0011-1100 (After Step 5)

The new variable state is changed as follows.
0111-1111 (After Step 2)
0011-1111 (After Step 4)

Delete Operation

  1. State &= VAR_DELETED (0xfd)
The state is directly to add VAR_DELETED flag. The variable state is changed as follows.

0011-1111 (Initial State)
0011-1100 (After Step 1)

In Conclusion.

The states of the variable in flash part are as follows.

0111-1111 = 0x7f (Variable is in creating.)
0011-1111 = 0x3f (Variable is created.)
0011-1110 = 0x3e (Variable is in deleting.)
0011-1100 = 0x3d (Variable is in deleted.)

The state diagram (life cycle) of a variable in flash part are.
0x7f ==> 0x3f ==> 0x3e ==> 0x3d.

The diagram is on-way and no circle.

Please note that the value in the bit field is changed from 1 to 0. It cannot be changed from 0 to 1 in flash-write operation. The reason is the hardware characters of flash part. I'll try to explain it in another page.


Wednesday, September 10, 2014

How to Tweak EDKII BaseTools

Sometimes BIOS engineers need to tweak EDKII BaseTools to add some enhancements. Most of EDKII build tools are developed in Python scripts that are frozen into executables (EXE files) by cx_freeze. The BaseTools directory contains Python source code. The path is, for example.


We can modify them and rebuilt them into executables. This page teach us how to do it. If you have modified Python scripts files in the BaseTools directory, please follow the following steps to build executables.

Step 1. Install Python 2.7.

Please make sure that if the Python 2.7 is installed. If not, please go to python website, download it, and install it. The installed path is, for example.

Step 2. Install cx_freeze.

Please check if cx_freeze installs in Python 2.7. How do we check it? Just run dir.
C:\Python27\Scripts> dir *freeze*.*

If the cx_freeze has been installed. The dir command displays as follows.

    68 cxfreeze
 1,256 cxfreeze-postinstall
    78 cxfreeze-quickstart
    78 cxfreeze-quickstart.bat
    67 cxfreeze.bat

If the cx_freeze has not been installed, please go to the page.

Please click the PyPI link in the page,  download cx_Freeze-4.3.3.win32-py2.7.exe, and install it.

Step 3. Check if the cxfreeze.bat exists.

Sometimes the cxfreeze.bat is missed after we install cx_freeze. If the file doesn't exist in python script path, (e.g., C:\Python27\Scripts), please manually create it as follows.

@echo off
C:\Python27\python.exe C:\Python27\Scripts\cxfreeze %*

Step 4. Go to BaseTools directory. 

For example,

Step 5. Check the Makefile file and change it if necessary.

Sometimes we use old BaseTools where Makefile uses FreezePython.exe not cxfreeze.bat. Please modify it as follows.


Step 6. Set environment variables before building.

> set PYTHON_FREEZER_PATH=C:\Python27\Scripts

Step 7. Run "nmake all"

D:\EDKII\BaseTools\Source\Python> nmake all

You can find that the executables are generated in BaseTools Win32 path. For example,

Wednesday, September 3, 2014

How does debugger work with PDB?

Debugger is a big topic but this page only describes how debugger works with PDB file.

As WIKI description, PDB is developed by Microsoft for storing debugging information about a program. Debugger consumes PDB files to know the line number and file path for a given symbol or memory address where the program is broken. Unfortunately, the PDB format is undocumented. We only access PDB via DIA (Debug Interface Access) or DBGHELP DLL library (dbghelp.dll). We just imagine that PDB stores the following information at least.

  • Line Number
  • File Path
  • Offset (of Base Address)
  • Symbol Name

The most basic question is how to find file path and line number by a given memory address. The answer is very simple. Debugger just queries PDB by memory address to get the information.

Let's write a test program with VC to verify my assumption. The following is a simple program to print "Hello World!"

We build the program in debug mode. When debugging it, Visual Studio displays that the VcApp.exe is loaded at the Base Address, 00400000h.

When we move the cursor to the function decorations of main() and PrintMessage(), it displays their memory address are as follows.

PrintMessage = 004113B0h
main         = 00411410h

We have the second question. How does debugger know the file path and line number of the PrintMessage. For the question, I suppose that there are two tables in PDB.

Table1 = (Symbol Name, Offset)
Table2 = (Offset, Line Number, File Path)

Let's manually create Table1.

PrintMessage - Base Address = 000113B0h
main - Base Address         = 00011410h

Symbol Name  Offset
------------ ---------
PrintMessage 000113B0h
main         00011410h

When we run provided by PDBparse that was describe in another page, it reads PDB to display offsets of symbols as follows.

> python VcApp.pdb 0

It displays that the offset of PrintMessage is 000113B0h and main is 00011410h.

It proofs that Table1 really exists. How about Table2? Unfortunately, there are not examples of PDBparse to display file path and line number. I have not dug the information by reading PDB file yet. I decided to use dbghelp.dll to get file path and line number in PDB. I wrote a Python program to consume the DLL. The following are part of the program.

The whole source code was published in the page.
Write Python Program to Use DBGHELP.DLL to Access a PDB File.

BaseAddr = ctypes.c_uint64 (0x400000)
Status = SymLoadModule64 (ProcessHandle, 0, "VcApp.pdb", None, BaseAddr, SizeOfDll)
dwAddr = ctypes.c_uint64 (0x4113B0)
Status = SymGetLineFromAddr64 (ProcessHandle, dwAddr, ctypes.byref (pdwDisplacement), ctypes.byref (Line))
print ("Dump Line:")
print ("  SizeOfStruct = %d" % Line.SizeOfStruct)
print ("  Key = %s" % Line.Key)
print ("  LineNumber = %d" % Line.LineNumber)
print ("  FileName = %s" % Line.FileName)
print ("  Address = %xh" % Line.Address)
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)

The result displays:
Dump Line:
  SizeOfStruct = 0
  Key = None
  LineNumber = 7
  FileName = d:\vcapp\vcapp.c
  Address = 4113b0h

As the result, this program reports file path (d:\vcapp\vcapp.c) and line number (7) by the given memory address (0x4113B0) where the VcApp.exe is broken at the PrintMessage(). This program proofs that Table2 exists.

This program also help to solve the first question, how to find file path and line number by an given memory address? I consider that Table1 and Table2 or the merged table fulfil the solution of the first question.

I've explains how debugger works with PDB file.

Tuesday, September 2, 2014

Use Python PDBparse to parse PDB files.

PDBparse is a Python package for parsing Microsoft PDB files. I like Python solution for parsing PDB files because it is open source. By running its examples, I easily understand PDB format without reading PDB spec initially if there is really a PDB spec.

This page just describes how to install the package and how to run its examples.

Below is the website of PDBparse.

We can download the source code here.

So please check out it and put it in a directory. For example, D:\pdbparse. Let's build it and install it.
D:\pdbparse>python build
running build
running build_py
running build_ext
building 'pdbparse._undname' extension
error: Unable to find vcvarsall.bat

If the error occurs, please don't give up. When running, Python 2.7 searches for Visual Studio 2008. If we don't have the version or another version of Visual Studio, the error occurs. We can trick  Python by setting an environment variable. For example, environment has Visual Studio 2005.

Let's build the package again.
D:\pdbparse>python build
LINK : error LNK2001: unresolved external symbol init_undname
build\temp.win32-2.7\Release\src\_undname.lib : fatal error LNK1120: 1 unresolved externals
error: command '"C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe"' failed with exit status 1120

If the error occurs, we should check if the init_undname is defined in C files. There is only one C file, undname.c, in the source files of package. The file doesn't define init_undname but has undname() routine.
char *undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
    return __unDName(buffer, mangled, buflen, malloc, free, flags);

Therefore my solution is to create init_undname() to wrap undname(). Please add the following init_undname() in undname.c.
char *init_undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
  return undname (buffer, mangled, buflen, flags);

Let's try it again. It should be successful.
D:\pdbparse>python build

Let's install the package in Python environment.
D:\pdbparse>python install

Now we can run the examples of the package. It should be run successfully
D:\pdbparse\examples\python Test.pdb

Please run another example.
python Test.pdb
  File "C:\Python27\lib\site-packages\pdbparse\", line 1, in <module>
    from construct import *
ImportError: No module named construct

If the ImportError occurs, please download the contruct package here.

Download the source code of construct package and put it in a directory. For example, D:\construct-2.5.2. Please build it and install it.
D:\construct>python build
D:\construct>python install

Please run the example again
python Test.pdb
  File "C:\Python27\lib\site-packages\construct\lib\", line 1, in <module>
    import six
ImportError: No module named six

The error occurs because construct package depends on six package and we miss it. Please download it, build it, and install it.
D:\six-1.7.3>python build
D:\six-1.7.3>python install

Please run the example again
python Test.pdb
Traceback (most recent call last):
  File "", line 9, in <module>
    from pefile import PE
ImportError: No module named pefile

So the example need pefile package to parse EXE file and we miss the package. Please download it, build it and install it.
D:\pefile-1.2.10-139>python build
D:\pefile-1.2.10-139>python install

Now the example can be run successfully. There are many others examples help us to understand the PDB file format. W can try them.

Monday, September 1, 2014

The Design Pattern of UEFI Report Status Code

Observer is a kind of design pattern that is described in the book, Design Pattern - Elements of Reusable Object-Oriented Software, 1994. I recommend the book as a bible.

This blog page describes that the design patterns are not only used in the implementation with Object Oriented Program (e.g., C++, JAVA), but also used in the implementation (e.g., UEFI) with non OOP (e.g., C). Mostly EDKII are implemented in C language. We can find there are many Observer patterns in it. ReportStatusCode is an example.

If you have the book on your hand, you can compare the package diagram of the blog page with the structure diagram in Observer chapter of the book.

  • The ReportStatusCodeRouterRuntimeDxe.inf  is corresponding to the Subject object.
  • The FirmwarePerformanceDxe.inf and StatusCodeHandlerRuntimeDxe.inf are corresponding to the Observer objects.
  • The Register () is corresponding to the Subject.Attach()
  • The Unregister () is corresponding to the Subject.Detach()
  • The ReportStatusCode() is corresponding to the Subject.Notify()

In OOP implementation, the Notify() calls all Observer->Update() to update all registered Observer objects.
Notify ()
  for o in Observers {

But for non-OOP implementation, ReportStatusCode(), how does it work? Below is the protocol definition of EFI_RSC_HANDLER_PROTOCOL.

typedef struct {

Let's focus on Register() in which we only be interested. For Unregister(), it seems that it is seldom used.

  IN EFI_TPL                    Tpl

The most important parameter of the Register() is Callback. Below is the prototype of the Callback, EFI_RSC_HANDLER_CALLBACK.

  IN UINT32                 Instance,
  IN EFI_GUID               *CallerId,

Below is the prototype of the EFI_STATUS_CODE_PROTOCOL.ReportStatusCode().

  IN UINT32                   Instance,
  IN EFI_GUID                 *CallerId  OPTIONAL,

We found that the prototype of Register() and ReportStatusCode() are same. Therefore I guess that the implementation of the ReportStatusCode() should be as follows. Forgive me that I use python-style pseudo code to explain it.

ReportStatusCode (params):
  for CallBack in mCallBackList:
    CallBack (params)  

Is pseudo code really true? You can trace the EDKII source code by yourself to verify if my assumption is true. This page just describes that there are many design patterns in EDK and EDKII. Furthermore, the ReportStatusCodeRouterRuntimeDxe.inf should be a design pattern of Singleton. What is Singleton? Please read book or GOOGLE it.