Friday, September 19, 2014

Use MkLink to Solve the MASM611 EXE at 8.3 Directory Name

The title is shorten from "How to use MkLink to solve the problem that the EXE file built by MASM611 cannot be workable at the directory named with 8.3 format in Windows environment."

Sometimes we use MASM611 to write Assembly code in our Windows 7+ environment. If your name of source code directory is long (non 8.3 format), it will be a problem. For example,

Below is the content of Hellow.asm.

Before compiling the ASM with MASM611, please set the MASM611 path as follows.
>set path=%path%;c:\MASM611\BIN;c:\MASM611\BINR

Then please run ml to compile Hellow.asm. (Sometimes the system hangs because the you run it in the long directory.)

The HELLO.EXE is generated. Please run it.

Unfortunately, it cannot be workable. I guess that it is caused by the EXE file built by MASM611 cannot be workable at the long directory. How do you keep the long directory and make it workable?  You can 1) copy the EXE file to the root directory and run it, or 2) use MkLink to create a link directory, Hello, with 8.3 format.
>mklink /d "D:\Test\Hello" "D:\My Personal Data\範例程式\Hello World"

The link Hello is created in the Test directory.

Then you can run HELLO.exe in the link directory.

Finally you can edit the Hello.asm, compile it, and run it again in the link directory. All of your changes will impact the original directory.


Tuesday, September 16, 2014

Install XBMC in Raspberry Pi

"Raspberry Pi is a credit-card sized computer that plugs into you TV and a keyboard."

"XBMC is an award-winning free and open source (GPL) software media player, ...."

Yesterday I installed XBMC in my Raspberry Pi to make a portable media player. It was interesting. Below was my Raspberry Pi with SD card. The device was connected to Internet, a USB hub, a USB HDD, and a monitor via HDMI. The USB hub was connected to a keyboard and a wireless mouse. I selected wireless mouse to be a remote control of the media player.

Here was a USB HDD with 2TB capability that stores movies.

I installed Raspbmc in the SD card (SDHC 8GB Class2). "Raspbmc is a minimal Linux distribution based on Debian that brings XBMC to your Raspberry Pi."

The following steps were installation of Raspbmc.

Step 1. Format the SD card otherwise the Raspbmc cannot be installed successfully in step 3.

Step 2. Download Windows Installation of Raspbmc.

Step 3. After decompressing the downloaded file, please run installer.exe.


Step 4. Check if Raspbmc installer is installed. Please open the SD card that should contain the following files.

Step 5. Connect Raspberry Pi with Internet because the installer will load files from Internet. Insert the SD card with the installer into the Raspberry Pi and turn on the device. The following pictures in the screen display in progress of installation.

Step 6. Now you have XBMC in Raspberry Pi. You can add movie information from web (e.g., IMDB) into your movies in USB HDD.

Saturday, September 13, 2014

Use Bochs to Lean BIOS Entry Point

It's Sunday in autumn. Really nice weather with wind, not hot. I can see sun, white cloud, and a small far island on the sea out the window. I want to be outdoor to enjoy the weather, but there is a more interesting thing I cannot bear to skip it. That is to use Bochs to trace BIOS entry point. You should be interested in the task if you are/will be/was a BIOS engineer.

Bochs is a PC emulator. I've introduced it in my blog page.

Bochs reads a file, BIOS-bochs-latest file, to be a BIOS of the PC emulator. We know that CPU considers the address f000:fff0 as a BIOS entry point when the PC is power on. So we open the  BIOS file to get the last line.

We can find that there are machine codes and a date signature, 08/02/13, that I guess as the released date of the BIOS. What do the machine codes mean? Le's use debug command in prompt.

> copy BIOS-bochs-latest test.bin
> debug test.bin

So the the assembly language of the machine codes is JMP F000:E05B. Le's show the machine codes at F000:E05B.

The code started at XOR is POST Entry Point. Please refer the link to know what it is.

Above is a way to manually trace the BIOS entry point. How to use Bochs to easily trace it? Just run bochsdbg.exe, a debug mode of Bochs.

>  bochsdbg.exe

It is just like the DOS debug command, it is broken at the line of <bochs:1> to wait for user command. I send the help command to display how to use it. Then I select the n command to trace BIOS entry point step by step. It displays in assembly language and teaches us how BIOS entry point works. Is it interesting?


Bochs - A PC Emulator

Bochs is a PC emulator. What is my expectation to Bochs? I want to write a simplest BIOS and burn it in the emulator without breaking my real PC. I'm too excited to read Bochs document. Let's download it, run it, and try to change the default BIOS of the emulator.

Here is the Bochs official web page.

Please download Bochs-2.6.6.exe in the web page in your Windows environment and install it. Then, you can find Bochs 2.6.6 in your Windows start menu.

Just click Bochs 2.6.6 icon to start the PC emulator. You can see a window and a dialog.

No doubt, just click the Start button in the dialog to turn on the PC. I'll explain the Load button later. You can see another window that displays as a monitor of the PC. Another dialog is also popped up because a fatal error, no-bootable-device, occurs.

We can see a default BIOS, Bochs BIOS - build: 08/02/13, in the window. The question is how to replace the default BIOS? Do you remember the Load button? If you click it, a dialog, Load Bochs Config File, displays. It loads a *.bxrc file that you specify. You should be interested in the file. Please search them.

C:\Program Files\Bochs-2.6.6>dir *.bxrc /s
Directory of C:\Program Files\Bochs-2.6.6\dlxlinux
2014/09/13  ?? 10:08             1,533 bochsrc.bxrc

You can find the bochsrc.bxrc. Please open it and search BIOS or rom keyword.

# filename of ROM images
romimage: file=../BIOS-bochs-latest
vgaromimage: file=../VGABIOS-lgpl-latest

You can change the romimage config to specify your BIOS.

Another way is to run the following command in the prompt.
C:\Program Files\Bochs-2.6.6>bochs -f dlxlinux\bochsrc.bxrc -q

What does the command mean? Please see the help text by running "bochs -h".
C:\Program Files\Bochs-2.6.6>bochs -h

The below page shows you how to write the simplest BIOS and burn it with Bochs.


Thursday, September 11, 2014

Write Python Program to Use DBGHELP.DLL to Access a PDB File

The DBGHELP.DLL is a library provided by Microsoft to access PDB file. This page teaches us,
  1. How to use Python to load DLL.
  2. How to read PDB file to report file path and line number by a given IP register value.
  3. How to use python ctypes.
I select Python to try DBGHELP.DLL because Python is easy for use. Below is the source code. You can run it with the command.
> python

You can see the results.

# This source code is free software; you can redistribute it and/or
# modify it under the terms of version 2.1 of the GNU Lesser General Public
# License as published by the Free Software Foundation.
# This Python program demonstrates the usage of Windows DBGHELP.DLL by the
# below steps.
# 1. Locate functions of DBGHELP.DLL.
# 2. Load a PDB file at base address.
# 3. Input IP register value.
# 4. Output file path and line number of source code.
# Copyright (c) 2014 Count Chu.

import os
import ctypes

# Please set parameters here.

PdbFileName = "VcApp.pdb"               # A PDB file that was created by you.
BaseAddr = ctypes.c_uint64 (0x400000)   # Base address of the PDB file that will be loaded.
IpAddr = ctypes.c_uint64 (0x4113B0)     # IP register value that will be input.

# Locate functions of DBGHELP.DLL that are used in the program.

DbgHelp = ctypes.WinDLL (r"dbghelp.dll")
print DbgHelp                           # for debug.
SymInitialize = DbgHelp ["SymInitialize"]    
SymSetOptions = DbgHelp ["SymSetOptions"]
SymLoadModule64 = DbgHelp ["SymLoadModule64"]
SymGetLineFromAddr64 = DbgHelp ["SymGetLineFromAddr64"]
SymUnloadModule64 = DbgHelp ["SymUnloadModule64"]

# Define enumeration that is used in SymSetOptions.

class SymOpt:
    LOAD_LINES = 0x00000010
    UNDNAME = 0x00000002
    DEFERRED_LOADS = 0x00000004
# Call SymSetOptions().

print ("Call SymSetOptions()")          # for debug.

# Get the current process handle. It should be -1.

ProcessHandle = ctypes.windll.kernel32.GetCurrentProcess ()
print ("ProcessHandle = %xh" % ProcessHandle) # for debug.

# Call SymInitialize().

print ("Call SymInitialize()")          # for debug.
Status = SymInitialize (ProcessHandle, None, False)
if not Status:
    print "Error. Calling SymInitialize() failed."
# Call SymLoadModule64().

FileSize = os.path.getsize (PdbFileName)
print ("FileSize = %d" % FileSize)      # for debug.
print ("Call SymLoadModule64()")        # for debug.
OutBaseAddr = SymLoadModule64 (         # DWORD64 WINAPI SymLoadModule64 (
             ProcessHandle,             #   _In_      HANDLE hProcess,
             0,                         #   _In_opt_  HANDLE hFile,
             PdbFileName,               #   _In_opt_  PCSTR ImageName,
             None,                      #   _In_opt_  PCSTR ModuleName,
             BaseAddr,                  #   _In_      DWORD64 BaseOfDll,
             FileSize)                  #   _In_      DWORD SizeOfDll
                                        #   );
print ("OutBaseAddr = %xh" % OutBaseAddr) # for debug.

# Declare IMAGEHLP_LINE64 structure and create a instance of it.

class IMAGEHLP_LINE64 (ctypes.Structure):   # typedef struct _IMAGEHLP_LINE64 {
    _fields_ = [                            #   
        ('SizeOfStruct', ctypes.c_uint32),  #   DWORD   SizeOfStruct;    
        ('Key', ctypes.c_void_p),           #   PVOID   Key;
        ('LineNumber', ctypes.c_uint32),    #   DWORD   LineNumber;
        ('FileName', ctypes.c_char_p),      #   PTSTR   FileName;
        ('Address', ctypes.c_uint64)        #   DWORD64 Address;
        ]                                   # } IMAGEHLP_LINE64, *PIMAGEHLP_LINE64;

Line = IMAGEHLP_LINE64 ()                # Create a instance of IMAGEHLP_LINE64.

# Call SymGetLineFromAddr64()

print ("Call SymGetLineFromAddr64()")   # for debug.
Displacement = ctypes.c_uint32 (0)
Status = SymGetLineFromAddr64 (             # BOOL WINAPI SymGetLineFromAddr64 (
            ProcessHandle,                  #   _In_   HANDLE hProcess,
            IpAddr,                         #   _In_   DWORD64 dwAddr,
            ctypes.byref (Displacement),    #   _Out_  PDWORD pdwDisplacement,
            ctypes.byref (Line))            #   _Out_  PIMAGEHLP_LINE64 Line
                                            # );
print ("Status = %xh" % Status)         # for debug.
if not Status:
    print "Error. Calling SymGetLineFromAddr64() failed."
# Dump Displacement and IMAGEHLP_LINE64.
print ("Displacement = %d" % Displacement.value)
print ("Dump Line:")
print ("  SizeOfStruct = %d" % Line.SizeOfStruct)
print ("  Key          = %s" % Line.Key)
print ("  LineNumber   = %d" % Line.LineNumber)
print ("  FileName     = %s" % Line.FileName)
print ("  Address      = %xh" % Line.Address)

# Call SymUnloadModule64()
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)
if not Status:
    print "Error. Calling SymUnloadModule64() failed."

The Life Cycle of a UEFI Variable in Flash Part

Below is the data structure of one UEFI Variable stored in variable store.

We want to know the life cycle of the UEFI Variable in flash part.

In UEFI spec, We can do the following operations for one non-volatile Variable by calling SetVariable().
  • Add Operation
  • Update Operation
  • Delete Operation
We use state diagram to explain the life cycle of a UEFI variable.
Added ==> Updated ==> Deleted.

There is a circle on the Updated state. It means the variable can be updated many times.

Physically, if the variable exists in flash part, we cannot directly update variable on the same region in variable store. Variable driver must create a new variable with the same Name and GUID in the available region and set an invalid flag in the old variable. If the variable store is about to full, Variable driver reclaims the variable store to cleanup invalid variables.

Therefore the life cycle of a variable in flash part should be Added ==> Deleted. How is EDKII Variable designed to support Add/Update/Delete operations on variables in flash part and to assure fault tolerant? Let's focus on the State field of VARIABLE_HEADER. We expand them as binary format.

    0xfe = 1111-1110  // Variable is in obsolete transition
    0xfd = 1111-1100  // Variable is obsolete.
    0x7f = 0111-1111  // Variable header has been valid.
    0x3f = 0011-1111  // Variable has been completely added.

Add Operation

  1. State = VAR_HEADER_VALID_ONLY (0x7f)
  2. Write Data
  3. State = VAR_ADDED (0x3f)

The variable state is changed as follows.
0111-1111 (After Step 1)
0011-1111 (After Step 3)

Update Operation

  1. Old State &= VAR_IN_DELETED_TRANSITION (0xfe)
  2. State = VAR_HEADER_VALID_ONLY (0x7f)
  3. Write Data
  4. State = VAR_ADDED (0x3f)
  5. Old State &= VAR_DELETED (0xfd)

The old variable state is changed as follows.
0011-1111 (Initial state)
0011-1110 (After Step 1)
0011-1100 (After Step 5)

The new variable state is changed as follows.
0111-1111 (After Step 2)
0011-1111 (After Step 4)

Delete Operation

  1. State &= VAR_DELETED (0xfd)
The state is directly to add VAR_DELETED flag. The variable state is changed as follows.

0011-1111 (Initial State)
0011-1100 (After Step 1)

In Conclusion.

The states of the variable in flash part are as follows.

0111-1111 = 0x7f (Variable is in creating.)
0011-1111 = 0x3f (Variable is created.)
0011-1110 = 0x3e (Variable is in deleting.)
0011-1100 = 0x3d (Variable is in deleted.)

The state diagram (life cycle) of a variable in flash part are.
0x7f ==> 0x3f ==> 0x3e ==> 0x3d.

The diagram is on-way and no circle.

Please note that the value in the bit field is changed from 1 to 0. It cannot be changed from 0 to 1 in flash-write operation. The reason is the hardware characters of flash part. I'll try to explain it in another page.


Wednesday, September 10, 2014

How to Tweak EDKII BaseTools

Sometimes BIOS engineers need to tweak EDKII BaseTools to add some enhancements. Most of EDKII build tools are developed in Python scripts that are frozen into executables (EXE files) by cx_freeze. The BaseTools directory contains Python source code. The path is, for example.


We can modify them and rebuilt them into executables. This page teach us how to do it. If you have modified Python scripts files in the BaseTools directory, please follow the following steps to build executables.

Step 1. Install Python 2.7.

Please make sure that if the Python 2.7 is installed. If not, please go to python website, download it, and install it. The installed path is, for example.

Step 2. Install cx_freeze.

Please check if cx_freeze installs in Python 2.7. How do we check it? Just run dir.
C:\Python27\Scripts> dir *freeze*.*

If the cx_freeze has been installed. The dir command displays as follows.

    68 cxfreeze
 1,256 cxfreeze-postinstall
    78 cxfreeze-quickstart
    78 cxfreeze-quickstart.bat
    67 cxfreeze.bat

If the cx_freeze has not been installed, please go to the page.

Please click the PyPI link in the page,  download cx_Freeze-4.3.3.win32-py2.7.exe, and install it.

Step 3. Check if the cxfreeze.bat exists.

Sometimes the cxfreeze.bat is missed after we install cx_freeze. If the file doesn't exist in python script path, (e.g., C:\Python27\Scripts), please manually create it as follows.

@echo off
C:\Python27\python.exe C:\Python27\Scripts\cxfreeze %*

Step 4. Go to BaseTools directory. 

For example,

Step 5. Check the Makefile file and change it if necessary.

Sometimes we use old BaseTools where Makefile uses FreezePython.exe not cxfreeze.bat. Please modify it as follows.


Step 6. Set environment variables before building.

> set PYTHON_FREEZER_PATH=C:\Python27\Scripts

Step 7. Run "nmake all"

D:\EDKII\BaseTools\Source\Python> nmake all

You can find that the executables are generated in BaseTools Win32 path. For example,

Wednesday, September 3, 2014

How does debugger work with PDB?

Debugger is a big topic but this page only describes how debugger works with PDB file.

As WIKI description, PDB is developed by Microsoft for storing debugging information about a program. Debugger consumes PDB files to know the line number and file path for a given symbol or memory address where the program is broken. Unfortunately, the PDB format is undocumented. We only access PDB via DIA (Debug Interface Access) or DBGHELP DLL library (dbghelp.dll). We just imagine that PDB stores the following information at least.

  • Line Number
  • File Path
  • Offset (of Base Address)
  • Symbol Name

The most basic question is how to find file path and line number by a given memory address. The answer is very simple. Debugger just queries PDB by memory address to get the information.

Let's write a test program with VC to verify my assumption. The following is a simple program to print "Hello World!"

We build the program in debug mode. When debugging it, Visual Studio displays that the VcApp.exe is loaded at the Base Address, 00400000h.

When we move the cursor to the function decorations of main() and PrintMessage(), it displays their memory address are as follows.

PrintMessage = 004113B0h
main         = 00411410h

We have the second question. How does debugger know the file path and line number of the PrintMessage. For the question, I suppose that there are two tables in PDB.

Table1 = (Symbol Name, Offset)
Table2 = (Offset, Line Number, File Path)

Let's manually create Table1.

PrintMessage - Base Address = 000113B0h
main - Base Address         = 00011410h

Symbol Name  Offset
------------ ---------
PrintMessage 000113B0h
main         00011410h

When we run provided by PDBparse that was describe in another page, it reads PDB to display offsets of symbols as follows.

> python VcApp.pdb 0

It displays that the offset of PrintMessage is 000113B0h and main is 00011410h.

It proofs that Table1 really exists. How about Table2? Unfortunately, there are not examples of PDBparse to display file path and line number. I have not dug the information by reading PDB file yet. I decided to use dbghelp.dll to get file path and line number in PDB. I wrote a Python program to consume the DLL. The following are part of the program.

The whole source code was published in the page.
Write Python Program to Use DBGHELP.DLL to Access a PDB File.

BaseAddr = ctypes.c_uint64 (0x400000)
Status = SymLoadModule64 (ProcessHandle, 0, "VcApp.pdb", None, BaseAddr, SizeOfDll)
dwAddr = ctypes.c_uint64 (0x4113B0)
Status = SymGetLineFromAddr64 (ProcessHandle, dwAddr, ctypes.byref (pdwDisplacement), ctypes.byref (Line))
print ("Dump Line:")
print ("  SizeOfStruct = %d" % Line.SizeOfStruct)
print ("  Key = %s" % Line.Key)
print ("  LineNumber = %d" % Line.LineNumber)
print ("  FileName = %s" % Line.FileName)
print ("  Address = %xh" % Line.Address)
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)

The result displays:
Dump Line:
  SizeOfStruct = 0
  Key = None
  LineNumber = 7
  FileName = d:\vcapp\vcapp.c
  Address = 4113b0h

As the result, this program reports file path (d:\vcapp\vcapp.c) and line number (7) by the given memory address (0x4113B0) where the VcApp.exe is broken at the PrintMessage(). This program proofs that Table2 exists.

This program also help to solve the first question, how to find file path and line number by an given memory address? I consider that Table1 and Table2 or the merged table fulfil the solution of the first question.

I've explains how debugger works with PDB file.

Tuesday, September 2, 2014

Use Python PDBparse to parse PDB files.

PDBparse is a Python package for parsing Microsoft PDB files. I like Python solution for parsing PDB files because it is open source. By running its examples, I easily understand PDB format without reading PDB spec initially if there is really a PDB spec.

This page just describes how to install the package and how to run its examples.

Below is the website of PDBparse.

We can download the source code here.

So please check out it and put it in a directory. For example, D:\pdbparse. Let's build it and install it.
D:\pdbparse>python build
running build
running build_py
running build_ext
building 'pdbparse._undname' extension
error: Unable to find vcvarsall.bat

If the error occurs, please don't give up. When running, Python 2.7 searches for Visual Studio 2008. If we don't have the version or another version of Visual Studio, the error occurs. We can trick  Python by setting an environment variable. For example, environment has Visual Studio 2005.

Let's build the package again.
D:\pdbparse>python build
LINK : error LNK2001: unresolved external symbol init_undname
build\temp.win32-2.7\Release\src\_undname.lib : fatal error LNK1120: 1 unresolved externals
error: command '"C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe"' failed with exit status 1120

If the error occurs, we should check if the init_undname is defined in C files. There is only one C file, undname.c, in the source files of package. The file doesn't define init_undname but has undname() routine.
char *undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
    return __unDName(buffer, mangled, buflen, malloc, free, flags);

Therefore my solution is to create init_undname() to wrap undname(). Please add the following init_undname() in undname.c.
char *init_undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
  return undname (buffer, mangled, buflen, flags);

Let's try it again. It should be successful.
D:\pdbparse>python build

Let's install the package in Python environment.
D:\pdbparse>python install

Now we can run the examples of the package. It should be run successfully
D:\pdbparse\examples\python Test.pdb

Please run another example.
python Test.pdb
  File "C:\Python27\lib\site-packages\pdbparse\", line 1, in <module>
    from construct import *
ImportError: No module named construct

If the ImportError occurs, please download the contruct package here.

Download the source code of construct package and put it in a directory. For example, D:\construct-2.5.2. Please build it and install it.
D:\construct>python build
D:\construct>python install

Please run the example again
python Test.pdb
  File "C:\Python27\lib\site-packages\construct\lib\", line 1, in <module>
    import six
ImportError: No module named six

The error occurs because construct package depends on six package and we miss it. Please download it, build it, and install it.
D:\six-1.7.3>python build
D:\six-1.7.3>python install

Please run the example again
python Test.pdb
Traceback (most recent call last):
  File "", line 9, in <module>
    from pefile import PE
ImportError: No module named pefile

So the example need pefile package to parse EXE file and we miss the package. Please download it, build it and install it.
D:\pefile-1.2.10-139>python build
D:\pefile-1.2.10-139>python install

Now the example can be run successfully. There are many others examples help us to understand the PDB file format. W can try them.

Monday, September 1, 2014

The Design Pattern of UEFI Report Status Code

Observer is a kind of design pattern that is described in the book, Design Pattern - Elements of Reusable Object-Oriented Software, 1994. I recommend the book as a bible.

This blog page describes that the design patterns are not only used in the implementation with Object Oriented Program (e.g., C++, JAVA), but also used in the implementation (e.g., UEFI) with non OOP (e.g., C). Mostly EDKII are implemented in C language. We can find there are many Observer patterns in it. ReportStatusCode is an example.

If you have the book on your hand, you can compare the package diagram of the blog page with the structure diagram in Observer chapter of the book.

  • The ReportStatusCodeRouterRuntimeDxe.inf  is corresponding to the Subject object.
  • The FirmwarePerformanceDxe.inf and StatusCodeHandlerRuntimeDxe.inf are corresponding to the Observer objects.
  • The Register () is corresponding to the Subject.Attach()
  • The Unregister () is corresponding to the Subject.Detach()
  • The ReportStatusCode() is corresponding to the Subject.Notify()

In OOP implementation, the Notify() calls all Observer->Update() to update all registered Observer objects.
Notify ()
  for o in Observers {

But for non-OOP implementation, ReportStatusCode(), how does it work? Below is the protocol definition of EFI_RSC_HANDLER_PROTOCOL.

typedef struct {

Let's focus on Register() in which we only be interested. For Unregister(), it seems that it is seldom used.

  IN EFI_TPL                    Tpl

The most important parameter of the Register() is Callback. Below is the prototype of the Callback, EFI_RSC_HANDLER_CALLBACK.

  IN UINT32                 Instance,
  IN EFI_GUID               *CallerId,

Below is the prototype of the EFI_STATUS_CODE_PROTOCOL.ReportStatusCode().

  IN UINT32                   Instance,
  IN EFI_GUID                 *CallerId  OPTIONAL,

We found that the prototype of Register() and ReportStatusCode() are same. Therefore I guess that the implementation of the ReportStatusCode() should be as follows. Forgive me that I use python-style pseudo code to explain it.

ReportStatusCode (params):
  for CallBack in mCallBackList:
    CallBack (params)  

Is pseudo code really true? You can trace the EDKII source code by yourself to verify if my assumption is true. This page just describes that there are many design patterns in EDK and EDKII. Furthermore, the ReportStatusCodeRouterRuntimeDxe.inf should be a design pattern of Singleton. What is Singleton? Please read book or GOOGLE it.