Some Bugs Happen When Disabling Debug Message

We sometimes find a hard bug that only happens on disabling debug. It is hard to fix because if we enable debug, the bug disappears.  In my experience, we can summarize the list that are principles of tracing source code.

1) Need to add delay.

We need to add delay-time to wait a response from a device. If we disable debug, we lost delay-time so that a bug occurs.

2) A variable must be initialized.

Sometimes an uninitialized variable will be got a good value when debug is enabled. If we disable debug, the value of uninitialized variable is bad so that a bug occurs.

3) Check the bound of a buffer.

We are always easy to write data out of a buffer. If we enable debug, the data doesn't affect our program. But if we disable debug, the data may crack our program.

-Count

Write Python Program to Use DBGHELP.DLL to Access a PDB File

The DBGHELP.DLL is a library provided by Microsoft to access PDB file. This page teaches us,

1. How to use Python to load DLL.
2. How to read PDB file to report file path and line number by a given IP register value.
3. How to use python ctypes.

I select Python to try DBGHELP.DLL because Python is easy for use. Below is the source code. You can run it with the command.

> python DbgHelpTest.py

You can see the results.

---

DbgHelpTest.py

#
# This source code is free software; you can redistribute it and/or
# modify it under the terms of version 2.1 of the GNU Lesser General Public
# License as published by the Free Software Foundation.
#
# This Python program demonstrates the usage of Windows DBGHELP.DLL by the
# below steps.
# 
# 1. Locate functions of DBGHELP.DLL.
# 2. Load a PDB file at base address.
# 3. Input IP register value.
# 4. Output file path and line number of source code.
#
# Copyright (c) 2014 Count Chu.
#

import os
import ctypes

#
# Please set parameters here.
#

PdbFileName = "VcApp.pdb"               # A PDB file that was created by you.
BaseAddr = ctypes.c_uint64 (0x400000)   # Base address of the PDB file that will be loaded.
IpAddr = ctypes.c_uint64 (0x4113B0)     # IP register value that will be input.

#
# Locate functions of DBGHELP.DLL that are used in the program.
#

DbgHelp = ctypes.WinDLL (r"dbghelp.dll")
print DbgHelp                           # for debug.
SymInitialize = DbgHelp ["SymInitialize"]    
SymSetOptions = DbgHelp ["SymSetOptions"]
SymLoadModule64 = DbgHelp ["SymLoadModule64"]
SymGetLineFromAddr64 = DbgHelp ["SymGetLineFromAddr64"]
SymUnloadModule64 = DbgHelp ["SymUnloadModule64"]

#
# Define enumeration that is used in SymSetOptions.
#

class SymOpt:
    LOAD_LINES = 0x00000010
    UNDNAME = 0x00000002
    DEFERRED_LOADS = 0x00000004
    
#
# Call SymSetOptions().
#

print ("Call SymSetOptions()")          # for debug.
SymSetOptions (SymOpt.LOAD_LINES | SymOpt.UNDNAME | SymOpt.DEFERRED_LOADS)

#
# Get the current process handle. It should be -1.
#

ProcessHandle = ctypes.windll.kernel32.GetCurrentProcess ()
print ("ProcessHandle = %xh" % ProcessHandle) # for debug.

#
# Call SymInitialize().
#

print ("Call SymInitialize()")          # for debug.
Status = SymInitialize (ProcessHandle, None, False)
if not Status:
    print "Error. Calling SymInitialize() failed."
    
#
# Call SymLoadModule64().
#    

FileSize = os.path.getsize (PdbFileName)
print ("FileSize = %d" % FileSize)      # for debug.
print ("Call SymLoadModule64()")        # for debug.
OutBaseAddr = SymLoadModule64 (         # DWORD64 WINAPI SymLoadModule64 (
             ProcessHandle,             #   _In_      HANDLE hProcess,
             0,                         #   _In_opt_  HANDLE hFile,
             PdbFileName,               #   _In_opt_  PCSTR ImageName,
             None,                      #   _In_opt_  PCSTR ModuleName,
             BaseAddr,                  #   _In_      DWORD64 BaseOfDll,
             FileSize)                  #   _In_      DWORD SizeOfDll
                                        #   );
print ("OutBaseAddr = %xh" % OutBaseAddr) # for debug.

#
# Declare IMAGEHLP_LINE64 structure and create a instance of it.
#

class IMAGEHLP_LINE64 (ctypes.Structure):   # typedef struct _IMAGEHLP_LINE64 {
    _fields_ = [                            #   
        ('SizeOfStruct', ctypes.c_uint32),  #   DWORD   SizeOfStruct;    
        ('Key', ctypes.c_void_p),           #   PVOID   Key;
        ('LineNumber', ctypes.c_uint32),    #   DWORD   LineNumber;
        ('FileName', ctypes.c_char_p),      #   PTSTR   FileName;
        ('Address', ctypes.c_uint64)        #   DWORD64 Address;
        ]                                   # } IMAGEHLP_LINE64, *PIMAGEHLP_LINE64;

Line = IMAGEHLP_LINE64 ()                # Create a instance of IMAGEHLP_LINE64.

#
# Call SymGetLineFromAddr64()
#

print ("Call SymGetLineFromAddr64()")   # for debug.
Displacement = ctypes.c_uint32 (0)
Status = SymGetLineFromAddr64 (             # BOOL WINAPI SymGetLineFromAddr64 (
            ProcessHandle,                  #   _In_   HANDLE hProcess,
            IpAddr,                         #   _In_   DWORD64 dwAddr,
            ctypes.byref (Displacement),    #   _Out_  PDWORD pdwDisplacement,
            ctypes.byref (Line))            #   _Out_  PIMAGEHLP_LINE64 Line
                                            # );
print ("Status = %xh" % Status)         # for debug.
if not Status:
    print "Error. Calling SymGetLineFromAddr64() failed."
    
#
# Dump Displacement and IMAGEHLP_LINE64.
#    
    
print ("Displacement = %d" % Displacement.value)
print ("Dump Line:")
print ("  SizeOfStruct = %d" % Line.SizeOfStruct)
print ("  Key          = %s" % Line.Key)
print ("  LineNumber   = %d" % Line.LineNumber)
print ("  FileName     = %s" % Line.FileName)
print ("  Address      = %xh" % Line.Address)

#
# Call SymUnloadModule64()
#
    
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)
if not Status:
    print "Error. Calling SymUnloadModule64() failed."

How does debugger work with PDB?

Debugger is a big topic but this page only describes how debugger works with PDB file.

As WIKI description, PDB is developed by Microsoft for storing debugging information about a program. Debugger consumes PDB files to know the line number and file path for a given symbol or memory address where the program is broken. Unfortunately, the PDB format is undocumented. We only access PDB via DIA (Debug Interface Access) or DBGHELP DLL library (dbghelp.dll). We just imagine that PDB stores the following information at least.

• Line Number
• File Path
• Offset (of Base Address)
• Symbol Name

The most basic question is how to find file path and line number by a given memory address. The answer is very simple. Debugger just queries PDB by memory address to get the information.

Let's write a test program with VC to verify my assumption. The following is a simple program to print "Hello World!"

We build the program in debug mode. When debugging it, Visual Studio displays that the VcApp.exe is loaded at the Base Address, 00400000h.

When we move the cursor to the function decorations of main() and PrintMessage(), it displays their memory address are as follows.

PrintMessage = 004113B0h
main         = 00411410h

We have the second question. How does debugger know the file path and line number of the PrintMessage. For the question, I suppose that there are two tables in PDB.

Table1 = (Symbol Name, Offset)
Table2 = (Offset, Line Number, File Path)

Let's manually create Table1.

PrintMessage - Base Address = 000113B0h
main - Base Address         = 00011410h


Symbol Name  Offset
------------ ---------
PrintMessage 000113B0h
main         00011410h


When we run pdb_print_gvars.py provided by PDBparse that was describe in another page, it reads PDB to display offsets of symbols as follows.

> python pdb_print_gvars.py VcApp.pdb 0
__enc$textbss$begin,0x1000,0,.textbss
__enc$textbss$end,0x11000,0,.textbss
_PrintMessage,0x113b0,2,.text
__imp__printf,0x182bc,0,.idata
__RTC_CheckEsp,0x114e0,2,.text
__RTC_Shutdown,0x11720,2,.text
__RTC_InitBase,0x116e0,2,.text
_main,0x11410,2,.text
??_C@_0M@FKNCOEJD@Hello?5Word?6?$AA@,0x1573c,0,.rdata
___security_cookie,0x17000,0,.data

It displays that the offset of PrintMessage is 000113B0h and main is 00011410h.

It proofs that Table1 really exists. How about Table2? Unfortunately, there are not examples of PDBparse to display file path and line number. I have not dug the information by reading PDB file yet. I decided to use dbghelp.dll to get file path and line number in PDB. I wrote a Python program to consume the DLL. The following are part of the program.

The whole source code was published in the page.
Write Python Program to Use DBGHELP.DLL to Access a PDB File.

...
BaseAddr = ctypes.c_uint64 (0x400000)
Status = SymLoadModule64 (ProcessHandle, 0, "VcApp.pdb", None, BaseAddr, SizeOfDll)
...
dwAddr = ctypes.c_uint64 (0x4113B0)
Status = SymGetLineFromAddr64 (ProcessHandle, dwAddr, ctypes.byref (pdwDisplacement), ctypes.byref (Line))
print ("Dump Line:")
print ("  SizeOfStruct = %d" % Line.SizeOfStruct)
print ("  Key = %s" % Line.Key)
print ("  LineNumber = %d" % Line.LineNumber)
print ("  FileName = %s" % Line.FileName)
print ("  Address = %xh" % Line.Address)
...
Status = SymUnloadModule64 (ProcessHandle, BaseAddr)
...

The result displays:
Dump Line:
  SizeOfStruct = 0
  Key = None
  LineNumber = 7
  FileName = d:\vcapp\vcapp.c
  Address = 4113b0h

As the result, this program reports file path (d:\vcapp\vcapp.c) and line number (7) by the given memory address (0x4113B0) where the VcApp.exe is broken at the PrintMessage(). This program proofs that Table2 exists.

This program also help to solve the first question, how to find file path and line number by an given memory address? I consider that Table1 and Table2 or the merged table fulfil the solution of the first question.

I've explains how debugger works with PDB file.

Use Python PDBparse to parse PDB files.

PDBparse is a Python package for parsing Microsoft PDB files. I like Python solution for parsing PDB files because it is open source. By running its examples, I easily understand PDB format without reading PDB spec initially if there is really a PDB spec.

This page just describes how to install the package and how to run its examples.

Below is the website of PDBparse.
https://code.google.com/p/pdbparse/

We can download the source code here.
http://pdbparse.googlecode.com/svn/trunk

So please check out it and put it in a directory. For example, D:\pdbparse. Let's build it and install it.
D:\pdbparse>python setup.py build
running build
running build_py
running build_ext
building 'pdbparse._undname' extension
error: Unable to find vcvarsall.bat

If the error occurs, please don't give up. When running setup.py, Python 2.7 searches for Visual Studio 2008. If we don't have the version or another version of Visual Studio, the error occurs. We can trick  Python by setting an environment variable. For example, environment has Visual Studio 2005.
>set VS90COMNTOOLS=%VS80COMNTOOLS%

Let's build the package again.
D:\pdbparse>python setup.py build
LINK : error LNK2001: unresolved external symbol init_undname
build\temp.win32-2.7\Release\src\_undname.lib : fatal error LNK1120: 1 unresolved externals
error: command '"C:\Program Files\Microsoft Visual Studio 8\VC\BIN\link.exe"' failed with exit status 1120

If the error occurs, we should check if the init_undname is defined in C files. There is only one C file, undname.c, in the source files of package. The file doesn't define init_undname but has undname() routine.
char *undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
{
    return __unDName(buffer, mangled, buflen, malloc, free, flags);
}

Therefore my solution is to create init_undname() to wrap undname(). Please add the following init_undname() in undname.c.
char *init_undname(char *buffer, char *mangled, int buflen, unsigned short int flags)
{
  return undname (buffer, mangled, buflen, flags);
}

Let's try it again. It should be successful.
D:\pdbparse>python setup.py build

Let's install the package in Python environment.
D:\pdbparse>python setup.py install

Now we can run the examples of the package. It should be run successfully
D:\pdbparse\examples\python pdb_dump.py Test.pdb

Please run another example.
python pdb_get_syscall_table.py Test.pdb
  File "C:\Python27\lib\site-packages\pdbparse\info.py", line 1, in <module>
    from construct import *
ImportError: No module named construct

If the ImportError occurs, please download the contruct package here.
http://construct.readthedocs.org/en/latest/

Download the source code of construct package and put it in a directory. For example, D:\construct-2.5.2. Please build it and install it.
D:\construct>python setup.py build
D:\construct>python setup.py install

Please run the example again
python pdb_get_syscall_table.py Test.pdb
  File "C:\Python27\lib\site-packages\construct\lib\binary.py", line 1, in <module>
    import six
ImportError: No module named six

The error occurs because construct package depends on six package and we miss it. Please download it, build it, and install it.
https://pypi.python.org/pypi/six/1.7.3
D:\six-1.7.3>python setup.py build
D:\six-1.7.3>python setup.py install

Please run the example again
python pdb_get_syscall_table.py Test.pdb
Traceback (most recent call last):
  File "pdb_get_syscall_table.py", line 9, in <module>
    from pefile import PE
ImportError: No module named pefile

So the example need pefile package to parse EXE file and we miss the package. Please download it, build it and install it.

https://pypi.python.org/pypi/pefile/1.2.10-139

D:\pefile-1.2.10-139>python setup.py build

D:\pefile-1.2.10-139>python setup.py install

Now the example can be run successfully. There are many others examples help us to understand the PDB file format. W can try them.

pdb_dump.py

pdb_get_syscall_table.py

pdb_lookup.py

pdb_print_ctypes.py

pdb_print_gvars.py

pdb_print_tpi.py

pdb_tpi_vtypes.py

symchk.py

tpi_closure.py

tpi_print_construct.py

tpi_size.py

Use Debug Command to Dump BDA

We can use debug command in Windows prompt to dump BDA (BIOS Data Area.) The BDA is created at memory location 0040:000h with a 256 bytes or over.

The word in 40:0e is the segment address of EBDA. The address is 9ac0:0000. We can use debug to dump it.

The word in 40:13 is the system available memory size in Kbytes below 640k. The value is 0280h = 640. Therefore the memory size is 640k.

We can refer the following link for the detail of  BDA.
http://www.bioscentral.com/misc/bda.htm