Use UEFI Shell Script to Automatically Test Your UEFI Module

We know that test program is very import for developing software components. It is not only for QA staff, but also for ourselves programmer to program quickly and to be quality.

Software developing is actually a serial of automatic processes. Code is an automatic process run by computer. It is more fast and exact than human brain. that is why a computer was developed to encrypt and decrypt messages. Test is a process we can do it automatically to reduce our effort and assurance the quality of software product. Design is a process, but we only design with our brain now. Maybe design will be delivered by a computer in the future. Programming is a process that is also be written by programmers, but Hollywood movies show us code can update itself.

This page focus on automatically testing UEFI module with UEFI shell script.

The idea is very simple. For example, we want to use a smart way to erase data on flash ROM with size in 4096-byte alignment. Flash ROM supports the following commands to erase data:

4K-Erase: Erasing data in 4K bytes
32K-Erase: Erasing data in 32k bytes
64K-Erase: Erasing data in 64k bytes

The performance is bad if we only use 4K-Erase command to erase large data. For example, To erase data with 1024K, we send 256 commands of 4K-Erase. If we use 64K-Erase command, we only sends 16 commands. We cannot always use 64K-Erase command because it is only be used in 64K alignment.

For example, if we want to erase 548K data at 100K offset, we only use the following 14 commands of 4K-Erase, 32K-Erase, and 64K-Erase.

100K Offset, 4K-Erase
104K Offset, 4K-Erase
108K Offset, 4K-Erase
112K Offset, 4K-Erase
116K Offset, 4K-Erase
120K Offset, 4K-Erase
128K Offset, 64K-Erase
192K Offset, 64K-Erase
256K Offset, 64K-Erase
320K Offset, 64K-Erase
384K Offset, 64K-Erase
448K Offset, 64K-Erase
512K Offset, 32K-Erase
544K Offset, 4K-Erase

We define the SmartFlashErase() for this requirement.

EFI_STATIS
EFIAPI
SmartFlashErase (
  IN UINT64 Offset
  IN UINT32 Size
  );

Our test program is SmartFlashEraseTest.efi with the following usage.

SmartFlashEraseTest.efi -offset offset -size -size

Out test script is SmartFlashEraseTest.nsh as below.

@echo [Case]
SmartFlashEraseTest.efi -offset 100k -size 548k
@if not %LastError% == 0 then
  @echo [Error] LastError = %LastError%
  goto exit
@endif
@echo [CheckPoint] Status = 0 (Success)

:exit

We can use redirect the NSH to a log file.
SmarFlashEraseTest.nsh > Case.log

The log file contains tags, [Case], [Error], [CheckPoint]. We can write a python program to analyze these tags. I'll write another page for this topic.

This way not only for UEFI shell, any environment that supports script can use the way to automatically test our software components.

-Count

Observe Reclaiming Process of EDKII Variable Services

This page provides a way to observe reclaiming process of EDKII Variable Services with UEFI Fault Tolerant Write Protocol.

EDKII Variable Services provides reclaim process with the recovery mechanism. When the platform restarts after power-off happens on NV reclaiming process, the driver checks the flag NV_FTW_WORKING that the NV reclaiming is not completed, it restores variables from NV_FTW_SPARE into NV_VARIABLE_STORE. Below files which are generated in build-time provide the regions information.

FlashMap.h
#define FLASH_REGION_NVSTORAGE_SUBREGION_NV_VARIABLE_STORE_OFFSET 0x590000
#define FLASH_REGION_NVSTORAGE_SUBREGION_NV_FTW_WORKING_OFFSET    0x5ae000
#define FLASH_REGION_NVSTORAGE_SUBREGION_NV_FTW_SPARE_OFFSET      0x5b0000

AutoGenFlashMap.h
#define _PCD_VALUE_PcdFlashNvStorageVariableBase    0xFFF90000U
#define _PCD_VALUE_PcdFlashNvStorageVariableSize    0x0001D000U
#define _PCD_VALUE_PcdFlashNvStorageFtwWorkingBase  0xFFFAE000U
#define _PCD_VALUE_PcdFlashNvStorageFtwWorkingSize  0x00002000U
#define _PCD_VALUE_PcdFlashNvStorageFtwSpareBase    0xFFFB0000U
#define _PCD_VALUE_PcdFlashNvStorageFtwSpareSize    0x00020000U

Please follow the steps to observe the reclaiming process.

Step 1. Burn BIOS in ROM and dump it with DediProg to observe the region of NV_FTW_WORKING.

We see the header of NV_FTW_WORKING. The header is used to check if data is successfully written by FTW protocol. If the writing data is completed, FTW driver set a flag in NV_FTW_WORKING to indicator the completion.

Step 2. Boot UEFI shell and observe NV_FTW_WORKING.

We use a tool to observe the region of NV_FTW_WORKING.

> ToolA.efi -read -id 3 -lba 1 -offset e000 -size 70

We see that the region only has header.

Step 3. Make variable reclaiming.

We use a tool to make reclaiming process happen.

> ToolB.efi -tc1:rt=1,size=1000

Step 4. Observe NV_FTW_WORKING and NV_FTW_SPARE

We use a tool to observe the region of NV_FTW_WORKING.

> ToolA.efi -read -id 3 -lba 1 -offset e000 -size 70

We a tool to observe the region of NV_FTW_SPARE.

> ToolA.efi -read -id 3 -lba 2 -offset 0000 -size 70

We find that the NV_FTW_SPARE is empty because FTW erased it after the large data were written successfully in NV variable store. If we disable the following code in FaultTolerantWrite.c,

#if 0
  Status  = FtwEraseSpareBlock (FtwDevice);
  Ptr     = SpareBuffer;
  for (Index = 0; Index < FtwDevice->NumberOfSpareBlock; Index += 1) {
    MyLength = FtwDevice->BlockSize;
    Status = FtwDevice->FtwBackupFvb->Write (
                                        FtwDevice->FtwBackupFvb,
                                        FtwDevice->FtwSpareLba + Index,
                                        0,
                                        &MyLength,
                                        Ptr
                                        );
    if (EFI_ERROR (Status)) {
      FreePool (SpareBuffer);
      DPRINTF_INST ("}..19\n");
      return EFI_ABORTED;
    }

    Ptr += MyLength;
  }
#endif

and repeat the step 1, 2, and 3 again, we can find that the the NV_FTW_SPARE stores the backup data.

> ToolA.efi -read -id 3 -lba 2 -offset 0000 -size 70

It proofs that the region of NV_FTW_SPARE is built by UEFI Fault Tolerant Write Protocol. The region is a backup of NV_VARIABLE_STORE that is reclaimed.

The Life Cycle of a UEFI Variable in Flash Part

Below is the data structure of one UEFI Variable stored in variable store.

We want to know the life cycle of the UEFI Variable in flash part.

In UEFI spec, We can do the following operations for one non-volatile Variable by calling SetVariable().

• Add Operation
• Update Operation
• Delete Operation

We use state diagram to explain the life cycle of a UEFI variable.

Added ==> Updated ==> Deleted.

There is a circle on the Updated state. It means the variable can be updated many times.

Physically, if the variable exists in flash part, we cannot directly update variable on the same region in variable store. Variable driver must create a new variable with the same Name and GUID in the available region and set an invalid flag in the old variable. If the variable store is about to full, Variable driver reclaims the variable store to cleanup invalid variables.

Therefore the life cycle of a variable in flash part should be Added ==> Deleted. How is EDKII Variable designed to support Add/Update/Delete operations on variables in flash part and to assure fault tolerant? Let's focus on the State field of VARIABLE_HEADER. We expand them as binary format.

    0xfe = 1111-1110  // Variable is in obsolete transition
    0xfd = 1111-1100  // Variable is obsolete.
    0x7f = 0111-1111  // Variable header has been valid.
    0x3f = 0011-1111  // Variable has been completely added.

Add Operation

1. State = VAR_HEADER_VALID_ONLY (0x7f)
2. Write Data
3. State = VAR_ADDED (0x3f)

The variable state is changed as follows.
0111-1111 (After Step 1)
0011-1111 (After Step 3)

Update Operation

1. Old State &= VAR_IN_DELETED_TRANSITION (0xfe)
2. State = VAR_HEADER_VALID_ONLY (0x7f)
3. Write Data
4. State = VAR_ADDED (0x3f)
5. Old State &= VAR_DELETED (0xfd)

The old variable state is changed as follows.
0011-1111 (Initial state)
0011-1110 (After Step 1)
0011-1100 (After Step 5)

The new variable state is changed as follows.
0111-1111 (After Step 2)
0011-1111 (After Step 4)

Delete Operation

1. State &= VAR_DELETED (0xfd)

The state is directly to add VAR_DELETED flag. The variable state is changed as follows.

0011-1111 (Initial State)
0011-1100 (After Step 1)

In Conclusion.

The states of the variable in flash part are as follows.

0111-1111 = 0x7f (Variable is in creating.)
0011-1111 = 0x3f (Variable is created.)
0011-1110 = 0x3e (Variable is in deleting.)
0011-1100 = 0x3d (Variable is in deleted.)

The state diagram (life cycle) of a variable in flash part are.
0x7f ==> 0x3f ==> 0x3e ==> 0x3d.

The diagram is on-way and no circle.

Please note that the value in the bit field is changed from 1 to 0. It cannot be changed from 0 to 1 in flash-write operation. The reason is the hardware characters of flash part. I'll try to explain it in another page.

-Count

How does UEFI Fault Tolerant Write protocol work on Variable reclaiming process?

The FTW (Fault Tolerant Write) protocol is called when the variable reclaiming happens. Why does Variable driver use FTW during reclaiming? How does FTW work?

We know that variables are stored in ROM. When a variable is updated by SetVariable() calling, Variable driver appends a new variable in the tail and marks the old variable as invalid. Why not to update variable directly? The reason is the hardware characters of flash part. In Write operation, the bit in flash part can be changed from 1 to 0, but cannot be changed from 0 to 1. Erase operation is to erase a block in flash part with FF value. That is why the Variable driver needs to reclaim when the variable store is about full. Reclaiming means deleting invalid variables and combined all valid variables together. The simple reclaiming process is:

1. Put all valid variables into memory.
2. Erase variable store in ROM.
3. Write all variables from memory to ROM.

There is a risk that if the platform is shut down unexpectedly in the step 3, the variables reclaiming are not completed to make platform crashed. To avoid the risk, the reclaiming process is updated as follows, I guess.

1. Put all valid variables into memory.
2. Set a flag to announce that the reclaiming start.
3. Write all valid variables from memory to backup region in ROM
4. Erase variable store in ROM.
5. Write all variables from memory to variable store in ROM.
6. Update the flag to indicate that the reclaiming is completed.

So if the platform is shut down unexpectedly in step 5 and the platform restarts, Don't worry. The platform will recover variables from backup region because it detects the reclaiming has not been completed. The recovery process is:

1. Detect the flag that the reclaiming has not be completed yet.
2. Restore variables from backup region to variable store.
3. Update the flag to indicate that the reclaiming is completed.

Again, if the platform is shut down unexpectedly in step 2, the recovery steps are run again when the platform starts.

How does FTW support this idea? Variable driver uses FTW->Write() to write large data to implement step 2 - step 6 in reclaiming process.

• The flag is updated in NV_FTW_WORKING.
• The backup region is in NV_FTW_SPARE.

We can use a tool to set different variables to trigger variable reclaiming, and use another tool to dump NV_FTW_WORKING and NV_FTW_SPARE. How do we find the two regions? There are many ways to find them.

Method 1 - Dump ROM and search the signature of NV_FTW_WORKING

EFI_FAULT_TOLERANT_WORKING_BLOCK_HEADER->Signature

#define EDKII_WORKING_BLOCK_SIGNATURE_GUID \ 

{0x9e58292b, 0x7c68, 0x497d, {0xa0, 0xce, 0x65, 0x0, 0xfd, 0x9f, 0x1b, 0x95} }

Method 2 - Find the values of NV_FTW_WORKING and NV_FTW_SPARE in AutoGenFlashMap.h

The AutoGenFlashMap.h is generated by EDKII build system. We can find the values that we are interested.

#define _PCD_VALUE_PcdFlashNvStorageVariableBase    0xFFF90000U
#define _PCD_VALUE_PcdFlashNvStorageVariableSize    0x0001D000U
#define _PCD_VALUE_PcdFlashNvStorageFtwWorkingBase  0xFFFAE000U
#define _PCD_VALUE_PcdFlashNvStorageFtwWorkingSize  0x00002000U
#define _PCD_VALUE_PcdFlashNvStorageFtwSpareBase    0xFFFB0000U
#define _PCD_VALUE_PcdFlashNvStorageFtwSpareSize    0x00020000U

From the definitions, we know that the offset of FtwWorkingBase (NV_FTW_WORKING) is  FFFAE000h - FFF90000 = 1E000h

Furthermore, we can use a tool to dump the content in UEFI Shell. For example,
ToolA.efi -read -id 3 -lba 1 offset e000 -size 70

The tool consumes Firmware Volume Block protocols to dump data from ROM. I will describe it in another page.

Because the page is too long, please refer the page to observe the detail of reclaiming process.
http://countchu.blogspot.tw/2014/10/observe-reclaiming-process-of-edkii.html

I hope this page and the referred page provide you enough background knowledge of FTW and Variable reclaiming process.

Explain the UEFI Secure Boot

The Secure Boot technology is defined by UEFI specification. BIOS uses RSA algorithm to check the integration of the booted OS and to verify if the booted OS is released by the OS vendor.  Let's show the algorithm of Secure Boot.

OS vendor signs OS Loader with Private Key:
{PublicKey, PrivateKey} = GenerateKeyPair ()
Digest = Hash (OsLoaderCode)
Signature = Encrypt (Digest, PrivateKey)
OsLoader = {OsLoaderCode, Signature}

BIOS Vendor enrolls signature into BIOS:
Db = {header, SigList, ...}
SigList = {Sig, ...}
Sig = {..., Cert}
Cert = {..., PublicKey}

BIOS verifies OS Loader before starting it.
Digest = Hash (OsLoaderBody)
Digest2 = Decrypt (Signature, PublicKey)
if (Digest == Digest2) {
  The OsLoader is verified successfully.
  Start the OsLoader.
}