Note: This proof-of-concept project focuses on a single processor family, RISCV. Some results are checked against equivalent x86_64 processors, to see if pending issues are limited in scope or likely to hit a larger community

This project collects files that may stress - in a good way - Ghidra’s import, disassembly, and decompilation capabilities. Others are doing a great job extending Ghidra’s ability to import and recognize C++ structures and classes, so we will focus on lower level objects like instruction sets, relocation codes, and pending toolchain improvements. The primary CPU family will be based on the RISCV-64 processor. This processor is relatively new and easily modified, so it will likely show lots of new features early. Not all of these new features will make it into common use or arenas in which Ghidra is necessary, so we don’t really know how much effort is worth spending on any given feature.

The RISCV processor family being relatively new, we can expect compiler and toolchain support to be evolving more rapidly than more established families like x86_64. That means RISCV appliances may be more likely to be built with newer compiler toolchains than x86_64 appliances.

There are two key goals here:

1. Experiment with Ghidra import integration tests that can detect Ghidra regressions. This involves collecting a number of processor and toolchain binary exemplars to be imported plus analysis scripts to verify those import results remain valid. Example: verify that ELF relocation codes are properly handled when importing a RISCV-64 kernel module. These integration tests should always pass after changes to Ghidra’s source code.
2. Collect feature-specific binary exemplars that might highlight emergent gaps in Ghidra’s import processes. Ghidra will usually fail to properly import these exemplars, allowing the Ghidra development team to triage the gap and evaluate options for closing it. Example: pass the RISCV instruction set extension testsuite from binutils/gas into Ghidra to test whether Ghidra can recognize all of the new instructions gas can generate.

A secondary goal developed during testing - explore the impact on Ghidra users of vector instruction set extensions as used in aggressive compiler optimization. The RISCV 1.0 vector instructions as generated by the gcc 14.0 optimizing compiler can turn simple loops into more complex instruction sequences.

The initial scope focuses on RISCV 64 bit processors capable of running a full Linux network stack, likely implementing the 2023 standard profile.

We want to track recent additions to standard RISCV-64 toolchains (like binutils and gcc) to see how they might make life interesting for Ghidra developers. At present, that includes newly frozen or ratified instruction set architecture (ISA) changes and compiler autovectorization optimizations. Some vendor-specific instruction set extensions will be included if they are accepted into the binutils main branch.

Running integration tests

Note: These scripts use both unittest and logging frameworks, where the loglevel is variously set at INFO or WARN. The exact output may vary

The first two steps collect binary exemplars for Ghidra to import. Large binaries are extracted from public disk images, such as the latest Fedora RISCV-64 system disk image. Small binaries are generated locally from minimal C or C++ source files and gcc toolchains.

The large binaries are downloaded and extracted using acquireExternalExemplars.py. This script is built on the python unittest framework to either verify the existence of previously extracted exemplars or regenerate those if missing.

The small binaries are created - if not already present - with the generateInternalExemplars.py script

Warning: GCC-13 and GCC-14 binary toolchains are not included in this project. Sources should be downloaded, compiled, and installed to something like /opt/... then post-processed by bundled toolchain scripts into portable, hermetic tarballs.

$ ./acquireExternalExemplars.py 
...........
----------------------------------------------------------------------
Ran 11 tests in 0.003s

OK

$ ./generateInternalExemplars.py 
.......
----------------------------------------------------------------------
Ran 7 tests in 4.092s

The exemplar binaries can now be imported into two Ghidra projects - one for RISCV64 and another for x86_64. The import process includes pre- and post-script processing. Pre-script processing is used for the kernel import to fix symbol names and load address. Post-script processing is used for the kernel module import to gather relocation results for later regression testing. These relocation results are saved in testresults/*.json

Import processing generates a log file for each binary imported into Ghidra. If that log file is newer than the binary, the import process is skipped. If you want to rerun an import for foo.o, simply delete the matching log file in .../exemplars/foo.log.

OK
$ ./importExemplars.py
.INFO:root:Current Kernel import log file found - skipping import
.INFO:root:Current Kernel module import log file found - skipping import
...
.
----------------------------------------------------------------------
Ran 7 tests in 0.003s

OK

Test results gathered during binary imports and saved in testresults/*.json are now compared with expected values in the final script:

$ ./integrationTest.py 
inspecting the R_RISCV_BRANCH relocation test
inspecting the R_RISCV_JAL test
inspecting the R_RISCV_PCREL_HI20 1/2 test
inspecting the R_RISCV_PCREL_HI20 2/2 test
inspecting the R_RISCV_PCREL_LO12_I test
inspecting the R_RISCV_64 test
inspecting the R_RISCV_RVC_BRANCH test
inspecting the R_ADD_32 test
inspecting the R_RISCV_ADD64 test
inspecting the R_SUB_32 test
inspecting the R_RISCV_ADD64 test
inspecting the R_RISCV_RVC_JUMP test
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

Ghidra gap analysis

We are looking for processor features that may soon be commonplace but that current Ghidra releases do not support well. One such feature involves RISCV extensions to the instruction set architecture, especially vector and bit manipulation extensions. For each such feature we might consider the following questions:

1. What is a current example of this feature, especially examples that support analysis or pathologies of those features.
2. How and when might this feature impact a significant number of Ghidra analysts?
3. How much effort might it take Ghidra developers to fill the implied feature gap?
4. Is this feature specific to RISCV systems or more broadly applicable to other processor families? Would support for that feature be common to many processor families or vary widely by processor?
5. What are the existing frameworks within Ghidra that might most credibly be extended to support that feature?

Thread Local Storage (TLS) is a fairly simple feature we can use as an example. Addressing each of the five questions in turn we might find:

1. TLS relocation codes appear occasionally in multithreaded applications across most processor families. They might appear a few times within libc. Ghidra often doesn’t recognize these codes. Existing analytics like objdump and readelf certainly do recognize TLS codes, but do not pretend to provide semantic aid in interpreting those codes. TLS codes have well documented C source contexts in the form of compiler attributes.
2. The TLS handling gap is unlikely to affect many Ghidra users anytime soon, mostly because they appear only rarely and mostly apply to local variables where the decompiler can provide context.
3. Experienced Ghidra developers might be able to implement the general TLS case easily, but would then have to add supporting ELF import code to a broader range of processor definitions.
4. The TLS feature is common across most processor families supporting multithreading.
5. Support within Ghidra might grow out of existing memory space models and existing processor-specific ELF importers.

The general design questions boil down to:

• how long can we defer working on this gap?
• how long would it take to fill that gap after we got started?
• where would we likely want to start

The Ghidra design team might assign TLS support a relatively low priority, since the gap doesn’t currently have a large impact. If the incidence and complexity of TLS suddenly increased, the extension of existing Ghidra support could likely increase just as rapidly.

Extensions to Instruction Set Architectures make up a much more complicated example. Standardized instructions for cache management and cryptography are likely easy enough to fold into Ghidra’s framework, but vector instruction extensions will hit harder and sooner, without a clear path forward for Ghidra.