The CParser4 Python parser files (CLexer.py, CParser.py, CListener.py)
were generated 7 years ago with ANTLR 4.7.1.
Meanwhile, pip-requirements.txt pins antlr4-python3-runtime to version
4.9 in commit 4a7dd50, but the files were patched, not fully
regenerated. This version mismatch could result in failures when
running against non-trivial C code.
This change regenerates the CParser4 files with ANTLR 4.9 to resolve
the version mismatch. It also updates import statements to correctly
reference Eot instead of Ecc.
Steps used to regenerate the files:
1. Download the ANTLR 4.9 complete tool JAR:
- `https://www.antlr.org/download/antlr-4.9-complete.jar`
2. Generate Python3 parser files
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
The CParser4 Python parser files (CLexer.py, CParser.py, CListener.py)
were generated 7 years ago with ANTLR 4.7.1.
Meanwhile, pip-requirements.txt pins antlr4-python3-runtime to version
4.9 in commit 4a7dd50, but the files were patched, not fully
regenerated. This version mismatch produced two failures when running
EccMain.py against non-trivial C code:
1. A runtime warning on every file parsed:
"ANTLR runtime and generated code versions disagree: 4.9!=4.7.1"
2. A crash when parsing complex C constructs that exercise the
struct/union definition rule in CParser.py:
TypeError: '<' not supported between instances of 'tuple' and 'int'
This occurs in antlr4/BufferedTokenStream.py getText() because the
4.9 runtime changed the expected argument types for that method,
and the 4.7.1-generated parser was passing a tuple where an int is
now required.
This change regenerates the CParser4 files with ANTLR 4.9 to resolve
the version mismatch.
Steps used to regenerate the files:
1. Download the ANTLR 4.9 complete tool JAR:
- `https://www.antlr.org/download/antlr-4.9-complete.jar`
2. Generate Python3 parser files from the grammar:
```
java -jar antlr-4.9-complete.jar `
-Dlanguage=Python3 -visitor `
-o BaseTools/Source/Python/Ecc/CParser4_new `
BaseTools/Source/Python/Ecc/CParser4/C.g4
```
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
The codebase has moved from traditional `#ifndef` include guards to
`#pragma once`. Remove the ECC checks that validated include guard
presence and naming conventions since they are no longer applicable.
The following checks are removed:
- IncludeFileCheckIfndefStatement: Verified all header file contents
were guarded by a `#ifndef` statement, that the `#ifndef` was the
first line of code after the file header comment, and that the
`#endif` appeared on the last line.
- NamingConventionCheckIfndefStatement: Verified that the `#ifndef`
guard name at the start of an include file used a postfix underscore
and no prefix underscore character.
Also removed related error codes and configuration settings that were
specific to these checks.
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
This reverts commit 3fe1d56cc9.
PR https://github.com/tianocore/edk2/pull/11757 introduced a
"Breaking Change" feature for out of tree builds of tools.
This breaking change is blocking testing of edk2-stable202602
due to side effects on building FitGen tool in edk2-platforms.
Revert this feature for the edk2-stable202602 release and
work on this feature after the release.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Main EDK2 build supports out-of-tree builds but BaseTools make process
still creates tools and object files in-tree. In order to make
out-of-tree build support complete move the generated tools and
interim obj files to $WORKSPACE location as well.
This patch also changes the location of BaseTools for in-tree builds
(default behavior when WORKSPACE is not provided before calling
edksetup) to $WORKSPACE/BaseTools/Build/... It may potentially break
external workflows that invoke tools from the default location outside
of the build tool.
Signed-off-by: Oleksandr Tymoshenko <ovt@google.com>
The PCD value defined in module subsections can be added to global PCD
database. Therefore the unsolved expressions, even belongs to the global
scope, can incorrectly refer to the value from module subsection.
This only happens when the referred PCD has no value assignment in the
platform dsc file. Which also should raise an error.
Signed-off-by: Paddy Deng <paddydeng@ami.com>
The stuart tools automatically add -D WIN_HOST_BUILD to
edk2 build command line if a Windows build environment
is detected. This behavior is added to build.py so that
builds of the EmulatorPkg using build.py are not required
to add the option -D WIN_HOST_BUILD when building in a
Windows environment. This aligns Linux and Windows builds
of the EmulatorPkg removing the need to specify extra
defines.
In order to build the EmulatorPkg for Windows Mingw
environments, EmulatorPkg DSC/FDF files require a way
to detect if Windows Mingw environment is present.
The Windows Mingw environment can be detected if
CLANG_BIN is set and mingw32-make.exe is detected
in the CLANG_BIN directory.
If a Windows Mingw environment is detected, add
-D WIN_MINGW32_BUILD to the edk2 build command line.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
Use $(OS) in all GNUMakefiles to detect if the GNUMakefile
is being used in a Windows OS. If a Windows OS is detected,
then override SHELL to use cmd.exe. This prevents make
utility from using sh.exe if sh.exe happens to be in PATH.
If sh.exe is used, then backslash (\) characters in file
paths are removed and builds break for files not found.
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
When the number of build threads multiplied by per-thread file
descriptor usage exceeds the system's open file descriptor limit,
some threads may fail to acquire necessary resources (e.g., pipes
or semaphores), leading to deadlocks or hangs during parallel builds.
To prevent this situation, calculate the safety upper limit of
concurrency by dividing the system's maximum file descriptor limit by
3 (An empirical value derived from balancing performance overhead
against the theoretical number of file descriptors consumed per thread).
The actual thread count is then clamped to this safe value.
Other usages of ThreadNum()—such as during actual compilation or log
queue creation—do not significantly contribute to file descriptor
consumption. Therefore, adjusting ThreadNum() globally would be
unwarranted, as it could unnecessarily restrict parallelism in stages
that are not FD-bound.
This ensures stable parallel builds even under constrained resource
limits.
Signed-off-by: Ayden Meng <mengxiangdong@loongson.cn>
Previously, when file descriptors were exhausted in high-concurrency
builds (e.g., 512 threads with 1024 FD limit), the build would hang or
fail silently without clear indication of the root cause.
This change catches relevant OSError instances and terminates the build,
ensuring failures due to resource limits are explicit.
Signed-off-by: Ayden Meng <mengxiangdong@loongson.cn>
Replace manual loop-based tuple construction with the built-in tuple()
function when converting PcdDbBuffer to a tuple. This change
significantly improves performance—approximately three times
faster—resulting in substantial build time savings in large
environments.
Previously, the code iterated over each byte in PcdDbBuffer, unpacking
and appending it to a tuple. The new approach leverages
tuple(PcdDbBuffer) to achieve the same result more efficiently. The
generated tuple remains identical to the original implementation.
TEST=The generated tuple is the same than with to original code
Signed-off-by: Jeremy Compostella <jeremy.compostella@intel.com>
1.Add FFS file type check: Only allow rebase operation for
EFI_FV_FILETYPE_SECURITY_CORE, EFI_FV_FILETYPE_PEI_CORE,
EFI_FV_FILETYPE_DXE_CORE, EFI_FV_FILETYPE_PEIM, EFI_FV_FILETYPE_DRIVER,
EFI_FV_FILETYPE_COMBINED_PEIM_DRIVER,EFI_FV_FILETYPE_FIRMWARE_VOLUME_IMAGE
types, improving safety and compliance.
2.Automatically detect and complete the PE/COFF or TE image
relocation table (reloc section) to ensure integrity and compatibility
of the rebase operation.
3.After rebase, automatically update FFS checksum and FV header
information to ensure correct data structure.
4.Support recursive processing for nested FVs, ensuring all related FFS
files' PE/TE images are properly rebased and reloc tables are completed.
5.Use table-driven architecture for relocation types, making it easier to
extend to more platforms.
6.Improve error handling and logging for better robustness and
maintainability.
Please attention, only IA32 and X64 PE/COFF image are supported now.
For other Arch, will support it after testing.
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
This patch adds and improves the rebase functionality
for firmware images (PE/COFF/TE) in the FMMT tool.
Key features include:
1.Automatically rebases PE/COFF/TE images within FFS files
when the firmware volume (FV) layout is adjusted or FFS files
are moved, ensuring correct loading and execution at new addresses.
2.Implements recursive rebase logic for nested sections,
guaranteeing all relevant images are properly relocated.
3.Adds support for rebasing subsequent FFS files within the same FV,
enhancing compatibility and stability during firmware layout changes.
4.Core code changes are mainly in FvHandler.py, BiosTreeNode.py,
and BinaryFactoryProduct.py, covering rebase flag detection, address calculation,
and actual relocation operations.
5.This feature improves the flexibility of firmware space management
and enhances the automation and reliability of the FMMT tool.
Co-Auther: Ashraf Ali S <ashraf.ali.s@intel.com>
Signed-off-by: Yuwei Chen <yuwei.chen@intel.com>
There is another issue in FvHander.py similar to the one fixed by
a60334ad59 ("BaseTools: Fix FMMT FvHandler Padding operation issue").
For a guided section (ParTree.Data.Type == 0x02), the length of
ParTree.Data.OriData is used to obtain the original size of the data even
after ParTree.Data.OriData has changed, which is incorrect.
This caused the following issue I observed. I built OVMF image for Aarch64
and then tried to add a couple FFS modules to it with 'FMMT -a [...]'. The
resulting image turned out to be invalid: the total size of the image was
less than the size of the firmware volume within it.
UEFITool failed to load such firmware image and complained:
"parseRawArea: one of objects inside overlaps the end of data".
This patch fixes the issue.
Signed-off-by: Evgenii Shatokhin <euspectre@gmail.com>
This commit is the first in a series of multiple commits to fix various
typos in the code, originating mostly from copy&paste over the years.
Most of them only affect documentation and not code.
Signed-off-by: Philipp Schuster <philipp.schuster@cyberus-technology.de>
The DXE_SAL_DRIVER module type was introduced to support
Itanium (IPF) platforms. Since support for Itanium processors has been
dropped, the instances of DXE_SAL_DRIVER have been removed.
Ref: [3cb0a311cb]
Cc: Sachin Ganesh <sachinganesh@ami.com>
Signed-off-by: Sathya Ravichandran <sathyar@ami.com>
- Str is not being used in FvHeader.py
- Removed import of ast.Str as it was removed in Python 3.14.
- Ensures compatibility with Python 3.14 and later.
- https://docs.python.org/3/whatsnew/3.14.html#id9
This addresses ImportError caused by removal of deprecated AST classes
including ast.Str.
Signed-off-by: Ashraf Ali S <ashraf.ali.s@intel.com>
Currently the FMMT tool supports CRC32 GUID section (Ref. GuidTools.py).
But it is found that there are errors for the FV with CRC32 section.
For example, the errors are checked on the following commands.
--v : Show FV information except for the FV
--a : Show "Target Fv not found!!!" when adding an FFS file to the FV
--d : Show "Target Ffs not found!!!" when deleting an FFS file in the FV
They are caused by the mismatch for CRC32 section data between the FMMT
and GenCrc32 tools. The FMMT tool returns CRC32 section data without CRC
checksum field (4 bytes). The GenCrc32 tool (with -d option, verify CRC32
value for the input file) requires CRC32 section data including CRC
checksum field (4 bytes).
Fix the issue through adjusting the section data to include CRC checksum
field. Currently DataOffset field for CRC32 GUID section is reported as
0x1C differently (GUID Section header length: 0x18 + Checksum field: 0x4).
Instead of DataOffset field that includes CRC checksum field, configure
the section data based on the offset from section header length (0x18)
that was previously calculated. This update enables GUID sections to use
the same offset consistently.
Signed-off-by: Phil Noh <Phil.Noh@amd.com>
Set BUILD_TIME_EPOCH to the current UTC timestamp if not already present
in the environment.
Use the resulting value to print the "Build start time:" message.
Signed-off-by: Leif Lindholm <leif.lindholm@oss.qualcomm.com>
The function CheckEnvVariable in fact checks several environment
variables. And the comment at its invocation enumerates a specific set of
variables, which defeats half the point of abstracting it out into a
helper function.
Rename the function to the plural form and turn the comment into a list
of examples.
Signed-off-by: Leif Lindholm <leif.lindholm@oss.qualcomm.com>
edk2 is dropping support for the ARM32 architecture. This
commit removes ARM32 code from BaseTools.
Signed-off-by: Oliver Smith-Denny <osde@microsoft.com>
Commit f0a2015373 ("UefiPayloadPkg: Add AARCH64 support") changed
`[Components.X64]` to `[Components.X64, Components.AARCH64]` which
resulted in the following code within that section to not work as
expected (the code wasn't there, just providing a real world example
that uncovered the issue):
[Components.X64, Components.AARCH64]
FmpDevicePkg/FmpDxe/FmpDxe.inf {
...
<PcdsFixedAtBuild>
!include .../...PcdFmpDevicePkcs7CertBufferXdr.inc
...
}
At the same time `[Components.X64]` or even `[Components.AARCH64,
Components.X64]` (notice the swapped order) worked fine for X64 target.
The cause of the issue turned out to be skipping includes inside
`_PostProcess()` method of `DscParser` class. This method processes
list of items stored in a database filled on the first pass through a
DSC file in `Start()` method. One of the fields stored in each row
of a table is link to a parent object (owner). A section like
`[Components.X64, Components.AARCH64]` creates two objects and all of
its subelements are duplicated for both X64 and AARCH64. This was not
happening for !include statement in the example above.
Because `_PostProcess()` contracted a sequence of !include objects
disregarding their owner, it did not create instance for each of the
requested targets. Codewise, `self._ContentIndex` was incremented more
than once, while `__ProcessDirective()` method (invoked indirectly as
`Processer[self._ItemType]()`) queried owner of the current directive
as:
if self._InSubsection:
Owner = self._Content[self._ContentIndex - 1][8]
else: # not taken in this case
This is why order of targets made a difference, only the last was fully
initialized in this case.
An alternative fix is completely dropping merging of !include
directives, but hard to say whether it still has some utility (the code
is complex, hard to follow and barely documented). Safer to keep it, in
the worst case it doesn't do anything now.
Signed-off-by: Sergii Dmytruk <sergii.dmytruk@3mdeb.com>
