mono/linux-packaging-mono
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Imported Upstream version 5.18.0.207

Browse Source 
Former-commit-id: 3b152f462918d427ce18620a2cbe4f8b79650449
This commit is contained in:
Xamarin Public Jenkins (auto-signing)
2018-11-17 08:23:10 +00:00
parent 8e12397d70
commit eb85e2fc17
28480 changed files with 72 additions and 3866936 deletions
Download Patch File Download Diff File Expand all files Collapse all files

398
external/llvm/docs/tutorial/BuildingAJIT1.rst vendored
View File

@ -1,398 +0,0 @@
=======================================================
Building a JIT: Starting out with KaleidoscopeJIT
=======================================================
.. contents::
:local:
Chapter 1 Introduction
======================
Welcome to Chapter 1 of the "Building an ORC-based JIT in LLVM" tutorial. This
tutorial runs through the implementation of a JIT compiler using LLVM's
On-Request-Compilation (ORC) APIs. It begins with a simplified version of the
KaleidoscopeJIT class used in the
`Implementing a language with LLVM <LangImpl01.html>`_ tutorials and then
introduces new features like optimization, lazy compilation and remote
execution.
The goal of this tutorial is to introduce you to LLVM's ORC JIT APIs, show how
these APIs interact with other parts of LLVM, and to teach you how to recombine
them to build a custom JIT that is suited to your use-case.
The structure of the tutorial is:
- Chapter #1: Investigate the simple KaleidoscopeJIT class. This will
introduce some of the basic concepts of the ORC JIT APIs, including the
idea of an ORC *Layer*.
- `Chapter #2 <BuildingAJIT2.html>`_: Extend the basic KaleidoscopeJIT by adding
a new layer that will optimize IR and generated code.
- `Chapter #3 <BuildingAJIT3.html>`_: Further extend the JIT by adding a
Compile-On-Demand layer to lazily compile IR.
- `Chapter #4 <BuildingAJIT4.html>`_: Improve the laziness of our JIT by
replacing the Compile-On-Demand layer with a custom layer that uses the ORC
Compile Callbacks API directly to defer IR-generation until functions are
called.
- `Chapter #5 <BuildingAJIT5.html>`_: Add process isolation by JITing code into
a remote process with reduced privileges using the JIT Remote APIs.
To provide input for our JIT we will use the Kaleidoscope REPL from
`Chapter 7 <LangImpl07.html>`_ of the "Implementing a language in LLVM tutorial",
with one minor modification: We will remove the FunctionPassManager from the
code for that chapter and replace it with optimization support in our JIT class
in Chapter #2.
Finally, a word on API generations: ORC is the 3rd generation of LLVM JIT API.
It was preceded by MCJIT, and before that by the (now deleted) legacy JIT.
These tutorials don't assume any experience with these earlier APIs, but
readers acquainted with them will see many familiar elements. Where appropriate
we will make this connection with the earlier APIs explicit to help people who
are transitioning from them to ORC.
JIT API Basics
==============
The purpose of a JIT compiler is to compile code "on-the-fly" as it is needed,
rather than compiling whole programs to disk ahead of time as a traditional
compiler does. To support that aim our initial, bare-bones JIT API will be:
1. Handle addModule(Module &M) -- Make the given IR module available for
execution.
2. JITSymbol findSymbol(const std::string &Name) -- Search for pointers to
symbols (functions or variables) that have been added to the JIT.
3. void removeModule(Handle H) -- Remove a module from the JIT, releasing any
memory that had been used for the compiled code.
A basic use-case for this API, executing the 'main' function from a module,
will look like:
.. code-block:: c++
std::unique_ptr<Module> M = buildModule();
JIT J;
Handle H = J.addModule(*M);
int (*Main)(int, char*[]) = (int(*)(int, char*[]))J.getSymbolAddress("main");
int Result = Main();
J.removeModule(H);
The APIs that we build in these tutorials will all be variations on this simple
theme. Behind the API we will refine the implementation of the JIT to add
support for optimization and lazy compilation. Eventually we will extend the
API itself to allow higher-level program representations (e.g. ASTs) to be
added to the JIT.
KaleidoscopeJIT
===============
In the previous section we described our API, now we examine a simple
implementation of it: The KaleidoscopeJIT class [1]_ that was used in the
`Implementing a language with LLVM <LangImpl01.html>`_ tutorials. We will use
the REPL code from `Chapter 7 <LangImpl07.html>`_ of that tutorial to supply the
input for our JIT: Each time the user enters an expression the REPL will add a
new IR module containing the code for that expression to the JIT. If the
expression is a top-level expression like '1+1' or 'sin(x)', the REPL will also
use the findSymbol method of our JIT class find and execute the code for the
expression, and then use the removeModule method to remove the code again
(since there's no way to re-invoke an anonymous expression). In later chapters
of this tutorial we'll modify the REPL to enable new interactions with our JIT
class, but for now we will take this setup for granted and focus our attention on
the implementation of our JIT itself.
Our KaleidoscopeJIT class is defined in the KaleidoscopeJIT.h header. After the
usual include guards and #includes [2]_, we get to the definition of our class:
.. code-block:: c++
#ifndef LLVM_EXECUTIONENGINE_ORC_KALEIDOSCOPEJIT_H
#define LLVM_EXECUTIONENGINE_ORC_KALEIDOSCOPEJIT_H
#include "llvm/ADT/STLExtras.h"
#include "llvm/ExecutionEngine/ExecutionEngine.h"
#include "llvm/ExecutionEngine/JITSymbol.h"
#include "llvm/ExecutionEngine/RTDyldMemoryManager.h"
#include "llvm/ExecutionEngine/SectionMemoryManager.h"
#include "llvm/ExecutionEngine/Orc/CompileUtils.h"
#include "llvm/ExecutionEngine/Orc/IRCompileLayer.h"
#include "llvm/ExecutionEngine/Orc/LambdaResolver.h"
#include "llvm/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/Mangler.h"
#include "llvm/Support/DynamicLibrary.h"
#include "llvm/Support/raw_ostream.h"
#include "llvm/Target/TargetMachine.h"
#include <algorithm>
#include <memory>
#include <string>
#include <vector>
namespace llvm {
namespace orc {
class KaleidoscopeJIT {
private:
std::unique_ptr<TargetMachine> TM;
const DataLayout DL;
RTDyldObjectLinkingLayer ObjectLayer;
IRCompileLayer<decltype(ObjectLayer), SimpleCompiler> CompileLayer;
public:
using ModuleHandle = decltype(CompileLayer)::ModuleHandleT;
Our class begins with four members: A TargetMachine, TM, which will be used to
build our LLVM compiler instance; A DataLayout, DL, which will be used for
symbol mangling (more on that later), and two ORC *layers*: an
RTDyldObjectLinkingLayer and a CompileLayer. We'll be talking more about layers
in the next chapter, but for now you can think of them as analogous to LLVM
Passes: they wrap up useful JIT utilities behind an easy to compose interface.
The first layer, ObjectLayer, is the foundation of our JIT: it takes in-memory
object files produced by a compiler and links them on the fly to make them
executable. This JIT-on-top-of-a-linker design was introduced in MCJIT, however
the linker was hidden inside the MCJIT class. In ORC we expose the linker so
that clients can access and configure it directly if they need to. In this
tutorial our ObjectLayer will just be used to support the next layer in our
stack: the CompileLayer, which will be responsible for taking LLVM IR, compiling
it, and passing the resulting in-memory object files down to the object linking
layer below.
That's it for member variables, after that we have a single typedef:
ModuleHandle. This is the handle type that will be returned from our JIT's
addModule method, and can be passed to the removeModule method to remove a
module. The IRCompileLayer class already provides a convenient handle type
(IRCompileLayer::ModuleHandleT), so we just alias our ModuleHandle to this.
.. code-block:: c++
KaleidoscopeJIT()
: TM(EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
ObjectLayer([]() { return std::make_shared<SectionMemoryManager>(); }),
CompileLayer(ObjectLayer, SimpleCompiler(*TM)) {
llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
}
TargetMachine &getTargetMachine() { return *TM; }
Next up we have our class constructor. We begin by initializing TM using the
EngineBuilder::selectTarget helper method which constructs a TargetMachine for
the current process. Then we use our newly created TargetMachine to initialize
DL, our DataLayout. After that we need to initialize our ObjectLayer. The
ObjectLayer requires a function object that will build a JIT memory manager for
each module that is added (a JIT memory manager manages memory allocations,
memory permissions, and registration of exception handlers for JIT'd code). For
this we use a lambda that returns a SectionMemoryManager, an off-the-shelf
utility that provides all the basic memory management functionality required for
this chapter. Next we initialize our CompileLayer. The CompileLayer needs two
things: (1) A reference to our object layer, and (2) a compiler instance to use
to perform the actual compilation from IR to object files. We use the
off-the-shelf SimpleCompiler instance for now. Finally, in the body of the
constructor, we call the DynamicLibrary::LoadLibraryPermanently method with a
nullptr argument. Normally the LoadLibraryPermanently method is called with the
path of a dynamic library to load, but when passed a null pointer it will 'load'
the host process itself, making its exported symbols available for execution.
.. code-block:: c++
ModuleHandle addModule(std::unique_ptr<Module> M) {
// Build our symbol resolver:
// Lambda 1: Look back into the JIT itself to find symbols that are part of
// the same "logical dylib".
// Lambda 2: Search for external symbols in the host process.
auto Resolver = createLambdaResolver(
[&](const std::string &Name) {
if (auto Sym = CompileLayer.findSymbol(Name, false))
return Sym;
return JITSymbol(nullptr);
},
[](const std::string &Name) {
if (auto SymAddr =
RTDyldMemoryManager::getSymbolAddressInProcess(Name))
return JITSymbol(SymAddr, JITSymbolFlags::Exported);
return JITSymbol(nullptr);
});
// Add the set to the JIT with the resolver we created above and a newly
// created SectionMemoryManager.
return cantFail(CompileLayer.addModule(std::move(M),
std::move(Resolver)));
}
Now we come to the first of our JIT API methods: addModule. This method is
responsible for adding IR to the JIT and making it available for execution. In
this initial implementation of our JIT we will make our modules "available for
execution" by adding them straight to the CompileLayer, which will immediately
compile them. In later chapters we will teach our JIT to defer compilation
of individual functions until they're actually called.
To add our module to the CompileLayer we need to supply both the module and a
symbol resolver. The symbol resolver is responsible for supplying the JIT with
an address for each *external symbol* in the module we are adding. External
symbols are any symbol not defined within the module itself, including calls to
functions outside the JIT and calls to functions defined in other modules that
have already been added to the JIT. (It may seem as though modules added to the
JIT should know about one another by default, but since we would still have to
supply a symbol resolver for references to code outside the JIT it turns out to
be easier to re-use this one mechanism for all symbol resolution.) This has the
added benefit that the user has full control over the symbol resolution
process. Should we search for definitions within the JIT first, then fall back
on external definitions? Or should we prefer external definitions where
available and only JIT code if we don't already have an available
implementation? By using a single symbol resolution scheme we are free to choose
whatever makes the most sense for any given use case.
Building a symbol resolver is made especially easy by the *createLambdaResolver*
function. This function takes two lambdas [3]_ and returns a JITSymbolResolver
instance. The first lambda is used as the implementation of the resolver's
findSymbolInLogicalDylib method, which searches for symbol definitions that
should be thought of as being part of the same "logical" dynamic library as this
Module. If you are familiar with static linking: this means that
findSymbolInLogicalDylib should expose symbols with common linkage and hidden
visibility. If all this sounds foreign you can ignore the details and just
remember that this is the first method that the linker will use to try to find a
symbol definition. If the findSymbolInLogicalDylib method returns a null result
then the linker will call the second symbol resolver method, called findSymbol,
which searches for symbols that should be thought of as external to (but
visibile from) the module and its logical dylib. In this tutorial we will adopt
the following simple scheme: All modules added to the JIT will behave as if they
were linked into a single, ever-growing logical dylib. To implement this our
first lambda (the one defining findSymbolInLogicalDylib) will just search for
JIT'd code by calling the CompileLayer's findSymbol method. If we don't find a
symbol in the JIT itself we'll fall back to our second lambda, which implements
findSymbol. This will use the RTDyldMemoryManager::getSymbolAddressInProcess
method to search for the symbol within the program itself. If we can't find a
symbol definition via either of these paths, the JIT will refuse to accept our
module, returning a "symbol not found" error.
Now that we've built our symbol resolver, we're ready to add our module to the
JIT. We do this by calling the CompileLayer's addModule method. The addModule
method returns an ``Expected<CompileLayer::ModuleHandle>``, since in more
advanced JIT configurations it could fail. In our basic configuration we know
that it will always succeed so we use the cantFail utility to assert that no
error occurred, and extract the handle value. Since we have already typedef'd
our ModuleHandle type to be the same as the CompileLayer's handle type, we can
return the unwrapped handle directly.
.. code-block:: c++
JITSymbol findSymbol(const std::string Name) {
std::string MangledName;
raw_string_ostream MangledNameStream(MangledName);
Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
return CompileLayer.findSymbol(MangledNameStream.str(), true);
}
JITTargetAddress getSymbolAddress(const std::string Name) {
return cantFail(findSymbol(Name).getAddress());
}
void removeModule(ModuleHandle H) {
cantFail(CompileLayer.removeModule(H));
}
Now that we can add code to our JIT, we need a way to find the symbols we've
added to it. To do that we call the findSymbol method on our CompileLayer, but
with a twist: We have to *mangle* the name of the symbol we're searching for
first. The ORC JIT components use mangled symbols internally the same way a
static compiler and linker would, rather than using plain IR symbol names. This
allows JIT'd code to interoperate easily with precompiled code in the
application or shared libraries. The kind of mangling will depend on the
DataLayout, which in turn depends on the target platform. To allow us to remain
portable and search based on the un-mangled name, we just re-produce this
mangling ourselves.
Next we have a convenience function, getSymbolAddress, which returns the address
of a given symbol. Like CompileLayer's addModule function, JITSymbol's getAddress
function is allowed to fail [4]_, however we know that it will not in our simple
example, so we wrap it in a call to cantFail.
We now come to the last method in our JIT API: removeModule. This method is
responsible for destructing the MemoryManager and SymbolResolver that were
added with a given module, freeing any resources they were using in the
process. In our Kaleidoscope demo we rely on this method to remove the module
representing the most recent top-level expression, preventing it from being
treated as a duplicate definition when the next top-level expression is
entered. It is generally good to free any module that you know you won't need
to call further, just to free up the resources dedicated to it. However, you
don't strictly need to do this: All resources will be cleaned up when your
JIT class is destructed, if they haven't been freed before then. Like
``CompileLayer::addModule`` and ``JITSymbol::getAddress``, removeModule may
fail in general but will never fail in our example, so we wrap it in a call to
cantFail.
This brings us to the end of Chapter 1 of Building a JIT. You now have a basic
but fully functioning JIT stack that you can use to take LLVM IR and make it
executable within the context of your JIT process. In the next chapter we'll
look at how to extend this JIT to produce better quality code, and in the
process take a deeper look at the ORC layer concept.
`Next: Extending the KaleidoscopeJIT <BuildingAJIT2.html>`_
Full Code Listing
=================
Here is the complete code listing for our running example. To build this
example, use:
.. code-block:: bash
# Compile
clang++ -g toy.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core orcjit native` -O3 -o toy
# Run
./toy
Here is the code:
.. literalinclude:: ../../examples/Kaleidoscope/BuildingAJIT/Chapter1/KaleidoscopeJIT.h
:language: c++
.. [1] Actually we use a cut-down version of KaleidoscopeJIT that makes a
simplifying assumption: symbols cannot be re-defined. This will make it
impossible to re-define symbols in the REPL, but will make our symbol
lookup logic simpler. Re-introducing support for symbol redefinition is
left as an exercise for the reader. (The KaleidoscopeJIT.h used in the
original tutorials will be a helpful reference).
.. [2] +-----------------------------+-----------------------------------------------+
| File | Reason for inclusion |
+=============================+===============================================+
| STLExtras.h | LLVM utilities that are useful when working |
| | with the STL. |
+-----------------------------+-----------------------------------------------+
| ExecutionEngine.h | Access to the EngineBuilder::selectTarget |
| | method. |
+-----------------------------+-----------------------------------------------+
| | Access to the |
| RTDyldMemoryManager.h | RTDyldMemoryManager::getSymbolAddressInProcess|
| | method. |
+-----------------------------+-----------------------------------------------+
| CompileUtils.h | Provides the SimpleCompiler class. |
+-----------------------------+-----------------------------------------------+
| IRCompileLayer.h | Provides the IRCompileLayer class. |
+-----------------------------+-----------------------------------------------+
| | Access the createLambdaResolver function, |
| LambdaResolver.h | which provides easy construction of symbol |
| | resolvers. |
+-----------------------------+-----------------------------------------------+
| RTDyldObjectLinkingLayer.h | Provides the RTDyldObjectLinkingLayer class. |
+-----------------------------+-----------------------------------------------+
| Mangler.h | Provides the Mangler class for platform |
| | specific name-mangling. |
+-----------------------------+-----------------------------------------------+
| DynamicLibrary.h | Provides the DynamicLibrary class, which |
| | makes symbols in the host process searchable. |
+-----------------------------+-----------------------------------------------+
| | A fast output stream class. We use the |
| raw_ostream.h | raw_string_ostream subclass for symbol |
| | mangling |
+-----------------------------+-----------------------------------------------+
| TargetMachine.h | LLVM target machine description class. |
+-----------------------------+-----------------------------------------------+
.. [3] Actually they don't have to be lambdas, any object with a call operator
will do, including plain old functions or std::functions.
.. [4] ``JITSymbol::getAddress`` will force the JIT to compile the definition of
the symbol if it hasn't already been compiled, and since the compilation
process could fail getAddress must be able to return this failure.

329
external/llvm/docs/tutorial/BuildingAJIT2.rst vendored
View File

@ -1,329 +0,0 @@
=====================================================================
Building a JIT: Adding Optimizations -- An introduction to ORC Layers
=====================================================================
.. contents::
:local:
**This tutorial is under active development. It is incomplete and details may
change frequently.** Nonetheless we invite you to try it out as it stands, and
we welcome any feedback.
Chapter 2 Introduction
======================
Welcome to Chapter 2 of the "Building an ORC-based JIT in LLVM" tutorial. In
`Chapter 1 <BuildingAJIT1.html>`_ of this series we examined a basic JIT
class, KaleidoscopeJIT, that could take LLVM IR modules as input and produce
executable code in memory. KaleidoscopeJIT was able to do this with relatively
little code by composing two off-the-shelf *ORC layers*: IRCompileLayer and
ObjectLinkingLayer, to do much of the heavy lifting.
In this layer we'll learn more about the ORC layer concept by using a new layer,
IRTransformLayer, to add IR optimization support to KaleidoscopeJIT.
Optimizing Modules using the IRTransformLayer
=============================================
In `Chapter 4 <LangImpl04.html>`_ of the "Implementing a language with LLVM"
tutorial series the llvm *FunctionPassManager* is introduced as a means for
optimizing LLVM IR. Interested readers may read that chapter for details, but
in short: to optimize a Module we create an llvm::FunctionPassManager
instance, configure it with a set of optimizations, then run the PassManager on
a Module to mutate it into a (hopefully) more optimized but semantically
equivalent form. In the original tutorial series the FunctionPassManager was
created outside the KaleidoscopeJIT and modules were optimized before being
added to it. In this Chapter we will make optimization a phase of our JIT
instead. For now this will provide us a motivation to learn more about ORC
layers, but in the long term making optimization part of our JIT will yield an
important benefit: When we begin lazily compiling code (i.e. deferring
compilation of each function until the first time it's run), having
optimization managed by our JIT will allow us to optimize lazily too, rather
than having to do all our optimization up-front.
To add optimization support to our JIT we will take the KaleidoscopeJIT from
Chapter 1 and compose an ORC *IRTransformLayer* on top. We will look at how the
IRTransformLayer works in more detail below, but the interface is simple: the
constructor for this layer takes a reference to the layer below (as all layers
do) plus an *IR optimization function* that it will apply to each Module that
is added via addModule:
.. code-block:: c++
class KaleidoscopeJIT {
private:
std::unique_ptr<TargetMachine> TM;
const DataLayout DL;
RTDyldObjectLinkingLayer<> ObjectLayer;
IRCompileLayer<decltype(ObjectLayer)> CompileLayer;
using OptimizeFunction =
std::function<std::shared_ptr<Module>(std::shared_ptr<Module>)>;
IRTransformLayer<decltype(CompileLayer), OptimizeFunction> OptimizeLayer;
public:
using ModuleHandle = decltype(OptimizeLayer)::ModuleHandleT;
KaleidoscopeJIT()
: TM(EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
ObjectLayer([]() { return std::make_shared<SectionMemoryManager>(); }),
CompileLayer(ObjectLayer, SimpleCompiler(*TM)),
OptimizeLayer(CompileLayer,
[this](std::unique_ptr<Module> M) {
return optimizeModule(std::move(M));
}) {
llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
}
Our extended KaleidoscopeJIT class starts out the same as it did in Chapter 1,
but after the CompileLayer we introduce a typedef for our optimization function.
In this case we use a std::function (a handy wrapper for "function-like" things)
from a single unique_ptr<Module> input to a std::unique_ptr<Module> output. With
our optimization function typedef in place we can declare our OptimizeLayer,
which sits on top of our CompileLayer.
To initialize our OptimizeLayer we pass it a reference to the CompileLayer
below (standard practice for layers), and we initialize the OptimizeFunction
using a lambda that calls out to an "optimizeModule" function that we will
define below.
.. code-block:: c++
// ...
auto Resolver = createLambdaResolver(
[&](const std::string &Name) {
if (auto Sym = OptimizeLayer.findSymbol(Name, false))
return Sym;
return JITSymbol(nullptr);
},
// ...
.. code-block:: c++
// ...
return cantFail(OptimizeLayer.addModule(std::move(M),
std::move(Resolver)));
// ...
.. code-block:: c++
// ...
return OptimizeLayer.findSymbol(MangledNameStream.str(), true);
// ...
.. code-block:: c++
// ...
cantFail(OptimizeLayer.removeModule(H));
// ...
Next we need to replace references to 'CompileLayer' with references to
OptimizeLayer in our key methods: addModule, findSymbol, and removeModule. In
addModule we need to be careful to replace both references: the findSymbol call
inside our resolver, and the call through to addModule.
.. code-block:: c++
std::shared_ptr<Module> optimizeModule(std::shared_ptr<Module> M) {
// Create a function pass manager.
auto FPM = llvm::make_unique<legacy::FunctionPassManager>(M.get());
// Add some optimizations.
FPM->add(createInstructionCombiningPass());
FPM->add(createReassociatePass());
FPM->add(createGVNPass());
FPM->add(createCFGSimplificationPass());
FPM->doInitialization();
// Run the optimizations over all functions in the module being added to
// the JIT.
for (auto &F : *M)
FPM->run(F);
return M;
}
At the bottom of our JIT we add a private method to do the actual optimization:
*optimizeModule*. This function sets up a FunctionPassManager, adds some passes
to it, runs it over every function in the module, and then returns the mutated
module. The specific optimizations are the same ones used in
`Chapter 4 <LangImpl04.html>`_ of the "Implementing a language with LLVM"
tutorial series. Readers may visit that chapter for a more in-depth
discussion of these, and of IR optimization in general.
And that's it in terms of changes to KaleidoscopeJIT: When a module is added via
addModule the OptimizeLayer will call our optimizeModule function before passing
the transformed module on to the CompileLayer below. Of course, we could have
called optimizeModule directly in our addModule function and not gone to the
bother of using the IRTransformLayer, but doing so gives us another opportunity
to see how layers compose. It also provides a neat entry point to the *layer*
concept itself, because IRTransformLayer turns out to be one of the simplest
implementations of the layer concept that can be devised:
.. code-block:: c++
template <typename BaseLayerT, typename TransformFtor>
class IRTransformLayer {
public:
using ModuleHandleT = typename BaseLayerT::ModuleHandleT;
IRTransformLayer(BaseLayerT &BaseLayer,
TransformFtor Transform = TransformFtor())
: BaseLayer(BaseLayer), Transform(std::move(Transform)) {}
Expected<ModuleHandleT>
addModule(std::shared_ptr<Module> M,
std::shared_ptr<JITSymbolResolver> Resolver) {
return BaseLayer.addModule(Transform(std::move(M)), std::move(Resolver));
}
void removeModule(ModuleHandleT H) { BaseLayer.removeModule(H); }
JITSymbol findSymbol(const std::string &Name, bool ExportedSymbolsOnly) {
return BaseLayer.findSymbol(Name, ExportedSymbolsOnly);
}
JITSymbol findSymbolIn(ModuleHandleT H, const std::string &Name,
bool ExportedSymbolsOnly) {
return BaseLayer.findSymbolIn(H, Name, ExportedSymbolsOnly);
}
void emitAndFinalize(ModuleHandleT H) {
BaseLayer.emitAndFinalize(H);
}
TransformFtor& getTransform() { return Transform; }
const TransformFtor& getTransform() const { return Transform; }
private:
BaseLayerT &BaseLayer;
TransformFtor Transform;
};
This is the whole definition of IRTransformLayer, from
``llvm/include/llvm/ExecutionEngine/Orc/IRTransformLayer.h``, stripped of its
comments. It is a template class with two template arguments: ``BaesLayerT`` and
``TransformFtor`` that provide the type of the base layer and the type of the
"transform functor" (in our case a std::function) respectively. This class is
concerned with two very simple jobs: (1) Running every IR Module that is added
with addModule through the transform functor, and (2) conforming to the ORC
layer interface. The interface consists of one typedef and five methods:
+------------------+-----------------------------------------------------------+
| Interface | Description |
+==================+===========================================================+
| | Provides a handle that can be used to identify a module |
| ModuleHandleT | set when calling findSymbolIn, removeModule, or |
| | emitAndFinalize. |
+------------------+-----------------------------------------------------------+
| | Takes a given set of Modules and makes them "available |
| | for execution. This means that symbols in those modules |
| | should be searchable via findSymbol and findSymbolIn, and |
| | the address of the symbols should be read/writable (for |
| | data symbols), or executable (for function symbols) after |
| | JITSymbol::getAddress() is called. Note: This means that |
| addModule | addModule doesn't have to compile (or do any other |
| | work) up-front. It *can*, like IRCompileLayer, act |
| | eagerly, but it can also simply record the module and |
| | take no further action until somebody calls |
| | JITSymbol::getAddress(). In IRTransformLayer's case |
| | addModule eagerly applies the transform functor to |
| | each module in the set, then passes the resulting set |
| | of mutated modules down to the layer below. |
+------------------+-----------------------------------------------------------+
| | Removes a set of modules from the JIT. Code or data |
| removeModule | defined in these modules will no longer be available, and |
| | the memory holding the JIT'd definitions will be freed. |
+------------------+-----------------------------------------------------------+
| | Searches for the named symbol in all modules that have |
| | previously been added via addModule (and not yet |
| findSymbol | removed by a call to removeModule). In |
| | IRTransformLayer we just pass the query on to the layer |
| | below. In our REPL this is our default way to search for |
| | function definitions. |
+------------------+-----------------------------------------------------------+
| | Searches for the named symbol in the module set indicated |
| | by the given ModuleHandleT. This is just an optimized |
| | search, better for lookup-speed when you know exactly |
| | a symbol definition should be found. In IRTransformLayer |
| findSymbolIn | we just pass this query on to the layer below. In our |
| | REPL we use this method to search for functions |
| | representing top-level expressions, since we know exactly |
| | where we'll find them: in the top-level expression module |
| | we just added. |
+------------------+-----------------------------------------------------------+
| | Forces all of the actions required to make the code and |
| | data in a module set (represented by a ModuleHandleT) |
| | accessible. Behaves as if some symbol in the set had been |
| | searched for and JITSymbol::getSymbolAddress called. This |
| emitAndFinalize | is rarely needed, but can be useful when dealing with |
| | layers that usually behave lazily if the user wants to |
| | trigger early compilation (for example, to use idle CPU |
| | time to eagerly compile code in the background). |
+------------------+-----------------------------------------------------------+
This interface attempts to capture the natural operations of a JIT (with some
wrinkles like emitAndFinalize for performance), similar to the basic JIT API
operations we identified in Chapter 1. Conforming to the layer concept allows
classes to compose neatly by implementing their behaviors in terms of the these
same operations, carried out on the layer below. For example, an eager layer
(like IRTransformLayer) can implement addModule by running each module in the
set through its transform up-front and immediately passing the result to the
layer below. A lazy layer, by contrast, could implement addModule by
squirreling away the modules doing no other up-front work, but applying the
transform (and calling addModule on the layer below) when the client calls
findSymbol instead. The JIT'd program behavior will be the same either way, but
these choices will have different performance characteristics: Doing work
eagerly means the JIT takes longer up-front, but proceeds smoothly once this is
done. Deferring work allows the JIT to get up-and-running quickly, but will
force the JIT to pause and wait whenever some code or data is needed that hasn't
already been processed.
Our current REPL is eager: Each function definition is optimized and compiled as
soon as it's typed in. If we were to make the transform layer lazy (but not
change things otherwise) we could defer optimization until the first time we
reference a function in a top-level expression (see if you can figure out why,
then check out the answer below [1]_). In the next chapter, however we'll
introduce fully lazy compilation, in which function's aren't compiled until
they're first called at run-time. At this point the trade-offs get much more
interesting: the lazier we are, the quicker we can start executing the first
function, but the more often we'll have to pause to compile newly encountered
functions. If we only code-gen lazily, but optimize eagerly, we'll have a slow
startup (which everything is optimized) but relatively short pauses as each
function just passes through code-gen. If we both optimize and code-gen lazily
we can start executing the first function more quickly, but we'll have longer
pauses as each function has to be both optimized and code-gen'd when it's first
executed. Things become even more interesting if we consider interproceedural
optimizations like inlining, which must be performed eagerly. These are
complex trade-offs, and there is no one-size-fits all solution to them, but by
providing composable layers we leave the decisions to the person implementing
the JIT, and make it easy for them to experiment with different configurations.
`Next: Adding Per-function Lazy Compilation <BuildingAJIT3.html>`_
Full Code Listing
=================
Here is the complete code listing for our running example with an
IRTransformLayer added to enable optimization. To build this example, use:
.. code-block:: bash
# Compile
clang++ -g toy.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core orcjit native` -O3 -o toy
# Run
./toy
Here is the code:
.. literalinclude:: ../../examples/Kaleidoscope/BuildingAJIT/Chapter2/KaleidoscopeJIT.h
:language: c++
.. [1] When we add our top-level expression to the JIT, any calls to functions
that we defined earlier will appear to the RTDyldObjectLinkingLayer as
external symbols. The RTDyldObjectLinkingLayer will call the SymbolResolver
that we defined in addModule, which in turn calls findSymbol on the
OptimizeLayer, at which point even a lazy transform layer will have to
do its work.

186
external/llvm/docs/tutorial/BuildingAJIT3.rst vendored
View File

@ -1,186 +0,0 @@
=============================================
Building a JIT: Per-function Lazy Compilation
=============================================
.. contents::
:local:
**This tutorial is under active development. It is incomplete and details may
change frequently.** Nonetheless we invite you to try it out as it stands, and
we welcome any feedback.
Chapter 3 Introduction
======================
Welcome to Chapter 3 of the "Building an ORC-based JIT in LLVM" tutorial. This
chapter discusses lazy JITing and shows you how to enable it by adding an ORC
CompileOnDemand layer the JIT from `Chapter 2 <BuildingAJIT2.html>`_.
Lazy Compilation
================
When we add a module to the KaleidoscopeJIT class from Chapter 2 it is
immediately optimized, compiled and linked for us by the IRTransformLayer,
IRCompileLayer and RTDyldObjectLinkingLayer respectively. This scheme, where all the
work to make a Module executable is done up front, is simple to understand and
its performance characteristics are easy to reason about. However, it will lead
to very high startup times if the amount of code to be compiled is large, and
may also do a lot of unnecessary compilation if only a few compiled functions
are ever called at runtime. A truly "just-in-time" compiler should allow us to
defer the compilation of any given function until the moment that function is
first called, improving launch times and eliminating redundant work. In fact,
the ORC APIs provide us with a layer to lazily compile LLVM IR:
*CompileOnDemandLayer*.
The CompileOnDemandLayer class conforms to the layer interface described in
Chapter 2, but its addModule method behaves quite differently from the layers
we have seen so far: rather than doing any work up front, it just scans the
Modules being added and arranges for each function in them to be compiled the
first time it is called. To do this, the CompileOnDemandLayer creates two small
utilities for each function that it scans: a *stub* and a *compile
callback*. The stub is a pair of a function pointer (which will be pointed at
the function's implementation once the function has been compiled) and an
indirect jump through the pointer. By fixing the address of the indirect jump
for the lifetime of the program we can give the function a permanent "effective
address", one that can be safely used for indirection and function pointer
comparison even if the function's implementation is never compiled, or if it is
compiled more than once (due to, for example, recompiling the function at a
higher optimization level) and changes address. The second utility, the compile
callback, represents a re-entry point from the program into the compiler that
will trigger compilation and then execution of a function. By initializing the
function's stub to point at the function's compile callback, we enable lazy
compilation: The first attempted call to the function will follow the function
pointer and trigger the compile callback instead. The compile callback will
compile the function, update the function pointer for the stub, then execute
the function. On all subsequent calls to the function, the function pointer
will point at the already-compiled function, so there is no further overhead
from the compiler. We will look at this process in more detail in the next
chapter of this tutorial, but for now we'll trust the CompileOnDemandLayer to
set all the stubs and callbacks up for us. All we need to do is to add the
CompileOnDemandLayer to the top of our stack and we'll get the benefits of
lazy compilation. We just need a few changes to the source:
.. code-block:: c++
...
#include "llvm/ExecutionEngine/SectionMemoryManager.h"
#include "llvm/ExecutionEngine/Orc/CompileOnDemandLayer.h"
#include "llvm/ExecutionEngine/Orc/CompileUtils.h"
...
...
class KaleidoscopeJIT {
private:
std::unique_ptr<TargetMachine> TM;
const DataLayout DL;
RTDyldObjectLinkingLayer ObjectLayer;
IRCompileLayer<decltype(ObjectLayer), SimpleCompiler> CompileLayer;
using OptimizeFunction =
std::function<std::shared_ptr<Module>(std::shared_ptr<Module>)>;
IRTransformLayer<decltype(CompileLayer), OptimizeFunction> OptimizeLayer;
std::unique_ptr<JITCompileCallbackManager> CompileCallbackManager;
CompileOnDemandLayer<decltype(OptimizeLayer)> CODLayer;
public:
using ModuleHandle = decltype(CODLayer)::ModuleHandleT;
First we need to include the CompileOnDemandLayer.h header, then add two new
members: a std::unique_ptr<JITCompileCallbackManager> and a CompileOnDemandLayer,
to our class. The CompileCallbackManager member is used by the CompileOnDemandLayer
to create the compile callback needed for each function.
.. code-block:: c++
KaleidoscopeJIT()
: TM(EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
ObjectLayer([]() { return std::make_shared<SectionMemoryManager>(); }),
CompileLayer(ObjectLayer, SimpleCompiler(*TM)),
OptimizeLayer(CompileLayer,
[this](std::shared_ptr<Module> M) {
return optimizeModule(std::move(M));
}),
CompileCallbackManager(
orc::createLocalCompileCallbackManager(TM->getTargetTriple(), 0)),
CODLayer(OptimizeLayer,
[this](Function &F) { return std::set<Function*>({&F}); },
*CompileCallbackManager,
orc::createLocalIndirectStubsManagerBuilder(
TM->getTargetTriple())) {
llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
}
Next we have to update our constructor to initialize the new members. To create
an appropriate compile callback manager we use the
createLocalCompileCallbackManager function, which takes a TargetMachine and a
JITTargetAddress to call if it receives a request to compile an unknown
function. In our simple JIT this situation is unlikely to come up, so we'll
cheat and just pass '0' here. In a production quality JIT you could give the
address of a function that throws an exception in order to unwind the JIT'd
code's stack.
Now we can construct our CompileOnDemandLayer. Following the pattern from
previous layers we start by passing a reference to the next layer down in our
stack -- the OptimizeLayer. Next we need to supply a 'partitioning function':
when a not-yet-compiled function is called, the CompileOnDemandLayer will call
this function to ask us what we would like to compile. At a minimum we need to
compile the function being called (given by the argument to the partitioning
function), but we could also request that the CompileOnDemandLayer compile other
functions that are unconditionally called (or highly likely to be called) from
the function being called. For KaleidoscopeJIT we'll keep it simple and just
request compilation of the function that was called. Next we pass a reference to
our CompileCallbackManager. Finally, we need to supply an "indirect stubs
manager builder": a utility function that constructs IndirectStubManagers, which
are in turn used to build the stubs for the functions in each module. The
CompileOnDemandLayer will call the indirect stub manager builder once for each
call to addModule, and use the resulting indirect stubs manager to create
stubs for all functions in all modules in the set. If/when the module set is
removed from the JIT the indirect stubs manager will be deleted, freeing any
memory allocated to the stubs. We supply this function by using the
createLocalIndirectStubsManagerBuilder utility.
.. code-block:: c++
// ...
if (auto Sym = CODLayer.findSymbol(Name, false))
// ...
return cantFail(CODLayer.addModule(std::move(Ms),
std::move(Resolver)));
// ...
// ...
return CODLayer.findSymbol(MangledNameStream.str(), true);
// ...
// ...
CODLayer.removeModule(H);
// ...
Finally, we need to replace the references to OptimizeLayer in our addModule,
findSymbol, and removeModule methods. With that, we're up and running.
**To be done:**
** Chapter conclusion.**
Full Code Listing
=================
Here is the complete code listing for our running example with a CompileOnDemand
layer added to enable lazy function-at-a-time compilation. To build this example, use:
.. code-block:: bash
# Compile
clang++ -g toy.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core orcjit native` -O3 -o toy
# Run
./toy
Here is the code:
.. literalinclude:: ../../examples/Kaleidoscope/BuildingAJIT/Chapter3/KaleidoscopeJIT.h
:language: c++
`Next: Extreme Laziness -- Using Compile Callbacks to JIT directly from ASTs <BuildingAJIT4.html>`_

48
external/llvm/docs/tutorial/BuildingAJIT4.rst vendored
View File

@ -1,48 +0,0 @@
===========================================================================
Building a JIT: Extreme Laziness - Using Compile Callbacks to JIT from ASTs
===========================================================================
.. contents::
:local:
**This tutorial is under active development. It is incomplete and details may
change frequently.** Nonetheless we invite you to try it out as it stands, and
we welcome any feedback.
Chapter 4 Introduction
======================
Welcome to Chapter 4 of the "Building an ORC-based JIT in LLVM" tutorial. This
chapter introduces the Compile Callbacks and Indirect Stubs APIs and shows how
they can be used to replace the CompileOnDemand layer from
`Chapter 3 <BuildingAJIT3.html>`_ with a custom lazy-JITing scheme that JITs
directly from Kaleidoscope ASTs.
**To be done:**
**(1) Describe the drawbacks of JITing from IR (have to compile to IR first,
which reduces the benefits of laziness).**
**(2) Describe CompileCallbackManagers and IndirectStubManagers in detail.**
**(3) Run through the implementation of addFunctionAST.**
Full Code Listing
=================
Here is the complete code listing for our running example that JITs lazily from
Kaleidoscope ASTS. To build this example, use:
.. code-block:: bash
# Compile
clang++ -g toy.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core orcjit native` -O3 -o toy
# Run
./toy
Here is the code:
.. literalinclude:: ../../examples/Kaleidoscope/BuildingAJIT/Chapter4/KaleidoscopeJIT.h
:language: c++
`Next: Remote-JITing -- Process-isolation and laziness-at-a-distance <BuildingAJIT5.html>`_

57
external/llvm/docs/tutorial/BuildingAJIT5.rst vendored
View File

@ -1,57 +0,0 @@
=============================================================================
Building a JIT: Remote-JITing -- Process Isolation and Laziness at a Distance
=============================================================================
.. contents::
:local:
**This tutorial is under active development. It is incomplete and details may
change frequently.** Nonetheless we invite you to try it out as it stands, and
we welcome any feedback.
Chapter 5 Introduction
======================
Welcome to Chapter 5 of the "Building an ORC-based JIT in LLVM" tutorial. This
chapter introduces the ORC RemoteJIT Client/Server APIs and shows how to use
them to build a JIT stack that will execute its code via a communications
channel with a different process. This can be a separate process on the same
machine, a process on a different machine, or even a process on a different
platform/architecture. The code builds on top of the lazy-AST-compiling JIT
stack from `Chapter 4 <BuildingAJIT3.html>`_.
**To be done -- this is going to be a long one:**
**(1) Introduce channels, RPC, RemoteJIT Client and Server APIs**
**(2) Describe the client code in greater detail. Discuss modifications of the
KaleidoscopeJIT class, and the REPL itself.**
**(3) Describe the server code.**
**(4) Describe how to run the demo.**
Full Code Listing
=================
Here is the complete code listing for our running example that JITs lazily from
Kaleidoscope ASTS. To build this example, use:
.. code-block:: bash
# Compile
clang++ -g toy.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core orcjit native` -O3 -o toy
clang++ -g Server/server.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core orcjit native` -O3 -o toy-server
# Run
./toy-server &
./toy
Here is the code for the modified KaleidoscopeJIT:
.. literalinclude:: ../../examples/Kaleidoscope/BuildingAJIT/Chapter5/KaleidoscopeJIT.h
:language: c++
And the code for the JIT server:
.. literalinclude:: ../../examples/Kaleidoscope/BuildingAJIT/Chapter5/Server/server.cpp
:language: c++

293
external/llvm/docs/tutorial/LangImpl01.rst vendored
View File

@ -1,293 +0,0 @@
=================================================
Kaleidoscope: Tutorial Introduction and the Lexer
=================================================
.. contents::
:local:
Tutorial Introduction
=====================
Welcome to the "Implementing a language with LLVM" tutorial. This
tutorial runs through the implementation of a simple language, showing
how fun and easy it can be. This tutorial will get you up and started as
well as help to build a framework you can extend to other languages. The
code in this tutorial can also be used as a playground to hack on other
LLVM specific things.
The goal of this tutorial is to progressively unveil our language,
describing how it is built up over time. This will let us cover a fairly
broad range of language design and LLVM-specific usage issues, showing
and explaining the code for it all along the way, without overwhelming
you with tons of details up front.
It is useful to point out ahead of time that this tutorial is really
about teaching compiler techniques and LLVM specifically, *not* about
teaching modern and sane software engineering principles. In practice,
this means that we'll take a number of shortcuts to simplify the
exposition. For example, the code uses global variables
all over the place, doesn't use nice design patterns like
`visitors <http://en.wikipedia.org/wiki/Visitor_pattern>`_, etc... but
it is very simple. If you dig in and use the code as a basis for future
projects, fixing these deficiencies shouldn't be hard.
I've tried to put this tutorial together in a way that makes chapters
easy to skip over if you are already familiar with or are uninterested
in the various pieces. The structure of the tutorial is:
- `Chapter #1 <#language>`_: Introduction to the Kaleidoscope
language, and the definition of its Lexer - This shows where we are
going and the basic functionality that we want it to do. In order to
make this tutorial maximally understandable and hackable, we choose
to implement everything in C++ instead of using lexer and parser
generators. LLVM obviously works just fine with such tools, feel free
to use one if you prefer.
- `Chapter #2 <LangImpl02.html>`_: Implementing a Parser and AST -
With the lexer in place, we can talk about parsing techniques and
basic AST construction. This tutorial describes recursive descent
parsing and operator precedence parsing. Nothing in Chapters 1 or 2
is LLVM-specific, the code doesn't even link in LLVM at this point.
:)
- `Chapter #3 <LangImpl03.html>`_: Code generation to LLVM IR - With
the AST ready, we can show off how easy generation of LLVM IR really
is.
- `Chapter #4 <LangImpl04.html>`_: Adding JIT and Optimizer Support
- Because a lot of people are interested in using LLVM as a JIT,
we'll dive right into it and show you the 3 lines it takes to add JIT
support. LLVM is also useful in many other ways, but this is one
simple and "sexy" way to show off its power. :)
- `Chapter #5 <LangImpl05.html>`_: Extending the Language: Control
Flow - With the language up and running, we show how to extend it
with control flow operations (if/then/else and a 'for' loop). This
gives us a chance to talk about simple SSA construction and control
flow.
- `Chapter #6 <LangImpl06.html>`_: Extending the Language:
User-defined Operators - This is a silly but fun chapter that talks
about extending the language to let the user program define their own
arbitrary unary and binary operators (with assignable precedence!).
This lets us build a significant piece of the "language" as library
routines.
- `Chapter #7 <LangImpl07.html>`_: Extending the Language: Mutable
Variables - This chapter talks about adding user-defined local
variables along with an assignment operator. The interesting part
about this is how easy and trivial it is to construct SSA form in
LLVM: no, LLVM does *not* require your front-end to construct SSA
form!
- `Chapter #8 <LangImpl08.html>`_: Compiling to Object Files - This
chapter explains how to take LLVM IR and compile it down to object
files.
- `Chapter #9 <LangImpl09.html>`_: Extending the Language: Debug
Information - Having built a decent little programming language with
control flow, functions and mutable variables, we consider what it
takes to add debug information to standalone executables. This debug
information will allow you to set breakpoints in Kaleidoscope
functions, print out argument variables, and call functions - all
from within the debugger!
- `Chapter #10 <LangImpl10.html>`_: Conclusion and other useful LLVM
tidbits - This chapter wraps up the series by talking about
potential ways to extend the language, but also includes a bunch of
pointers to info about "special topics" like adding garbage
collection support, exceptions, debugging, support for "spaghetti
stacks", and a bunch of other tips and tricks.
By the end of the tutorial, we'll have written a bit less than 1000 lines
of non-comment, non-blank, lines of code. With this small amount of
code, we'll have built up a very reasonable compiler for a non-trivial
language including a hand-written lexer, parser, AST, as well as code
generation support with a JIT compiler. While other systems may have
interesting "hello world" tutorials, I think the breadth of this
tutorial is a great testament to the strengths of LLVM and why you
should consider it if you're interested in language or compiler design.
A note about this tutorial: we expect you to extend the language and
play with it on your own. Take the code and go crazy hacking away at it,
compilers don't need to be scary creatures - it can be a lot of fun to
play with languages!
The Basic Language
==================
This tutorial will be illustrated with a toy language that we'll call
"`Kaleidoscope <http://en.wikipedia.org/wiki/Kaleidoscope>`_" (derived
from "meaning beautiful, form, and view"). Kaleidoscope is a procedural
language that allows you to define functions, use conditionals, math,
etc. Over the course of the tutorial, we'll extend Kaleidoscope to
support the if/then/else construct, a for loop, user defined operators,
JIT compilation with a simple command line interface, etc.
Because we want to keep things simple, the only datatype in Kaleidoscope
is a 64-bit floating point type (aka 'double' in C parlance). As such,
all values are implicitly double precision and the language doesn't
require type declarations. This gives the language a very nice and
simple syntax. For example, the following simple example computes
`Fibonacci numbers: <http://en.wikipedia.org/wiki/Fibonacci_number>`_
::
# Compute the x'th fibonacci number.
def fib(x)
if x < 3 then
1
else
fib(x-1)+fib(x-2)
# This expression will compute the 40th number.
fib(40)
We also allow Kaleidoscope to call into standard library functions (the
LLVM JIT makes this completely trivial). This means that you can use the
'extern' keyword to define a function before you use it (this is also
useful for mutually recursive functions). For example:
::
extern sin(arg);
extern cos(arg);
extern atan2(arg1 arg2);
atan2(sin(.4), cos(42))
A more interesting example is included in Chapter 6 where we write a
little Kaleidoscope application that `displays a Mandelbrot
Set <LangImpl06.html#kicking-the-tires>`_ at various levels of magnification.
Lets dive into the implementation of this language!
The Lexer
=========
When it comes to implementing a language, the first thing needed is the
ability to process a text file and recognize what it says. The
traditional way to do this is to use a
"`lexer <http://en.wikipedia.org/wiki/Lexical_analysis>`_" (aka
'scanner') to break the input up into "tokens". Each token returned by
the lexer includes a token code and potentially some metadata (e.g. the
numeric value of a number). First, we define the possibilities:
.. code-block:: c++
// The lexer returns tokens [0-255] if it is an unknown character, otherwise one
// of these for known things.
enum Token {
tok_eof = -1,
// commands
tok_def = -2,
tok_extern = -3,
// primary
tok_identifier = -4,
tok_number = -5,
};
static std::string IdentifierStr; // Filled in if tok_identifier
static double NumVal; // Filled in if tok_number
Each token returned by our lexer will either be one of the Token enum
values or it will be an 'unknown' character like '+', which is returned
as its ASCII value. If the current token is an identifier, the
``IdentifierStr`` global variable holds the name of the identifier. If
the current token is a numeric literal (like 1.0), ``NumVal`` holds its
value. Note that we use global variables for simplicity, this is not the
best choice for a real language implementation :).
The actual implementation of the lexer is a single function named
``gettok``. The ``gettok`` function is called to return the next token
from standard input. Its definition starts as:
.. code-block:: c++
/// gettok - Return the next token from standard input.
static int gettok() {
static int LastChar = ' ';
// Skip any whitespace.
while (isspace(LastChar))
LastChar = getchar();
``gettok`` works by calling the C ``getchar()`` function to read
characters one at a time from standard input. It eats them as it
recognizes them and stores the last character read, but not processed,
in LastChar. The first thing that it has to do is ignore whitespace
between tokens. This is accomplished with the loop above.
The next thing ``gettok`` needs to do is recognize identifiers and
specific keywords like "def". Kaleidoscope does this with this simple
loop:
.. code-block:: c++
if (isalpha(LastChar)) { // identifier: [a-zA-Z][a-zA-Z0-9]*
IdentifierStr = LastChar;
while (isalnum((LastChar = getchar())))
IdentifierStr += LastChar;
if (IdentifierStr == "def")
return tok_def;
if (IdentifierStr == "extern")
return tok_extern;
return tok_identifier;
}
Note that this code sets the '``IdentifierStr``' global whenever it
lexes an identifier. Also, since language keywords are matched by the
same loop, we handle them here inline. Numeric values are similar:
.. code-block:: c++
if (isdigit(LastChar) || LastChar == '.') { // Number: [0-9.]+
std::string NumStr;
do {
NumStr += LastChar;
LastChar = getchar();
} while (isdigit(LastChar) || LastChar == '.');
NumVal = strtod(NumStr.c_str(), 0);
return tok_number;
}
This is all pretty straight-forward code for processing input. When
reading a numeric value from input, we use the C ``strtod`` function to
convert it to a numeric value that we store in ``NumVal``. Note that
this isn't doing sufficient error checking: it will incorrectly read
"1.23.45.67" and handle it as if you typed in "1.23". Feel free to
extend it :). Next we handle comments:
.. code-block:: c++
if (LastChar == '#') {
// Comment until end of line.
do
LastChar = getchar();
while (LastChar != EOF && LastChar != '\n' && LastChar != '\r');
if (LastChar != EOF)
return gettok();
}
We handle comments by skipping to the end of the line and then return
the next token. Finally, if the input doesn't match one of the above
cases, it is either an operator character like '+' or the end of the
file. These are handled with this code:
.. code-block:: c++
// Check for end of file. Don't eat the EOF.
if (LastChar == EOF)
return tok_eof;
// Otherwise, just return the character as its ascii value.
int ThisChar = LastChar;
LastChar = getchar();
return ThisChar;
}
With this, we have the complete lexer for the basic Kaleidoscope
language (the `full code listing <LangImpl02.html#full-code-listing>`_ for the Lexer
is available in the `next chapter <LangImpl02.html>`_ of the tutorial).
Next we'll `build a simple parser that uses this to build an Abstract
Syntax Tree <LangImpl02.html>`_. When we have that, we'll include a
driver so that you can use the lexer and parser together.
`Next: Implementing a Parser and AST <LangImpl02.html>`_

737
external/llvm/docs/tutorial/LangImpl02.rst vendored
View File

File diff suppressed because it is too large Load Diff

568
external/llvm/docs/tutorial/LangImpl03.rst vendored
View File

File diff suppressed because it is too large Load Diff

651
external/llvm/docs/tutorial/LangImpl04.rst vendored
View File

File diff suppressed because it is too large Load Diff

BIN
external/llvm/docs/tutorial/LangImpl05-cfg.png vendored
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 38 KiB

814
external/llvm/docs/tutorial/LangImpl05.rst vendored
View File

File diff suppressed because it is too large Load Diff

768
external/llvm/docs/tutorial/LangImpl06.rst vendored
View File

File diff suppressed because it is too large Load Diff

883
external/llvm/docs/tutorial/LangImpl07.rst vendored
View File

File diff suppressed because it is too large Load Diff

218
external/llvm/docs/tutorial/LangImpl08.rst vendored
View File

@ -1,218 +0,0 @@
========================================
Kaleidoscope: Compiling to Object Code
========================================
.. contents::
:local:
Chapter 8 Introduction
======================
Welcome to Chapter 8 of the "`Implementing a language with LLVM
<index.html>`_" tutorial. This chapter describes how to compile our
language down to object files.
Choosing a target
=================
LLVM has native support for cross-compilation. You can compile to the
architecture of your current machine, or just as easily compile for
other architectures. In this tutorial, we'll target the current
machine.
To specify the architecture that you want to target, we use a string
called a "target triple". This takes the form
``<arch><sub>-<vendor>-<sys>-<abi>`` (see the `cross compilation docs
<http://clang.llvm.org/docs/CrossCompilation.html#target-triple>`_).
As an example, we can see what clang thinks is our current target
triple:
::
$ clang --version | grep Target
Target: x86_64-unknown-linux-gnu
Running this command may show something different on your machine as
you might be using a different architecture or operating system to me.
Fortunately, we don't need to hard-code a target triple to target the
current machine. LLVM provides ``sys::getDefaultTargetTriple``, which
returns the target triple of the current machine.
.. code-block:: c++
auto TargetTriple = sys::getDefaultTargetTriple();
LLVM doesn't require us to to link in all the target
functionality. For example, if we're just using the JIT, we don't need
the assembly printers. Similarly, if we're only targeting certain
architectures, we can only link in the functionality for those
architectures.
For this example, we'll initialize all the targets for emitting object
code.
.. code-block:: c++
InitializeAllTargetInfos();
InitializeAllTargets();
InitializeAllTargetMCs();
InitializeAllAsmParsers();
InitializeAllAsmPrinters();
We can now use our target triple to get a ``Target``:
.. code-block:: c++
std::string Error;
auto Target = TargetRegistry::lookupTarget(TargetTriple, Error);
// Print an error and exit if we couldn't find the requested target.
// This generally occurs if we've forgotten to initialise the
// TargetRegistry or we have a bogus target triple.
if (!Target) {
errs() << Error;
return 1;
}
Target Machine
==============
We will also need a ``TargetMachine``. This class provides a complete
machine description of the machine we're targeting. If we want to
target a specific feature (such as SSE) or a specific CPU (such as
Intel's Sandylake), we do so now.
To see which features and CPUs that LLVM knows about, we can use
``llc``. For example, let's look at x86:
::
$ llvm-as < /dev/null | llc -march=x86 -mattr=help
Available CPUs for this target:
amdfam10 - Select the amdfam10 processor.
athlon - Select the athlon processor.
athlon-4 - Select the athlon-4 processor.
...
Available features for this target:
16bit-mode - 16-bit mode (i8086).
32bit-mode - 32-bit mode (80386).
3dnow - Enable 3DNow! instructions.
3dnowa - Enable 3DNow! Athlon instructions.
...
For our example, we'll use the generic CPU without any additional
features, options or relocation model.
.. code-block:: c++
auto CPU = "generic";
auto Features = "";
TargetOptions opt;
auto RM = Optional<Reloc::Model>();
auto TargetMachine = Target->createTargetMachine(TargetTriple, CPU, Features, opt, RM);
Configuring the Module
======================
We're now ready to configure our module, to specify the target and
data layout. This isn't strictly necessary, but the `frontend
performance guide <../Frontend/PerformanceTips.html>`_ recommends
this. Optimizations benefit from knowing about the target and data
layout.
.. code-block:: c++
TheModule->setDataLayout(TargetMachine->createDataLayout());
TheModule->setTargetTriple(TargetTriple);
Emit Object Code
================
We're ready to emit object code! Let's define where we want to write
our file to:
.. code-block:: c++
auto Filename = "output.o";
std::error_code EC;
raw_fd_ostream dest(Filename, EC, sys::fs::F_None);
if (EC) {
errs() << "Could not open file: " << EC.message();
return 1;
}
Finally, we define a pass that emits object code, then we run that
pass:
.. code-block:: c++
legacy::PassManager pass;
auto FileType = TargetMachine::CGFT_ObjectFile;
if (TargetMachine->addPassesToEmitFile(pass, dest, FileType)) {
errs() << "TargetMachine can't emit a file of this type";
return 1;
}
pass.run(*TheModule);
dest.flush();
Putting It All Together
=======================
Does it work? Let's give it a try. We need to compile our code, but
note that the arguments to ``llvm-config`` are different to the previous chapters.
::
$ clang++ -g -O3 toy.cpp `llvm-config --cxxflags --ldflags --system-libs --libs all` -o toy
Let's run it, and define a simple ``average`` function. Press Ctrl-D
when you're done.
::
$ ./toy
ready> def average(x y) (x + y) * 0.5;
^D
Wrote output.o
We have an object file! To test it, let's write a simple program and
link it with our output. Here's the source code:
.. code-block:: c++
#include <iostream>
extern "C" {
double average(double, double);
}
int main() {
std::cout << "average of 3.0 and 4.0: " << average(3.0, 4.0) << std::endl;
}
We link our program to output.o and check the result is what we
expected:
::
$ clang++ main.cpp output.o -o main
$ ./main
average of 3.0 and 4.0: 3.5
Full Code Listing
=================
.. literalinclude:: ../../examples/Kaleidoscope/Chapter8/toy.cpp
:language: c++
`Next: Adding Debug Information <LangImpl09.html>`_

465
external/llvm/docs/tutorial/LangImpl09.rst vendored
View File

@ -1,465 +0,0 @@
======================================
Kaleidoscope: Adding Debug Information
======================================
.. contents::
:local:
Chapter 9 Introduction
======================
Welcome to Chapter 9 of the "`Implementing a language with
LLVM <index.html>`_" tutorial. In chapters 1 through 8, we've built a
decent little programming language with functions and variables.
What happens if something goes wrong though, how do you debug your
program?
Source level debugging uses formatted data that helps a debugger
translate from binary and the state of the machine back to the
source that the programmer wrote. In LLVM we generally use a format
called `DWARF <http://dwarfstd.org>`_. DWARF is a compact encoding
that represents types, source locations, and variable locations.
The short summary of this chapter is that we'll go through the
various things you have to add to a programming language to
support debug info, and how you translate that into DWARF.
Caveat: For now we can't debug via the JIT, so we'll need to compile
our program down to something small and standalone. As part of this
we'll make a few modifications to the running of the language and
how programs are compiled. This means that we'll have a source file
with a simple program written in Kaleidoscope rather than the
interactive JIT. It does involve a limitation that we can only
have one "top level" command at a time to reduce the number of
changes necessary.
Here's the sample program we'll be compiling:
.. code-block:: python
def fib(x)
if x < 3 then
1
else
fib(x-1)+fib(x-2);
fib(10)
Why is this a hard problem?
===========================
Debug information is a hard problem for a few different reasons - mostly
centered around optimized code. First, optimization makes keeping source
locations more difficult. In LLVM IR we keep the original source location
for each IR level instruction on the instruction. Optimization passes
should keep the source locations for newly created instructions, but merged
instructions only get to keep a single location - this can cause jumping
around when stepping through optimized programs. Secondly, optimization
can move variables in ways that are either optimized out, shared in memory
with other variables, or difficult to track. For the purposes of this
tutorial we're going to avoid optimization (as you'll see with one of the
next sets of patches).
Ahead-of-Time Compilation Mode
==============================
To highlight only the aspects of adding debug information to a source
language without needing to worry about the complexities of JIT debugging
we're going to make a few changes to Kaleidoscope to support compiling
the IR emitted by the front end into a simple standalone program that
you can execute, debug, and see results.
First we make our anonymous function that contains our top level
statement be our "main":
.. code-block:: udiff
- auto Proto = llvm::make_unique<PrototypeAST>("", std::vector<std::string>());
+ auto Proto = llvm::make_unique<PrototypeAST>("main", std::vector<std::string>());
just with the simple change of giving it a name.
Then we're going to remove the command line code wherever it exists:
.. code-block:: udiff
@@ -1129,7 +1129,6 @@ static void HandleTopLevelExpression() {
/// top ::= definition | external | expression | ';'
static void MainLoop() {
while (1) {
- fprintf(stderr, "ready> ");
switch (CurTok) {
case tok_eof:
return;
@@ -1184,7 +1183,6 @@ int main() {
BinopPrecedence['*'] = 40; // highest.
// Prime the first token.
- fprintf(stderr, "ready> ");
getNextToken();
Lastly we're going to disable all of the optimization passes and the JIT so
that the only thing that happens after we're done parsing and generating
code is that the LLVM IR goes to standard error:
.. code-block:: udiff
@@ -1108,17 +1108,8 @@ static void HandleExtern() {
static void HandleTopLevelExpression() {
// Evaluate a top-level expression into an anonymous function.
if (auto FnAST = ParseTopLevelExpr()) {
- if (auto *FnIR = FnAST->codegen()) {
- // We're just doing this to make sure it executes.
- TheExecutionEngine->finalizeObject();
- // JIT the function, returning a function pointer.
- void *FPtr = TheExecutionEngine->getPointerToFunction(FnIR);
-
- // Cast it to the right type (takes no arguments, returns a double) so we
- // can call it as a native function.
- double (*FP)() = (double (*)())(intptr_t)FPtr;
- // Ignore the return value for this.
- (void)FP;
+ if (!F->codegen()) {
+ fprintf(stderr, "Error generating code for top level expr");
}
} else {
// Skip token for error recovery.
@@ -1439,11 +1459,11 @@ int main() {
// target lays out data structures.
TheModule->setDataLayout(TheExecutionEngine->getDataLayout());
OurFPM.add(new DataLayoutPass());
+#if 0
OurFPM.add(createBasicAliasAnalysisPass());
// Promote allocas to registers.
OurFPM.add(createPromoteMemoryToRegisterPass());
@@ -1218,7 +1210,7 @@ int main() {
OurFPM.add(createGVNPass());
// Simplify the control flow graph (deleting unreachable blocks, etc).
OurFPM.add(createCFGSimplificationPass());
-
+ #endif
OurFPM.doInitialization();
// Set the global so the code gen can use this.
This relatively small set of changes get us to the point that we can compile
our piece of Kaleidoscope language down to an executable program via this
command line:
.. code-block:: bash
Kaleidoscope-Ch9 < fib.ks | & clang -x ir -
which gives an a.out/a.exe in the current working directory.
Compile Unit
============
The top level container for a section of code in DWARF is a compile unit.
This contains the type and function data for an individual translation unit
(read: one file of source code). So the first thing we need to do is
construct one for our fib.ks file.
DWARF Emission Setup
====================
Similar to the ``IRBuilder`` class we have a
`DIBuilder <http://llvm.org/doxygen/classllvm_1_1DIBuilder.html>`_ class
that helps in constructing debug metadata for an LLVM IR file. It
corresponds 1:1 similarly to ``IRBuilder`` and LLVM IR, but with nicer names.
Using it does require that you be more familiar with DWARF terminology than
you needed to be with ``IRBuilder`` and ``Instruction`` names, but if you
read through the general documentation on the
`Metadata Format <http://llvm.org/docs/SourceLevelDebugging.html>`_ it
should be a little more clear. We'll be using this class to construct all
of our IR level descriptions. Construction for it takes a module so we
need to construct it shortly after we construct our module. We've left it
as a global static variable to make it a bit easier to use.
Next we're going to create a small container to cache some of our frequent
data. The first will be our compile unit, but we'll also write a bit of
code for our one type since we won't have to worry about multiple typed
expressions:
.. code-block:: c++
static DIBuilder *DBuilder;
struct DebugInfo {
DICompileUnit *TheCU;
DIType *DblTy;
DIType *getDoubleTy();
} KSDbgInfo;
DIType *DebugInfo::getDoubleTy() {
if (DblTy)
return DblTy;
DblTy = DBuilder->createBasicType("double", 64, dwarf::DW_ATE_float);
return DblTy;
}
And then later on in ``main`` when we're constructing our module:
.. code-block:: c++
DBuilder = new DIBuilder(*TheModule);
KSDbgInfo.TheCU = DBuilder->createCompileUnit(
dwarf::DW_LANG_C, DBuilder->createFile("fib.ks", "."),
"Kaleidoscope Compiler", 0, "", 0);
There are a couple of things to note here. First, while we're producing a
compile unit for a language called Kaleidoscope we used the language
constant for C. This is because a debugger wouldn't necessarily understand
the calling conventions or default ABI for a language it doesn't recognize
and we follow the C ABI in our LLVM code generation so it's the closest
thing to accurate. This ensures we can actually call functions from the
debugger and have them execute. Secondly, you'll see the "fib.ks" in the
call to ``createCompileUnit``. This is a default hard coded value since
we're using shell redirection to put our source into the Kaleidoscope
compiler. In a usual front end you'd have an input file name and it would
go there.
One last thing as part of emitting debug information via DIBuilder is that
we need to "finalize" the debug information. The reasons are part of the
underlying API for DIBuilder, but make sure you do this near the end of
main:
.. code-block:: c++
DBuilder->finalize();
before you dump out the module.
Functions
=========
Now that we have our ``Compile Unit`` and our source locations, we can add
function definitions to the debug info. So in ``PrototypeAST::codegen()`` we
add a few lines of code to describe a context for our subprogram, in this
case the "File", and the actual definition of the function itself.
So the context:
.. code-block:: c++
DIFile *Unit = DBuilder->createFile(KSDbgInfo.TheCU.getFilename(),
KSDbgInfo.TheCU.getDirectory());
giving us an DIFile and asking the ``Compile Unit`` we created above for the
directory and filename where we are currently. Then, for now, we use some
source locations of 0 (since our AST doesn't currently have source location
information) and construct our function definition:
.. code-block:: c++
DIScope *FContext = Unit;
unsigned LineNo = 0;
unsigned ScopeLine = 0;
DISubprogram *SP = DBuilder->createFunction(
FContext, P.getName(), StringRef(), Unit, LineNo,
CreateFunctionType(TheFunction->arg_size(), Unit),
false /* internal linkage */, true /* definition */, ScopeLine,
DINode::FlagPrototyped, false);
TheFunction->setSubprogram(SP);
and we now have an DISubprogram that contains a reference to all of our
metadata for the function.
Source Locations
================
The most important thing for debug information is accurate source location -
this makes it possible to map your source code back. We have a problem though,
Kaleidoscope really doesn't have any source location information in the lexer
or parser so we'll need to add it.
.. code-block:: c++
struct SourceLocation {
int Line;
int Col;
};
static SourceLocation CurLoc;
static SourceLocation LexLoc = {1, 0};
static int advance() {
int LastChar = getchar();
if (LastChar == '\n' || LastChar == '\r') {
LexLoc.Line++;
LexLoc.Col = 0;
} else
LexLoc.Col++;
return LastChar;
}
In this set of code we've added some functionality on how to keep track of the
line and column of the "source file". As we lex every token we set our current
current "lexical location" to the assorted line and column for the beginning
of the token. We do this by overriding all of the previous calls to
``getchar()`` with our new ``advance()`` that keeps track of the information
and then we have added to all of our AST classes a source location:
.. code-block:: c++
class ExprAST {
SourceLocation Loc;
public:
ExprAST(SourceLocation Loc = CurLoc) : Loc(Loc) {}
virtual ~ExprAST() {}
virtual Value* codegen() = 0;
int getLine() const { return Loc.Line; }
int getCol() const { return Loc.Col; }
virtual raw_ostream &dump(raw_ostream &out, int ind) {
return out << ':' << getLine() << ':' << getCol() << '\n';
}
that we pass down through when we create a new expression:
.. code-block:: c++
LHS = llvm::make_unique<BinaryExprAST>(BinLoc, BinOp, std::move(LHS),
std::move(RHS));
giving us locations for each of our expressions and variables.
To make sure that every instruction gets proper source location information,
we have to tell ``Builder`` whenever we're at a new source location.
We use a small helper function for this:
.. code-block:: c++
void DebugInfo::emitLocation(ExprAST *AST) {
DIScope *Scope;
if (LexicalBlocks.empty())
Scope = TheCU;
else
Scope = LexicalBlocks.back();
Builder.SetCurrentDebugLocation(
DebugLoc::get(AST->getLine(), AST->getCol(), Scope));
}
This both tells the main ``IRBuilder`` where we are, but also what scope
we're in. The scope can either be on compile-unit level or be the nearest
enclosing lexical block like the current function.
To represent this we create a stack of scopes:
.. code-block:: c++
std::vector<DIScope *> LexicalBlocks;
and push the scope (function) to the top of the stack when we start
generating the code for each function:
.. code-block:: c++
KSDbgInfo.LexicalBlocks.push_back(SP);
Also, we may not forget to pop the scope back off of the scope stack at the
end of the code generation for the function:
.. code-block:: c++
// Pop off the lexical block for the function since we added it
// unconditionally.
KSDbgInfo.LexicalBlocks.pop_back();
Then we make sure to emit the location every time we start to generate code
for a new AST object:
.. code-block:: c++
KSDbgInfo.emitLocation(this);
Variables
=========
Now that we have functions, we need to be able to print out the variables
we have in scope. Let's get our function arguments set up so we can get
decent backtraces and see how our functions are being called. It isn't
a lot of code, and we generally handle it when we're creating the
argument allocas in ``FunctionAST::codegen``.
.. code-block:: c++
// Record the function arguments in the NamedValues map.
NamedValues.clear();
unsigned ArgIdx = 0;
for (auto &Arg : TheFunction->args()) {
// Create an alloca for this variable.
AllocaInst *Alloca = CreateEntryBlockAlloca(TheFunction, Arg.getName());
// Create a debug descriptor for the variable.
DILocalVariable *D = DBuilder->createParameterVariable(
SP, Arg.getName(), ++ArgIdx, Unit, LineNo, KSDbgInfo.getDoubleTy(),
true);
DBuilder->insertDeclare(Alloca, D, DBuilder->createExpression(),
DebugLoc::get(LineNo, 0, SP),
Builder.GetInsertBlock());
// Store the initial value into the alloca.
Builder.CreateStore(&Arg, Alloca);
// Add arguments to variable symbol table.
NamedValues[Arg.getName()] = Alloca;
}
Here we're first creating the variable, giving it the scope (``SP``),
the name, source location, type, and since it's an argument, the argument
index. Next, we create an ``lvm.dbg.declare`` call to indicate at the IR
level that we've got a variable in an alloca (and it gives a starting
location for the variable), and setting a source location for the
beginning of the scope on the declare.
One interesting thing to note at this point is that various debuggers have
assumptions based on how code and debug information was generated for them
in the past. In this case we need to do a little bit of a hack to avoid
generating line information for the function prologue so that the debugger
knows to skip over those instructions when setting a breakpoint. So in
``FunctionAST::CodeGen`` we add some more lines:
.. code-block:: c++
// Unset the location for the prologue emission (leading instructions with no
// location in a function are considered part of the prologue and the debugger
// will run past them when breaking on a function)
KSDbgInfo.emitLocation(nullptr);
and then emit a new location when we actually start generating code for the
body of the function:
.. code-block:: c++
KSDbgInfo.emitLocation(Body.get());
With this we have enough debug information to set breakpoints in functions,
print out argument variables, and call functions. Not too bad for just a
few simple lines of code!
Full Code Listing
=================
Here is the complete code listing for our running example, enhanced with
debug information. To build this example, use:
.. code-block:: bash
# Compile
clang++ -g toy.cpp `llvm-config --cxxflags --ldflags --system-libs --libs core mcjit native` -O3 -o toy
# Run
./toy
Here is the code:
.. literalinclude:: ../../examples/Kaleidoscope/Chapter9/toy.cpp
:language: c++
`Next: Conclusion and other useful LLVM tidbits <LangImpl10.html>`_

259
external/llvm/docs/tutorial/LangImpl10.rst vendored
View File

@ -1,259 +0,0 @@
======================================================
Kaleidoscope: Conclusion and other useful LLVM tidbits
======================================================
.. contents::
:local:
Tutorial Conclusion
===================
Welcome to the final chapter of the "`Implementing a language with
LLVM <index.html>`_" tutorial. In the course of this tutorial, we have
grown our little Kaleidoscope language from being a useless toy, to
being a semi-interesting (but probably still useless) toy. :)
It is interesting to see how far we've come, and how little code it has
taken. We built the entire lexer, parser, AST, code generator, an
interactive run-loop (with a JIT!), and emitted debug information in
standalone executables - all in under 1000 lines of (non-comment/non-blank)
code.
Our little language supports a couple of interesting features: it
supports user defined binary and unary operators, it uses JIT
compilation for immediate evaluation, and it supports a few control flow
constructs with SSA construction.
Part of the idea of this tutorial was to show you how easy and fun it
can be to define, build, and play with languages. Building a compiler
need not be a scary or mystical process! Now that you've seen some of
the basics, I strongly encourage you to take the code and hack on it.
For example, try adding:
- **global variables** - While global variables have questional value
in modern software engineering, they are often useful when putting
together quick little hacks like the Kaleidoscope compiler itself.
Fortunately, our current setup makes it very easy to add global
variables: just have value lookup check to see if an unresolved
variable is in the global variable symbol table before rejecting it.
To create a new global variable, make an instance of the LLVM
``GlobalVariable`` class.
- **typed variables** - Kaleidoscope currently only supports variables
of type double. This gives the language a very nice elegance, because
only supporting one type means that you never have to specify types.
Different languages have different ways of handling this. The easiest
way is to require the user to specify types for every variable
definition, and record the type of the variable in the symbol table
along with its Value\*.
- **arrays, structs, vectors, etc** - Once you add types, you can start
extending the type system in all sorts of interesting ways. Simple
arrays are very easy and are quite useful for many different
applications. Adding them is mostly an exercise in learning how the
LLVM `getelementptr <../LangRef.html#getelementptr-instruction>`_ instruction
works: it is so nifty/unconventional, it `has its own
FAQ <../GetElementPtr.html>`_!
- **standard runtime** - Our current language allows the user to access
arbitrary external functions, and we use it for things like "printd"
and "putchard". As you extend the language to add higher-level
constructs, often these constructs make the most sense if they are
lowered to calls into a language-supplied runtime. For example, if
you add hash tables to the language, it would probably make sense to
add the routines to a runtime, instead of inlining them all the way.
- **memory management** - Currently we can only access the stack in
Kaleidoscope. It would also be useful to be able to allocate heap
memory, either with calls to the standard libc malloc/free interface
or with a garbage collector. If you would like to use garbage
collection, note that LLVM fully supports `Accurate Garbage
Collection <../GarbageCollection.html>`_ including algorithms that
move objects and need to scan/update the stack.
- **exception handling support** - LLVM supports generation of `zero
cost exceptions <../ExceptionHandling.html>`_ which interoperate with
code compiled in other languages. You could also generate code by
implicitly making every function return an error value and checking
it. You could also make explicit use of setjmp/longjmp. There are
many different ways to go here.
- **object orientation, generics, database access, complex numbers,
geometric programming, ...** - Really, there is no end of crazy
features that you can add to the language.
- **unusual domains** - We've been talking about applying LLVM to a
domain that many people are interested in: building a compiler for a
specific language. However, there are many other domains that can use
compiler technology that are not typically considered. For example,
LLVM has been used to implement OpenGL graphics acceleration,
translate C++ code to ActionScript, and many other cute and clever
things. Maybe you will be the first to JIT compile a regular
expression interpreter into native code with LLVM?
Have fun - try doing something crazy and unusual. Building a language
like everyone else always has, is much less fun than trying something a
little crazy or off the wall and seeing how it turns out. If you get
stuck or want to talk about it, feel free to email the `llvm-dev mailing
list <http://lists.llvm.org/mailman/listinfo/llvm-dev>`_: it has lots
of people who are interested in languages and are often willing to help
out.
Before we end this tutorial, I want to talk about some "tips and tricks"
for generating LLVM IR. These are some of the more subtle things that
may not be obvious, but are very useful if you want to take advantage of
LLVM's capabilities.
Properties of the LLVM IR
=========================
We have a couple of common questions about code in the LLVM IR form -
let's just get these out of the way right now, shall we?
Target Independence
-------------------
Kaleidoscope is an example of a "portable language": any program written
in Kaleidoscope will work the same way on any target that it runs on.
Many other languages have this property, e.g. lisp, java, haskell,
javascript, python, etc (note that while these languages are portable,
not all their libraries are).
One nice aspect of LLVM is that it is often capable of preserving target
independence in the IR: you can take the LLVM IR for a
Kaleidoscope-compiled program and run it on any target that LLVM
supports, even emitting C code and compiling that on targets that LLVM
doesn't support natively. You can trivially tell that the Kaleidoscope
compiler generates target-independent code because it never queries for
any target-specific information when generating code.
The fact that LLVM provides a compact, target-independent,
representation for code gets a lot of people excited. Unfortunately,
these people are usually thinking about C or a language from the C
family when they are asking questions about language portability. I say
"unfortunately", because there is really no way to make (fully general)
C code portable, other than shipping the source code around (and of
course, C source code is not actually portable in general either - ever
port a really old application from 32- to 64-bits?).
The problem with C (again, in its full generality) is that it is heavily
laden with target specific assumptions. As one simple example, the
preprocessor often destructively removes target-independence from the
code when it processes the input text:
.. code-block:: c
#ifdef __i386__
int X = 1;
#else
int X = 42;
#endif
While it is possible to engineer more and more complex solutions to
problems like this, it cannot be solved in full generality in a way that
is better than shipping the actual source code.
That said, there are interesting subsets of C that can be made portable.
If you are willing to fix primitive types to a fixed size (say int =
32-bits, and long = 64-bits), don't care about ABI compatibility with
existing binaries, and are willing to give up some other minor features,
you can have portable code. This can make sense for specialized domains
such as an in-kernel language.
Safety Guarantees
-----------------
Many of the languages above are also "safe" languages: it is impossible
for a program written in Java to corrupt its address space and crash the
process (assuming the JVM has no bugs). Safety is an interesting
property that requires a combination of language design, runtime
support, and often operating system support.
It is certainly possible to implement a safe language in LLVM, but LLVM
IR does not itself guarantee safety. The LLVM IR allows unsafe pointer
casts, use after free bugs, buffer over-runs, and a variety of other
problems. Safety needs to be implemented as a layer on top of LLVM and,
conveniently, several groups have investigated this. Ask on the `llvm-dev
mailing list <http://lists.llvm.org/mailman/listinfo/llvm-dev>`_ if
you are interested in more details.
Language-Specific Optimizations
-------------------------------
One thing about LLVM that turns off many people is that it does not
solve all the world's problems in one system (sorry 'world hunger',
someone else will have to solve you some other day). One specific
complaint is that people perceive LLVM as being incapable of performing
high-level language-specific optimization: LLVM "loses too much
information".
Unfortunately, this is really not the place to give you a full and
unified version of "Chris Lattner's theory of compiler design". Instead,
I'll make a few observations:
First, you're right that LLVM does lose information. For example, as of
this writing, there is no way to distinguish in the LLVM IR whether an
SSA-value came from a C "int" or a C "long" on an ILP32 machine (other
than debug info). Both get compiled down to an 'i32' value and the
information about what it came from is lost. The more general issue
here, is that the LLVM type system uses "structural equivalence" instead
of "name equivalence". Another place this surprises people is if you
have two types in a high-level language that have the same structure
(e.g. two different structs that have a single int field): these types
will compile down into a single LLVM type and it will be impossible to
tell what it came from.
Second, while LLVM does lose information, LLVM is not a fixed target: we
continue to enhance and improve it in many different ways. In addition
to adding new features (LLVM did not always support exceptions or debug
info), we also extend the IR to capture important information for
optimization (e.g. whether an argument is sign or zero extended,
information about pointers aliasing, etc). Many of the enhancements are
user-driven: people want LLVM to include some specific feature, so they
go ahead and extend it.
Third, it is *possible and easy* to add language-specific optimizations,
and you have a number of choices in how to do it. As one trivial
example, it is easy to add language-specific optimization passes that
"know" things about code compiled for a language. In the case of the C
family, there is an optimization pass that "knows" about the standard C
library functions. If you call "exit(0)" in main(), it knows that it is
safe to optimize that into "return 0;" because C specifies what the
'exit' function does.
In addition to simple library knowledge, it is possible to embed a
variety of other language-specific information into the LLVM IR. If you
have a specific need and run into a wall, please bring the topic up on
the llvm-dev list. At the very worst, you can always treat LLVM as if it
were a "dumb code generator" and implement the high-level optimizations
you desire in your front-end, on the language-specific AST.
Tips and Tricks
===============
There is a variety of useful tips and tricks that you come to know after
working on/with LLVM that aren't obvious at first glance. Instead of
letting everyone rediscover them, this section talks about some of these
issues.
Implementing portable offsetof/sizeof
-------------------------------------
One interesting thing that comes up, if you are trying to keep the code
generated by your compiler "target independent", is that you often need
to know the size of some LLVM type or the offset of some field in an
llvm structure. For example, you might need to pass the size of a type
into a function that allocates memory.
Unfortunately, this can vary widely across targets: for example the
width of a pointer is trivially target-specific. However, there is a
`clever way to use the getelementptr
instruction <http://nondot.org/sabre/LLVMNotes/SizeOf-OffsetOf-VariableSizedStructs.txt>`_
that allows you to compute this in a portable way.
Garbage Collected Stack Frames
------------------------------
Some languages want to explicitly manage their stack frames, often so
that they are garbage collected or to allow easy implementation of
closures. There are often better ways to implement these features than
explicit stack frames, but `LLVM does support
them, <http://nondot.org/sabre/LLVMNotes/ExplicitlyManagedStackFrames.txt>`_
if you want. It requires your front-end to convert the code into
`Continuation Passing
Style <http://en.wikipedia.org/wiki/Continuation-passing_style>`_ and
the use of tail calls (which LLVM also supports).

285
external/llvm/docs/tutorial/OCamlLangImpl1.rst vendored
View File

@ -1,285 +0,0 @@
=================================================
Kaleidoscope: Tutorial Introduction and the Lexer
=================================================
.. contents::
:local:
Tutorial Introduction
=====================
Welcome to the "Implementing a language with LLVM" tutorial. This
tutorial runs through the implementation of a simple language, showing
how fun and easy it can be. This tutorial will get you up and started as
well as help to build a framework you can extend to other languages. The
code in this tutorial can also be used as a playground to hack on other
LLVM specific things.
The goal of this tutorial is to progressively unveil our language,
describing how it is built up over time. This will let us cover a fairly
broad range of language design and LLVM-specific usage issues, showing
and explaining the code for it all along the way, without overwhelming
you with tons of details up front.
It is useful to point out ahead of time that this tutorial is really
about teaching compiler techniques and LLVM specifically, *not* about
teaching modern and sane software engineering principles. In practice,
this means that we'll take a number of shortcuts to simplify the
exposition. For example, the code leaks memory, uses global variables
all over the place, doesn't use nice design patterns like
`visitors <http://en.wikipedia.org/wiki/Visitor_pattern>`_, etc... but
it is very simple. If you dig in and use the code as a basis for future
projects, fixing these deficiencies shouldn't be hard.
I've tried to put this tutorial together in a way that makes chapters
easy to skip over if you are already familiar with or are uninterested
in the various pieces. The structure of the tutorial is:
- `Chapter #1 <#language>`_: Introduction to the Kaleidoscope
language, and the definition of its Lexer - This shows where we are
going and the basic functionality that we want it to do. In order to
make this tutorial maximally understandable and hackable, we choose
to implement everything in Objective Caml instead of using lexer and
parser generators. LLVM obviously works just fine with such tools,
feel free to use one if you prefer.
- `Chapter #2 <OCamlLangImpl2.html>`_: Implementing a Parser and
AST - With the lexer in place, we can talk about parsing techniques
and basic AST construction. This tutorial describes recursive descent
parsing and operator precedence parsing. Nothing in Chapters 1 or 2
is LLVM-specific, the code doesn't even link in LLVM at this point.
:)
- `Chapter #3 <OCamlLangImpl3.html>`_: Code generation to LLVM IR -
With the AST ready, we can show off how easy generation of LLVM IR
really is.
- `Chapter #4 <OCamlLangImpl4.html>`_: Adding JIT and Optimizer
Support - Because a lot of people are interested in using LLVM as a
JIT, we'll dive right into it and show you the 3 lines it takes to
add JIT support. LLVM is also useful in many other ways, but this is
one simple and "sexy" way to shows off its power. :)
- `Chapter #5 <OCamlLangImpl5.html>`_: Extending the Language:
Control Flow - With the language up and running, we show how to
extend it with control flow operations (if/then/else and a 'for'
loop). This gives us a chance to talk about simple SSA construction
and control flow.
- `Chapter #6 <OCamlLangImpl6.html>`_: Extending the Language:
User-defined Operators - This is a silly but fun chapter that talks
about extending the language to let the user program define their own
arbitrary unary and binary operators (with assignable precedence!).
This lets us build a significant piece of the "language" as library
routines.
- `Chapter #7 <OCamlLangImpl7.html>`_: Extending the Language:
Mutable Variables - This chapter talks about adding user-defined
local variables along with an assignment operator. The interesting
part about this is how easy and trivial it is to construct SSA form
in LLVM: no, LLVM does *not* require your front-end to construct SSA
form!
- `Chapter #8 <OCamlLangImpl8.html>`_: Conclusion and other useful
LLVM tidbits - This chapter wraps up the series by talking about
potential ways to extend the language, but also includes a bunch of
pointers to info about "special topics" like adding garbage
collection support, exceptions, debugging, support for "spaghetti
stacks", and a bunch of other tips and tricks.
By the end of the tutorial, we'll have written a bit less than 700 lines
of non-comment, non-blank, lines of code. With this small amount of
code, we'll have built up a very reasonable compiler for a non-trivial
language including a hand-written lexer, parser, AST, as well as code
generation support with a JIT compiler. While other systems may have
interesting "hello world" tutorials, I think the breadth of this
tutorial is a great testament to the strengths of LLVM and why you
should consider it if you're interested in language or compiler design.
A note about this tutorial: we expect you to extend the language and
play with it on your own. Take the code and go crazy hacking away at it,
compilers don't need to be scary creatures - it can be a lot of fun to
play with languages!
The Basic Language
==================
This tutorial will be illustrated with a toy language that we'll call
"`Kaleidoscope <http://en.wikipedia.org/wiki/Kaleidoscope>`_" (derived
from "meaning beautiful, form, and view"). Kaleidoscope is a procedural
language that allows you to define functions, use conditionals, math,
etc. Over the course of the tutorial, we'll extend Kaleidoscope to
support the if/then/else construct, a for loop, user defined operators,
JIT compilation with a simple command line interface, etc.
Because we want to keep things simple, the only datatype in Kaleidoscope
is a 64-bit floating point type (aka 'float' in OCaml parlance). As
such, all values are implicitly double precision and the language
doesn't require type declarations. This gives the language a very nice
and simple syntax. For example, the following simple example computes
`Fibonacci numbers: <http://en.wikipedia.org/wiki/Fibonacci_number>`_
::
# Compute the x'th fibonacci number.
def fib(x)
if x < 3 then
1
else
fib(x-1)+fib(x-2)
# This expression will compute the 40th number.
fib(40)
We also allow Kaleidoscope to call into standard library functions (the
LLVM JIT makes this completely trivial). This means that you can use the
'extern' keyword to define a function before you use it (this is also
useful for mutually recursive functions). For example:
::
extern sin(arg);
extern cos(arg);
extern atan2(arg1 arg2);
atan2(sin(.4), cos(42))
A more interesting example is included in Chapter 6 where we write a
little Kaleidoscope application that `displays a Mandelbrot
Set <OCamlLangImpl6.html#kicking-the-tires>`_ at various levels of magnification.
Lets dive into the implementation of this language!
The Lexer
=========
When it comes to implementing a language, the first thing needed is the
ability to process a text file and recognize what it says. The
traditional way to do this is to use a
"`lexer <http://en.wikipedia.org/wiki/Lexical_analysis>`_" (aka
'scanner') to break the input up into "tokens". Each token returned by
the lexer includes a token code and potentially some metadata (e.g. the
numeric value of a number). First, we define the possibilities:
.. code-block:: ocaml
(* The lexer returns these 'Kwd' if it is an unknown character, otherwise one of
* these others for known things. *)
type token =
(* commands *)
| Def | Extern
(* primary *)
| Ident of string | Number of float
(* unknown *)
| Kwd of char
Each token returned by our lexer will be one of the token variant
values. An unknown character like '+' will be returned as
``Token.Kwd '+'``. If the curr token is an identifier, the value will be
``Token.Ident s``. If the current token is a numeric literal (like 1.0),
the value will be ``Token.Number 1.0``.
The actual implementation of the lexer is a collection of functions
driven by a function named ``Lexer.lex``. The ``Lexer.lex`` function is
called to return the next token from standard input. We will use
`Camlp4 <http://caml.inria.fr/pub/docs/manual-camlp4/index.html>`_ to
simplify the tokenization of the standard input. Its definition starts
as:
.. code-block:: ocaml
(*===----------------------------------------------------------------------===
* Lexer
*===----------------------------------------------------------------------===*)
let rec lex = parser
(* Skip any whitespace. *)
| [< ' (' ' | '\n' | '\r' | '\t'); stream >] -> lex stream
``Lexer.lex`` works by recursing over a ``char Stream.t`` to read
characters one at a time from the standard input. It eats them as it
recognizes them and stores them in in a ``Token.token`` variant. The
first thing that it has to do is ignore whitespace between tokens. This
is accomplished with the recursive call above.
The next thing ``Lexer.lex`` needs to do is recognize identifiers and
specific keywords like "def". Kaleidoscope does this with a pattern
match and a helper function.
.. code-block:: ocaml
(* identifier: [a-zA-Z][a-zA-Z0-9] *)
| [< ' ('A' .. 'Z' | 'a' .. 'z' as c); stream >] ->
let buffer = Buffer.create 1 in
Buffer.add_char buffer c;
lex_ident buffer stream
...
and lex_ident buffer = parser
| [< ' ('A' .. 'Z' | 'a' .. 'z' | '0' .. '9' as c); stream >] ->
Buffer.add_char buffer c;
lex_ident buffer stream
| [< stream=lex >] ->
match Buffer.contents buffer with
| "def" -> [< 'Token.Def; stream >]
| "extern" -> [< 'Token.Extern; stream >]
| id -> [< 'Token.Ident id; stream >]
Numeric values are similar:
.. code-block:: ocaml
(* number: [0-9.]+ *)
| [< ' ('0' .. '9' as c); stream >] ->
let buffer = Buffer.create 1 in
Buffer.add_char buffer c;
lex_number buffer stream
...
and lex_number buffer = parser
| [< ' ('0' .. '9' | '.' as c); stream >] ->
Buffer.add_char buffer c;
lex_number buffer stream
| [< stream=lex >] ->
[< 'Token.Number (float_of_string (Buffer.contents buffer)); stream >]
This is all pretty straight-forward code for processing input. When
reading a numeric value from input, we use the ocaml ``float_of_string``
function to convert it to a numeric value that we store in
``Token.Number``. Note that this isn't doing sufficient error checking:
it will raise ``Failure`` if the string "1.23.45.67". Feel free to
extend it :). Next we handle comments:
.. code-block:: ocaml
(* Comment until end of line. *)
| [< ' ('#'); stream >] ->
lex_comment stream
...
and lex_comment = parser
| [< ' ('\n'); stream=lex >] -> stream
| [< 'c; e=lex_comment >] -> e
| [< >] -> [< >]
We handle comments by skipping to the end of the line and then return
the next token. Finally, if the input doesn't match one of the above
cases, it is either an operator character like '+' or the end of the
file. These are handled with this code:
.. code-block:: ocaml
(* Otherwise, just return the character as its ascii value. *)
| [< 'c; stream >] ->
[< 'Token.Kwd c; lex stream >]
(* end of stream. *)
| [< >] -> [< >]
With this, we have the complete lexer for the basic Kaleidoscope
language (the `full code listing <OCamlLangImpl2.html#full-code-listing>`_ for the
Lexer is available in the `next chapter <OCamlLangImpl2.html>`_ of the
tutorial). Next we'll `build a simple parser that uses this to build an
Abstract Syntax Tree <OCamlLangImpl2.html>`_. When we have that, we'll
include a driver so that you can use the lexer and parser together.
`Next: Implementing a Parser and AST <OCamlLangImpl2.html>`_

899
external/llvm/docs/tutorial/OCamlLangImpl2.rst vendored
View File

File diff suppressed because it is too large Load Diff

961
external/llvm/docs/tutorial/OCamlLangImpl3.rst vendored
View File

File diff suppressed because it is too large Load Diff

915
external/llvm/docs/tutorial/OCamlLangImpl4.rst vendored
View File

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More