[ORC] Add generic initializer/deinitializer support.

Initializers and deinitializers are used to implement C++ static constructors
and destructors, runtime registration for some languages (e.g. with the
Objective-C runtime for Objective-C/C++ code) and other tasks that would
typically be performed when a shared-object/dylib is loaded or unloaded by a
statically compiled program.

MCJIT and ORC have historically provided limited support for discovering and
running initializers/deinitializers by scanning the llvm.global_ctors and
llvm.global_dtors variables and recording the functions to be run. This approach
suffers from several drawbacks: (1) It only works for IR inputs, not for object
files (including cached JIT'd objects). (2) It only works for initializers
described by llvm.global_ctors and llvm.global_dtors, however not all
initializers are described in this way (Objective-C, for example, describes
initializers via specially named metadata sections). (3) To make the
initializer/deinitializer functions described by llvm.global_ctors and
llvm.global_dtors searchable they must be promoted to extern linkage, polluting
the JIT symbol table (extra care must be taken to ensure this promotion does
not result in symbol name clashes).

This patch introduces several interdependent changes to ORCv2 to support the
construction of new initialization schemes, and includes an implementation of a
backwards-compatible llvm.global_ctor/llvm.global_dtor scanning scheme, and a
MachO specific scheme that handles Objective-C runtime registration (if the
Objective-C runtime is available) enabling execution of LLVM IR compiled from
Objective-C and Swift.

The major changes included in this patch are:

(1) The MaterializationUnit and MaterializationResponsibility classes are
extended to describe an optional "initializer" symbol for the module (see the
getInitializerSymbol method on each class). The presence or absence of this
symbol indicates whether the module contains any initializers or
deinitializers. The initializer symbol otherwise behaves like any other:
searching for it triggers materialization.

(2) A new Platform interface is introduced in llvm/ExecutionEngine/Orc/Core.h
which provides the following callback interface:

  - Error setupJITDylib(JITDylib &JD): Can be used to install standard symbols
    in JITDylibs upon creation. E.g. __dso_handle.

  - Error notifyAdding(JITDylib &JD, const MaterializationUnit &MU): Generally
    used to record initializer symbols.

  - Error notifyRemoving(JITDylib &JD, VModuleKey K): Used to notify a platform
    that a module is being removed.

  Platform implementations can use these callbacks to track outstanding
initializers and implement a platform-specific approach for executing them. For
example, the MachOPlatform installs a plugin in the JIT linker to scan for both
__mod_inits sections (for C++ static constructors) and ObjC metadata sections.
If discovered, these are processed in the usual platform order: Objective-C
registration is carried out first, then static initializers are executed,
ensuring that calls to Objective-C from static initializers will be safe.

This patch updates LLJIT to use the new scheme for initialization. Two
LLJIT::PlatformSupport classes are implemented: A GenericIR platform and a MachO
platform. The GenericIR platform implements a modified version of the previous
llvm.global-ctor scraping scheme to provide support for Windows and
Linux. LLJIT's MachO platform uses the MachOPlatform class to provide MachO
specific initialization as described above.

Reviewers: sgraenitz, dblaikie

Subscribers: mgorny, hiraditya, mgrang, ributzka, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D74300

llvm/examples/Kaleidoscope/BuildingAJIT/Chapter1/KaleidoscopeJIT.h

@@ -49,7 +49,7 @@ public:
                      std::make_unique<ConcurrentIRCompiler>(std::move(JTMB))),
         DL(std::move(DL)), Mangle(ES, this->DL),
         Ctx(std::make_unique<LLVMContext>()),
-        MainJD(ES.createJITDylib("<main>")) {
+        MainJD(ES.createBareJITDylib("<main>")) {
     MainJD.addGenerator(
         cantFail(DynamicLibrarySearchGenerator::GetForCurrentProcess(
             DL.getGlobalPrefix())));

llvm/examples/Kaleidoscope/BuildingAJIT/Chapter2/KaleidoscopeJIT.h

@@ -55,7 +55,7 @@ public:
                      std::make_unique<ConcurrentIRCompiler>(std::move(JTMB))),
         OptimizeLayer(ES, CompileLayer, optimizeModule), DL(std::move(DL)),
         Mangle(ES, this->DL), Ctx(std::make_unique<LLVMContext>()),
-        MainJD(ES.createJITDylib("<main>")) {
+        MainJD(ES.createBareJITDylib("<main>")) {
     MainJD.addGenerator(
         cantFail(DynamicLibrarySearchGenerator::GetForCurrentProcess(
             DL.getGlobalPrefix())));

llvm/include/llvm/ExecutionEngine/JITLink/JITLink.h

@@ -488,6 +488,8 @@ public:
 
   /// Set the visibility for this Symbol.
   void setScope(Scope S) {
+    assert((!Name.empty() || S == Scope::Local) &&
+           "Can not set anonymous symbol to non-local scope");
     assert((S == Scope::Default || Base->isDefined() || Base->isAbsolute()) &&
            "Invalid visibility for symbol type");
     this->S = static_cast<uint8_t>(S);

llvm/include/llvm/ExecutionEngine/Orc/CompileOnDemandLayer.h

@@ -94,6 +94,7 @@ public:
 
   /// Sets the ImplSymbolMap
   void setImplMap(ImplSymbolMap *Imp);
+
   /// Emits the given module. This should not be called by clients: it will be
   /// called by the JIT when a definition added via the add method is requested.
   void emit(MaterializationResponsibility R, ThreadSafeModule TSM) override;

llvm/include/llvm/ExecutionEngine/Orc/CompileUtils.h

@@ -30,7 +30,7 @@ namespace orc {
 
 class JITTargetMachineBuilder;
 
-IRMaterializationUnit::ManglingOptions
+IRSymbolMapper::ManglingOptions
 irManglingOptionsFromTargetOptions(const TargetOptions &Opts);
 
 /// Simple compile functor: Takes a single IR module and returns an ObjectFile.
@@ -52,7 +52,7 @@ public:
   Expected<CompileResult> operator()(Module &M) override;
 
 private:
-  IRMaterializationUnit::ManglingOptions
+  IRSymbolMapper::ManglingOptions
   manglingOptionsForTargetMachine(const TargetMachine &TM);
 
   CompileResult tryToLoadFromObjectCache(const Module &M);

llvm/include/llvm/ExecutionEngine/Orc/Core.h

@@ -14,11 +14,11 @@
 #define LLVM_EXECUTIONENGINE_ORC_CORE_H
 
 #include "llvm/ADT/BitmaskEnum.h"
+#include "llvm/ADT/DenseSet.h"
 #include "llvm/ADT/FunctionExtras.h"
 #include "llvm/ExecutionEngine/JITSymbol.h"
 #include "llvm/ExecutionEngine/Orc/SymbolStringPool.h"
 #include "llvm/ExecutionEngine/OrcV1Deprecation.h"
-#include "llvm/IR/Module.h"
 #include "llvm/Support/Debug.h"
 
 #include <memory>
@@ -456,6 +456,11 @@ public:
   /// before using.
   const SymbolFlagsMap &getSymbols() const { return SymbolFlags; }
 
+  /// Returns the initialization pseudo-symbol, if any. This symbol will also
+  /// be present in the SymbolFlagsMap for this MaterializationResponsibility
+  /// object.
+  const SymbolStringPtr &getInitializerSymbol() const { return InitSymbol; }
+
   /// Returns the names of any symbols covered by this
   /// MaterializationResponsibility object that have queries pending. This
   /// information can be used to return responsibility for unrequested symbols
@@ -532,10 +537,15 @@ private:
   /// Create a MaterializationResponsibility for the given JITDylib and
   ///        initial symbols.
   MaterializationResponsibility(JITDylib &JD, SymbolFlagsMap SymbolFlags,
-                                VModuleKey K);
+                                SymbolStringPtr InitSymbol, VModuleKey K)
+      : JD(JD), SymbolFlags(std::move(SymbolFlags)),
+        InitSymbol(std::move(InitSymbol)), K(std::move(K)) {
+    assert(!this->SymbolFlags.empty() && "Materializing nothing?");
+  }
 
   JITDylib &JD;
   SymbolFlagsMap SymbolFlags;
+  SymbolStringPtr InitSymbol;
   VModuleKey K;
 };
 
@@ -549,8 +559,13 @@ private:
 /// stronger definition is added or already present.
 class MaterializationUnit {
 public:
-  MaterializationUnit(SymbolFlagsMap InitalSymbolFlags, VModuleKey K)
-      : SymbolFlags(std::move(InitalSymbolFlags)), K(std::move(K)) {}
+  MaterializationUnit(SymbolFlagsMap InitalSymbolFlags,
+                      SymbolStringPtr InitSymbol, VModuleKey K)
+      : SymbolFlags(std::move(InitalSymbolFlags)),
+        InitSymbol(std::move(InitSymbol)), K(std::move(K)) {
+    assert((!this->InitSymbol || this->SymbolFlags.count(this->InitSymbol)) &&
+           "If set, InitSymbol should appear in InitialSymbolFlags map");
+  }
 
   virtual ~MaterializationUnit() {}
 
@@ -561,12 +576,15 @@ public:
   /// Return the set of symbols that this source provides.
   const SymbolFlagsMap &getSymbols() const { return SymbolFlags; }
 
+  /// Returns the initialization symbol for this MaterializationUnit (if any).
+  const SymbolStringPtr &getInitializerSymbol() const { return InitSymbol; }
+
   /// Called by materialization dispatchers (see
   /// ExecutionSession::DispatchMaterializationFunction) to trigger
   /// materialization of this MaterializationUnit.
   void doMaterialize(JITDylib &JD) {
-    materialize(MaterializationResponsibility(JD, std::move(SymbolFlags),
-                                              std::move(K)));
+    materialize(MaterializationResponsibility(
+        JD, std::move(SymbolFlags), std::move(InitSymbol), std::move(K)));
   }
 
   /// Called by JITDylibs to notify MaterializationUnits that the given symbol
@@ -578,6 +596,7 @@ public:
 
 protected:
   SymbolFlagsMap SymbolFlags;
+  SymbolStringPtr InitSymbol;
   VModuleKey K;
 
 private:
@@ -774,6 +793,7 @@ private:
 class JITDylib {
   friend class AsynchronousSymbolQuery;
   friend class ExecutionSession;
+  friend class Platform;
   friend class MaterializationResponsibility;
 public:
   /// Definition generators can be attached to JITDylibs to generate new
@@ -1054,6 +1074,35 @@ private:
   JITDylibSearchOrder SearchOrder;
 };
 
+/// Platforms set up standard symbols and mediate interactions between dynamic
+/// initializers (e.g. C++ static constructors) and ExecutionSession state.
+/// Note that Platforms do not automatically run initializers: clients are still
+/// responsible for doing this.
+class Platform {
+public:
+  virtual ~Platform();
+
+  /// This method will be called outside the session lock each time a JITDylib
+  /// is created (unless it is created with EmptyJITDylib set) to allow the
+  /// Platform to install any JITDylib specific standard symbols (e.g
+  /// __dso_handle).
+  virtual Error setupJITDylib(JITDylib &JD) = 0;
+
+  /// This method will be called under the ExecutionSession lock each time a
+  /// MaterializationUnit is added to a JITDylib.
+  virtual Error notifyAdding(JITDylib &JD, const MaterializationUnit &MU) = 0;
+
+  /// This method will be called under the ExecutionSession lock when a
+  /// VModuleKey is removed.
+  virtual Error notifyRemoving(JITDylib &JD, VModuleKey K) = 0;
+
+  /// A utility function for looking up initializer symbols. Performs a blocking
+  /// lookup for the given symbols in each of the given JITDylibs.
+  static Expected<DenseMap<JITDylib *, SymbolMap>>
+  lookupInitSymbols(ExecutionSession &ES,
+                    const DenseMap<JITDylib *, SymbolLookupSet> &InitSyms);
+};
+
 /// An ExecutionSession represents a running JIT program.
 class ExecutionSession {
   // FIXME: Remove this when we remove the old ORC layers.
@@ -1078,6 +1127,13 @@ public:
   /// Returns a shared_ptr to the SymbolStringPool for this ExecutionSession.
   std::shared_ptr<SymbolStringPool> getSymbolStringPool() const { return SSP; }
 
+  /// Set the Platform for this ExecutionSession.
+  void setPlatform(std::unique_ptr<Platform> P) { this->P = std::move(P); }
+
+  /// Get the Platform for this session.
+  /// Will return null if no Platform has been set for this ExecutionSession.
+  Platform *getPlatform() { return P.get(); }
+
   /// Run the given lambda with the session mutex locked.
   template <typename Func> decltype(auto) runSessionLocked(Func &&F) {
     std::lock_guard<std::recursive_mutex> Lock(SessionMutex);
@@ -1088,12 +1144,26 @@ public:
   /// Ownership of JITDylib remains within Execution Session
   JITDylib *getJITDylibByName(StringRef Name);
 
+  /// Add a new bare JITDylib to this ExecutionSession.
+  ///
+  /// The JITDylib Name is required to be unique. Clients should verify that
+  /// names are not being re-used (E.g. by calling getJITDylibByName) if names
+  /// are based on user input.
+  ///
+  /// This call does not install any library code or symbols into the newly
+  /// created JITDylib. The client is responsible for all configuration.
+  JITDylib &createBareJITDylib(std::string Name);
+
   /// Add a new JITDylib to this ExecutionSession.
   ///
   /// The JITDylib Name is required to be unique. Clients should verify that
   /// names are not being re-used (e.g. by calling getJITDylibByName) if names
   /// are based on user input.
-  JITDylib &createJITDylib(std::string Name);
+  ///
+  /// If a Platform is attached then Platform::setupJITDylib will be called to
+  /// install standard platform symbols (e.g. standard library interposes).
+  /// If no Platform is attached this call is equivalent to createBareJITDylib.
+  Expected<JITDylib &> createJITDylib(std::string Name);
 
   /// Allocate a module key for a new module to add to the JIT.
   VModuleKey allocateVModule() {
@@ -1177,20 +1247,23 @@ public:
   /// Convenience version of blocking lookup.
   /// Searches each of the JITDylibs in the search order in turn for the given
   /// symbol.
-  Expected<JITEvaluatedSymbol> lookup(const JITDylibSearchOrder &SearchOrder,
-                                      SymbolStringPtr Symbol);
+  Expected<JITEvaluatedSymbol>
+  lookup(const JITDylibSearchOrder &SearchOrder, SymbolStringPtr Symbol,
+         SymbolState RequiredState = SymbolState::Ready);
 
   /// Convenience version of blocking lookup.
   /// Searches each of the JITDylibs in the search order in turn for the given
   /// symbol. The search will not find non-exported symbols.
-  Expected<JITEvaluatedSymbol> lookup(ArrayRef<JITDylib *> SearchOrder,
-                                      SymbolStringPtr Symbol);
+  Expected<JITEvaluatedSymbol>
+  lookup(ArrayRef<JITDylib *> SearchOrder, SymbolStringPtr Symbol,
+         SymbolState RequiredState = SymbolState::Ready);
 
   /// Convenience version of blocking lookup.
   /// Searches each of the JITDylibs in the search order in turn for the given
   /// symbol. The search will not find non-exported symbols.
-  Expected<JITEvaluatedSymbol> lookup(ArrayRef<JITDylib *> SearchOrder,
-                                      StringRef Symbol);
+  Expected<JITEvaluatedSymbol>
+  lookup(ArrayRef<JITDylib *> SearchOrder, StringRef Symbol,
+         SymbolState RequiredState = SymbolState::Ready);
 
   /// Materialize the given unit.
   void dispatchMaterialization(JITDylib &JD,
@@ -1221,6 +1294,7 @@ private:
 
   mutable std::recursive_mutex SessionMutex;
   std::shared_ptr<SymbolStringPool> SSP;
+  std::unique_ptr<Platform> P;
   VModuleKey LastKey = 0;
   ErrorReporter ReportError = logErrorsToStdErr;
   DispatchMaterializationFunction DispatchMaterialization =
@@ -1256,6 +1330,11 @@ Error JITDylib::define(std::unique_ptr<MaterializationUnitType> &&MU) {
     if (auto Err = defineImpl(*MU))
       return Err;
 
+    if (auto *P = ES.getPlatform()) {
+      if (auto Err = P->notifyAdding(*this, *MU))
+        return Err;
+    }
+
     /// defineImpl succeeded.
     auto UMI = std::make_shared<UnmaterializedInfo>(std::move(MU));
     for (auto &KV : UMI->MU->getSymbols())
@@ -1273,6 +1352,11 @@ Error JITDylib::define(std::unique_ptr<MaterializationUnitType> &MU) {
     if (auto Err = defineImpl(*MU))
       return Err;
 
+    if (auto *P = ES.getPlatform()) {
+      if (auto Err = P->notifyAdding(*this, *MU))
+        return Err;
+    }
+
     /// defineImpl succeeded.
     auto UMI = std::make_shared<UnmaterializedInfo>(std::move(MU));
     for (auto &KV : UMI->MU->getSymbols())
@@ -1305,18 +1389,6 @@ private:
   SymbolPredicate Allow;
 };
 
-/// Mangles symbol names then uniques them in the context of an
-/// ExecutionSession.
-class MangleAndInterner {
-public:
-  MangleAndInterner(ExecutionSession &ES, const DataLayout &DL);
-  SymbolStringPtr operator()(StringRef Name);
-
-private:
-  ExecutionSession &ES;
-  const DataLayout &DL;
-};
-
 } // End namespace orc
 } // End namespace llvm
 

llvm/include/llvm/ExecutionEngine/Orc/ExecutionUtils.h

@@ -17,6 +17,7 @@
 #include "llvm/ADT/iterator_range.h"
 #include "llvm/ExecutionEngine/JITSymbol.h"
 #include "llvm/ExecutionEngine/Orc/Core.h"
+#include "llvm/ExecutionEngine/Orc/Mangling.h"
 #include "llvm/ExecutionEngine/Orc/OrcError.h"
 #include "llvm/ExecutionEngine/RuntimeDyld.h"
 #include "llvm/Object/Archive.h"
@@ -104,6 +105,53 @@ iterator_range<CtorDtorIterator> getConstructors(const Module &M);
 ///        array.
 iterator_range<CtorDtorIterator> getDestructors(const Module &M);
 
+/// This iterator provides a convenient way to iterate over GlobalValues that
+/// have initialization effects.
+class StaticInitGVIterator {
+public:
+  StaticInitGVIterator() = default;
+
+  StaticInitGVIterator(Module &M)
+      : I(M.global_values().begin()), E(M.global_values().end()),
+        ObjFmt(Triple(M.getTargetTriple()).getObjectFormat()) {
+    if (I != E) {
+      if (!isStaticInitGlobal(*I))
+        moveToNextStaticInitGlobal();
+    } else
+      I = E = Module::global_value_iterator();
+  }
+
+  bool operator==(const StaticInitGVIterator &O) const { return I == O.I; }
+  bool operator!=(const StaticInitGVIterator &O) const { return I != O.I; }
+
+  StaticInitGVIterator &operator++() {
+    assert(I != E && "Increment past end of range");
+    moveToNextStaticInitGlobal();
+    return *this;
+  }
+
+  GlobalValue &operator*() { return *I; }
+
+private:
+  bool isStaticInitGlobal(GlobalValue &GV);
+  void moveToNextStaticInitGlobal() {
+    ++I;
+    while (I != E && !isStaticInitGlobal(*I))
+      ++I;
+    if (I == E)
+      I = E = Module::global_value_iterator();
+  }
+
+  Module::global_value_iterator I, E;
+  Triple::ObjectFormatType ObjFmt;
+};
+
+/// Create an iterator range over the GlobalValues that contribute to static
+/// initialization.
+inline iterator_range<StaticInitGVIterator> getStaticInitGVs(Module &M) {
+  return make_range(StaticInitGVIterator(M), StaticInitGVIterator());
+}
+
 /// Convenience class for recording constructor/destructor names for
 ///        later execution.
 template <typename JITLayerT>
@@ -246,6 +294,22 @@ public:
   Error enable(JITDylib &JD, MangleAndInterner &Mangler);
 };
 
+/// An interface for Itanium __cxa_atexit interposer implementations.
+class ItaniumCXAAtExitSupport {
+public:
+  struct AtExitRecord {
+    void (*F)(void *);
+    void *Ctx;
+  };
+
+  void registerAtExit(void (*F)(void *), void *Ctx, void *DSOHandle);
+  void runAtExits(void *DSOHandle);
+
+private:
+  std::mutex AtExitsMutex;
+  DenseMap<void *, std::vector<AtExitRecord>> AtExitRecords;
+};
+
 /// A utility class to expose symbols found via dlsym to the JIT.
 ///
 /// If an instance of this class is attached to a JITDylib as a fallback

llvm/include/llvm/ExecutionEngine/Orc/IRCompileLayer.h

@@ -31,18 +31,18 @@ class IRCompileLayer : public IRLayer {
 public:
   class IRCompiler {
   public:
-    IRCompiler(IRMaterializationUnit::ManglingOptions MO) : MO(std::move(MO)) {}
+    IRCompiler(IRSymbolMapper::ManglingOptions MO) : MO(std::move(MO)) {}
     virtual ~IRCompiler();
-    const IRMaterializationUnit::ManglingOptions &getManglingOptions() const {
+    const IRSymbolMapper::ManglingOptions &getManglingOptions() const {
       return MO;
     }
     virtual Expected<std::unique_ptr<MemoryBuffer>> operator()(Module &M) = 0;
 
   protected:
-    IRMaterializationUnit::ManglingOptions &manglingOptions() { return MO; }
+    IRSymbolMapper::ManglingOptions &manglingOptions() { return MO; }
 
   private:
-    IRMaterializationUnit::ManglingOptions MO;
+    IRSymbolMapper::ManglingOptions MO;
   };
 
   using NotifyCompiledFunction =
@@ -61,7 +61,7 @@ private:
   mutable std::mutex IRLayerMutex;
   ObjectLayer &BaseLayer;
   std::unique_ptr<IRCompiler> Compile;
-  const IRMaterializationUnit::ManglingOptions *ManglingOpts;
+  const IRSymbolMapper::ManglingOptions *ManglingOpts;
   NotifyCompiledFunction NotifyCompiled = NotifyCompiledFunction();
 };
 

llvm/include/llvm/ExecutionEngine/Orc/IRTransformLayer.h

@@ -28,7 +28,7 @@ namespace orc {
 class IRTransformLayer : public IRLayer {
 public:
   using TransformFunction = std::function<Expected<ThreadSafeModule>(
-      ThreadSafeModule, const MaterializationResponsibility &R)>;
+      ThreadSafeModule, MaterializationResponsibility &R)>;
 
   IRTransformLayer(ExecutionSession &ES, IRLayer &BaseLayer,
                    TransformFunction Transform = identityTransform);
@@ -39,9 +39,8 @@ public:
 
   void emit(MaterializationResponsibility R, ThreadSafeModule TSM) override;
 
-  static ThreadSafeModule
-  identityTransform(ThreadSafeModule TSM,
-                    const MaterializationResponsibility &R) {
+  static ThreadSafeModule identityTransform(ThreadSafeModule TSM,
+                                            MaterializationResponsibility &R) {
     return TSM;
   }
 

llvm/include/llvm/ExecutionEngine/Orc/IndirectionUtils.h

@@ -201,7 +201,7 @@ protected:
                             ExecutionSession &ES,
                             JITTargetAddress ErrorHandlerAddress)
       : TP(std::move(TP)), ES(ES),
-        CallbacksJD(ES.createJITDylib("<Callbacks>")),
+        CallbacksJD(ES.createBareJITDylib("<Callbacks>")),
         ErrorHandlerAddress(ErrorHandlerAddress) {}
 
   void setTrampolinePool(std::unique_ptr<TrampolinePool> TP) {

llvm/include/llvm/ExecutionEngine/Orc/LLJIT.h

@@ -35,7 +35,23 @@ class LLLazyJITBuilderState;
 class LLJIT {
   template <typename, typename, typename> friend class LLJITBuilderSetters;
 
+  friend void setUpGenericLLVMIRPlatform(LLJIT &J);
+
 public:
+  /// Initializer support for LLJIT.
+  class PlatformSupport {
+  public:
+    virtual ~PlatformSupport();
+
+    virtual Error initialize(JITDylib &JD) = 0;
+
+    virtual Error deinitialize(JITDylib &JD) = 0;
+
+  protected:
+    static void setInitTransform(LLJIT &J,
+                                 IRTransformLayer::TransformFunction T);
+  };
+
   static Expected<std::unique_ptr<LLJIT>> Create(LLJITBuilderState &S);
 
   /// Destruct this instance. If a multi-threaded instance, waits for all
@@ -52,7 +68,7 @@ public:
   const DataLayout &getDataLayout() const { return DL; }
 
   /// Returns a reference to the JITDylib representing the JIT'd main program.
-  JITDylib &getMainJITDylib() { return Main; }
+  JITDylib &getMainJITDylib() { return *Main; }
 
   /// Returns the JITDylib with the given name, or nullptr if no JITDylib with
   /// that name exists.
@@ -66,7 +82,7 @@ public:
   /// input or elsewhere in the environment then the client should check
   /// (e.g. by calling getJITDylibByName) that the given name is not already in
   /// use.
-  JITDylib &createJITDylib(std::string Name) {
+  Expected<JITDylib &> createJITDylib(std::string Name) {
     return ES->createJITDylib(std::move(Name));
   }
 
@@ -78,7 +94,7 @@ public:
 
   /// Adds an IR module to the Main JITDylib.
   Error addIRModule(ThreadSafeModule TSM) {
-    return addIRModule(Main, std::move(TSM));
+    return addIRModule(*Main, std::move(TSM));
   }
 
   /// Adds an object file to the given JITDylib.
@@ -86,7 +102,7 @@ public:
 
   /// Adds an object file to the given JITDylib.
   Error addObjectFile(std::unique_ptr<MemoryBuffer> Obj) {
-    return addObjectFile(Main, std::move(Obj));
+    return addObjectFile(*Main, std::move(Obj));
   }
 
   /// Look up a symbol in JITDylib JD by the symbol's linker-mangled name (to
@@ -98,7 +114,7 @@ public:
   /// (to look up symbols based on their IR name use the lookup function
   /// instead).
   Expected<JITEvaluatedSymbol> lookupLinkerMangled(StringRef Name) {
-    return lookupLinkerMangled(Main, Name);
+    return lookupLinkerMangled(*Main, Name);
   }
 
   /// Look up a symbol in JITDylib JD based on its IR symbol name.
@@ -108,14 +124,28 @@ public:
 
   /// Look up a symbol in the main JITDylib based on its IR symbol name.
   Expected<JITEvaluatedSymbol> lookup(StringRef UnmangledName) {
-    return lookup(Main, UnmangledName);
+    return lookup(*Main, UnmangledName);
   }
 
-  /// Runs all not-yet-run static constructors.
-  Error runConstructors() { return CtorRunner.run(); }
+  /// Set the PlatformSupport instance.
+  void setPlatformSupport(std::unique_ptr<PlatformSupport> PS) {
+    this->PS = std::move(PS);
+  }
 
-  /// Runs all not-yet-run static destructors.
-  Error runDestructors() { return DtorRunner.run(); }
+  /// Get the PlatformSupport instance.
+  PlatformSupport *getPlatformSupport() { return PS.get(); }
+
+  /// Run the initializers for the given JITDylib.
+  Error initialize(JITDylib &JD) {
+    assert(PS && "PlatformSupport must be set to run initializers.");
+    return PS->initialize(JD);
+  }
+
+  /// Run the deinitializers for the given JITDylib.
+  Error deinitialize(JITDylib &JD) {
+    assert(PS && "PlatformSupport must be set to run initializers.");
+    return PS->deinitialize(JD);
+  }
 
   /// Returns a reference to the ObjLinkingLayer
   ObjectLayer &getObjLinkingLayer() { return *ObjLinkingLayer; }
@@ -126,6 +156,9 @@ public:
   /// Returns a reference to the IR transform layer.
   IRTransformLayer &getIRTransformLayer() { return *TransformLayer; }
 
+  /// Returns a reference to the IR compile layer.
+  IRCompileLayer &getIRCompileLayer() { return *CompileLayer; }
+
 protected:
   static std::unique_ptr<ObjectLayer>
   createObjectLinkingLayer(LLJITBuilderState &S, ExecutionSession &ES);
@@ -143,7 +176,9 @@ protected:
   void recordCtorDtors(Module &M);
 
   std::unique_ptr<ExecutionSession> ES;
-  JITDylib &Main;
+  std::unique_ptr<PlatformSupport> PS;
+
+  JITDylib *Main = nullptr;
 
   DataLayout DL;
   Triple TT;
@@ -153,8 +188,7 @@ protected:
   ObjectTransformLayer ObjTransformLayer;
   std::unique_ptr<IRCompileLayer> CompileLayer;
   std::unique_ptr<IRTransformLayer> TransformLayer;
-
-  CtorDtorRunner CtorRunner, DtorRunner;
+  std::unique_ptr<IRTransformLayer> InitHelperTransformLayer;
 };
 
 /// An extended version of LLJIT that supports lazy function-at-a-time
@@ -175,7 +209,7 @@ public:
 
   /// Add a module to be lazily compiled to the main JITDylib.
   Error addLazyIRModule(ThreadSafeModule M) {
-    return addLazyIRModule(Main, std::move(M));
+    return addLazyIRModule(*Main, std::move(M));
   }
 
 private:
@@ -196,10 +230,14 @@ public:
       std::function<Expected<std::unique_ptr<IRCompileLayer::IRCompiler>>(
           JITTargetMachineBuilder JTMB)>;
 
+  using PlatformSetupFunction = std::function<Error(LLJIT &J)>;
+
   std::unique_ptr<ExecutionSession> ES;
   Optional<JITTargetMachineBuilder> JTMB;
+  Optional<DataLayout> DL;
   ObjectLinkingLayerCreator CreateObjectLinkingLayer;
   CompileFunctionCreator CreateCompileFunction;
+  PlatformSetupFunction SetUpPlatform;
   unsigned NumCompileThreads = 0;
 
   /// Called prior to JIT class construcion to fix up defaults.
@@ -224,6 +262,13 @@ public:
     return impl().JTMB;
   }
 
+  /// Set a DataLayout for this instance. If no data layout is specified then
+  /// the target's default data layout will be used.
+  SetterImpl &setDataLayout(Optional<DataLayout> DL) {
+    impl().DL = std::move(DL);
+    return impl();
+  }
+
   /// Set an ObjectLinkingLayer creation function.
   ///
   /// If this method is not called, a default creation function will be used
@@ -246,6 +291,16 @@ public:
     return impl();
   }
 
+  /// Set up an PlatformSetupFunction.
+  ///
+  /// If this method is not called then setUpGenericLLVMIRPlatform
+  /// will be used to configure the JIT's platform support.
+  SetterImpl &
+  setPlatformSetUp(LLJITBuilderState::PlatformSetupFunction SetUpPlatform) {
+    impl().SetUpPlatform = std::move(SetUpPlatform);
+    return impl();
+  }
+
   /// Set the number of compile threads to use.
   ///
   /// If set to zero, compilation will be performed on the execution thread when
@@ -334,6 +389,26 @@ class LLLazyJITBuilder
       public LLLazyJITBuilderSetters<LLLazyJIT, LLLazyJITBuilder,
                                      LLLazyJITBuilderState> {};
 
+/// Configure the LLJIT instance to scrape modules for llvm.global_ctors and
+/// llvm.global_dtors variables and (if present) build initialization and
+/// deinitialization functions. Platform specific initialization configurations
+/// should be preferred where available.
+void setUpGenericLLVMIRPlatform(LLJIT &J);
+
+/// Configure the LLJIT instance to use MachOPlatform support.
+///
+/// Warning: MachOPlatform *requires* that LLJIT be configured to use
+/// ObjectLinkingLayer (default on platforms supported by JITLink). If
+/// MachOPlatform is used with RTDyldObjectLinkingLayer it will result in
+/// undefined behavior).
+///
+/// MachOPlatform installs an ObjectLinkingLayer plugin to scrape initializers
+/// from the __mod_inits section. It also provides interposes for the dlfcn
+/// functions (dlopen, dlclose, dlsym, dlerror) that work for JITDylibs as
+/// well as regular libraries (JITDylibs will be preferenced, so make sure
+/// your JITDylib names do not shadow any real library paths).
+Error setUpMachOPlatform(LLJIT &J);
+
 } // End namespace orc
 } // End namespace llvm
 

llvm/include/llvm/ExecutionEngine/Orc/Layer.h

@@ -14,6 +14,7 @@
 #define LLVM_EXECUTIONENGINE_ORC_LAYER_H
 
 #include "llvm/ExecutionEngine/Orc/Core.h"
+#include "llvm/ExecutionEngine/Orc/Mangling.h"
 #include "llvm/ExecutionEngine/Orc/ThreadSafeModule.h"
 #include "llvm/IR/Module.h"
 #include "llvm/Support/MemoryBuffer.h"
@@ -27,15 +28,12 @@ namespace orc {
 /// their linkage is changed to available-externally.
 class IRMaterializationUnit : public MaterializationUnit {
 public:
-  struct ManglingOptions {
-    bool EmulatedTLS = false;
-  };
-
   using SymbolNameToDefinitionMap = std::map<SymbolStringPtr, GlobalValue *>;
 
   /// Create an IRMaterializationLayer. Scans the module to build the
   /// SymbolFlags and SymbolToDefinition maps.
-  IRMaterializationUnit(ExecutionSession &ES, const ManglingOptions &MO,
+  IRMaterializationUnit(ExecutionSession &ES,
+                        const IRSymbolMapper::ManglingOptions &MO,
                         ThreadSafeModule TSM, VModuleKey K);
 
   /// Create an IRMaterializationLayer from a module, and pre-existing
@@ -44,12 +42,13 @@ public:
   /// This constructor is useful for delegating work from one
   /// IRMaterializationUnit to another.
   IRMaterializationUnit(ThreadSafeModule TSM, VModuleKey K,
-                        SymbolFlagsMap SymbolFlags,
+                        SymbolFlagsMap SymbolFlags, SymbolStringPtr InitSymbol,
                         SymbolNameToDefinitionMap SymbolToDefinition);
 
   /// Return the ModuleIdentifier as the name for this MaterializationUnit.
   StringRef getName() const override;
 
+  /// Return a reference to the contained ThreadSafeModule.
   const ThreadSafeModule &getModule() const { return TSM; }
 
 protected:
@@ -57,14 +56,16 @@ protected:
   SymbolNameToDefinitionMap SymbolToDefinition;
 
 private:
+  static SymbolStringPtr getInitSymbol(ExecutionSession &ES,
+                                       const ThreadSafeModule &TSM);
+
   void discard(const JITDylib &JD, const SymbolStringPtr &Name) override;
 };
 
 /// Interface for layers that accept LLVM IR.
 class IRLayer {
 public:
-  IRLayer(ExecutionSession &ES,
-          const IRMaterializationUnit::ManglingOptions *&MO)
+  IRLayer(ExecutionSession &ES, const IRSymbolMapper::ManglingOptions *&MO)
       : ES(ES), MO(MO) {}
 
   virtual ~IRLayer();
@@ -73,7 +74,7 @@ public:
   ExecutionSession &getExecutionSession() { return ES; }
 
   /// Get the mangling options for this layer.
-  const IRMaterializationUnit::ManglingOptions *&getManglingOptions() const {
+  const IRSymbolMapper::ManglingOptions *&getManglingOptions() const {
     return MO;
   }
 
@@ -104,14 +105,15 @@ public:
 private:
   bool CloneToNewContextOnEmit = false;
   ExecutionSession &ES;
-  const IRMaterializationUnit::ManglingOptions *&MO;
+  const IRSymbolMapper::ManglingOptions *&MO;
 };
 
 /// MaterializationUnit that materializes modules by calling the 'emit' method
 /// on the given IRLayer.
 class BasicIRLayerMaterializationUnit : public IRMaterializationUnit {
 public:
-  BasicIRLayerMaterializationUnit(IRLayer &L, const ManglingOptions &MO,
+  BasicIRLayerMaterializationUnit(IRLayer &L,
+                                  const IRSymbolMapper::ManglingOptions &MO,
                                   ThreadSafeModule TSM, VModuleKey K);
 
 private:
@@ -153,7 +155,8 @@ public:
 
   BasicObjectLayerMaterializationUnit(ObjectLayer &L, VModuleKey K,
                                       std::unique_ptr<MemoryBuffer> O,
-                                      SymbolFlagsMap SymbolFlags);
+                                      SymbolFlagsMap SymbolFlags,
+                                      SymbolStringPtr InitSymbol);
 
   /// Return the buffer's identifier as the name for this MaterializationUnit.
   StringRef getName() const override;
@@ -167,12 +170,6 @@ private:
   std::unique_ptr<MemoryBuffer> O;
 };
 
-/// Returns a SymbolFlagsMap for the object file represented by the given
-/// buffer, or an error if the buffer does not contain a valid object file.
-// FIXME: Maybe move to Core.h?
-Expected<SymbolFlagsMap> getObjectSymbolFlags(ExecutionSession &ES,
-                                              MemoryBufferRef ObjBuffer);
-
 } // End namespace orc
 } // End namespace llvm
 

llvm/include/llvm/ExecutionEngine/Orc/MachOPlatform.h

@@ -0,0 +1,147 @@
+//===-- MachOPlatform.h - Utilities for executing MachO in Orc --*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Utilities for executing JIT'd MachO in Orc.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_EXECUTIONENGINE_ORC_MACHOPLATFORM_H
+#define LLVM_EXECUTIONENGINE_ORC_MACHOPLATFORM_H
+
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ExecutionEngine/Orc/Core.h"
+#include "llvm/ExecutionEngine/Orc/ExecutionUtils.h"
+#include "llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h"
+
+#include <future>
+#include <thread>
+#include <vector>
+
+namespace llvm {
+namespace orc {
+
+/// Enable registration of JIT'd ObjC classes and selectors.
+Error enableObjCRegistration(const char *PathToLibObjC);
+bool objCRegistrationEnabled();
+
+class MachOJITDylibInitializers {
+public:
+  struct SectionExtent {
+    SectionExtent() = default;
+    SectionExtent(JITTargetAddress Address, uint64_t NumPtrs)
+        : Address(Address), NumPtrs(NumPtrs) {}
+    JITTargetAddress Address = 0;
+    uint64_t NumPtrs = 0;
+  };
+
+  void setObjCImageInfoAddr(JITTargetAddress ObjCImageInfoAddr) {
+    this->ObjCImageInfoAddr = ObjCImageInfoAddr;
+  }
+
+  void addModInitsSection(SectionExtent ModInit) {
+    ModInitSections.push_back(std::move(ModInit));
+  }
+
+  void addObjCSelRefsSection(SectionExtent ObjCSelRefs) {
+    ObjCSelRefsSections.push_back(std::move(ObjCSelRefs));
+  }
+
+  void addObjCClassListSection(SectionExtent ObjCClassList) {
+    ObjCClassListSections.push_back(std::move(ObjCClassList));
+  }
+
+  void runModInits() const;
+  void registerObjCSelectors() const;
+  Error registerObjCClasses() const;
+
+  void dump() const;
+
+private:
+  using RawPointerSectionList = std::vector<SectionExtent>;
+
+  JITTargetAddress ObjCImageInfoAddr;
+  RawPointerSectionList ModInitSections;
+  RawPointerSectionList ObjCSelRefsSections;
+  RawPointerSectionList ObjCClassListSections;
+};
+
+class MachOJITDylibDeinitializers {};
+
+/// Mediates between MachO initialization and ExecutionSession state.
+class MachOPlatform : public Platform {
+public:
+  using InitializerSequence =
+      std::vector<std::pair<JITDylib *, MachOJITDylibInitializers>>;
+
+  using DeinitializerSequence =
+      std::vector<std::pair<JITDylib *, MachOJITDylibDeinitializers>>;
+
+  MachOPlatform(ExecutionSession &ES, ObjectLinkingLayer &ObjLinkingLayer,
+                std::unique_ptr<MemoryBuffer> StandardSymbolsObject);
+
+  ExecutionSession &getExecutionSession() const { return ES; }
+
+  Error setupJITDylib(JITDylib &JD) override;
+  Error notifyAdding(JITDylib &JD, const MaterializationUnit &MU) override;
+  Error notifyRemoving(JITDylib &JD, VModuleKey K) override;
+
+  Expected<InitializerSequence> getInitializerSequence(JITDylib &JD);
+
+  Expected<DeinitializerSequence> getDeinitializerSequence(JITDylib &JD);
+
+private:
+  // This ObjectLinkingLayer plugin scans JITLink graphs for __mod_init_func,
+  // __objc_classlist and __sel_ref sections and records their extents so that
+  // they can be run in the target process.
+  class InitScraperPlugin : public ObjectLinkingLayer::Plugin {
+  public:
+    InitScraperPlugin(MachOPlatform &MP) : MP(MP) {}
+
+    void modifyPassConfig(MaterializationResponsibility &MR, const Triple &TT,
+                          jitlink::PassConfiguration &Config) override;
+
+    LocalDependenciesMap getSyntheticSymbolLocalDependencies(
+        MaterializationResponsibility &MR) override;
+
+  private:
+    using InitSymbolDepMap =
+        DenseMap<MaterializationResponsibility *, JITLinkSymbolVector>;
+
+    void preserveInitSectionIfPresent(JITLinkSymbolVector &Syms,
+                                      jitlink::LinkGraph &G,
+                                      StringRef SectionName);
+
+    Error processObjCImageInfo(jitlink::LinkGraph &G,
+                               MaterializationResponsibility &MR);
+
+    std::mutex InitScraperMutex;
+    MachOPlatform &MP;
+    DenseMap<JITDylib *, std::pair<uint32_t, uint32_t>> ObjCImageInfos;
+    InitSymbolDepMap InitSymbolDeps;
+  };
+
+  static std::vector<JITDylib *> getDFSLinkOrder(JITDylib &JD);
+
+  void registerInitInfo(JITDylib &JD, JITTargetAddress ObjCImageInfoAddr,
+                        MachOJITDylibInitializers::SectionExtent ModInits,
+                        MachOJITDylibInitializers::SectionExtent ObjCSelRefs,
+                        MachOJITDylibInitializers::SectionExtent ObjCClassList);
+
+  std::mutex PlatformMutex;
+  ExecutionSession &ES;
+  ObjectLinkingLayer &ObjLinkingLayer;
+  std::unique_ptr<MemoryBuffer> StandardSymbolsObject;
+
+  DenseMap<JITDylib *, SymbolLookupSet> RegisteredInitSymbols;
+  DenseMap<JITDylib *, MachOJITDylibInitializers> InitSeqs;
+};
+
+} // end namespace orc
+} // end namespace llvm
+
+#endif // LLVM_EXECUTIONENGINE_ORC_MACHOPLATFORM_H

llvm/include/llvm/ExecutionEngine/Orc/Mangling.h

@@ -0,0 +1,66 @@
+//===------ Mangling.h -- Name Mangling Utilities for ORC -------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Name mangling utilities for ORC.
+//
+//===----------------------------------------------------------------------===//
+
+#ifndef LLVM_EXECUTIONENGINE_ORC_MANGLING_H
+#define LLVM_EXECUTIONENGINE_ORC_MANGLING_H
+
+#include "llvm/ExecutionEngine/Orc/Core.h"
+#include "llvm/ExecutionEngine/Orc/ThreadSafeModule.h"
+#include "llvm/IR/Module.h"
+#include "llvm/Support/MemoryBuffer.h"
+
+namespace llvm {
+namespace orc {
+
+/// Mangles symbol names then uniques them in the context of an
+/// ExecutionSession.
+class MangleAndInterner {
+public:
+  MangleAndInterner(ExecutionSession &ES, const DataLayout &DL);
+  SymbolStringPtr operator()(StringRef Name);
+
+private:
+  ExecutionSession &ES;
+  const DataLayout &DL;
+};
+
+/// Maps IR global values to their linker symbol names / flags.
+///
+/// This utility can be used when adding new IR globals in the JIT.
+class IRSymbolMapper {
+public:
+  struct ManglingOptions {
+    bool EmulatedTLS = false;
+  };
+
+  using SymbolNameToDefinitionMap = std::map<SymbolStringPtr, GlobalValue *>;
+
+  /// Add mangled symbols for the given GlobalValues to SymbolFlags.
+  /// If a SymbolToDefinitionMap pointer is supplied then it will be populated
+  /// with Name-to-GlobalValue* mappings. Note that this mapping is not
+  /// necessarily one-to-one: thread-local GlobalValues, for example, may
+  /// produce more than one symbol, in which case the map will contain duplicate
+  /// values.
+  static void add(ExecutionSession &ES, const ManglingOptions &MO,
+                  ArrayRef<GlobalValue *> GVs, SymbolFlagsMap &SymbolFlags,
+                  SymbolNameToDefinitionMap *SymbolToDefinition = nullptr);
+};
+
+/// Returns a SymbolFlagsMap for the object file represented by the given
+/// buffer, or an error if the buffer does not contain a valid object file.
+Expected<std::pair<SymbolFlagsMap, SymbolStringPtr>>
+getObjectSymbolInfo(ExecutionSession &ES, MemoryBufferRef ObjBuffer);
+
+} // End namespace orc
+} // End namespace llvm
+
+#endif // LLVM_EXECUTIONENGINE_ORC_MANGLING_H

llvm/include/llvm/ExecutionEngine/Orc/ObjectLinkingLayer.h

@@ -35,6 +35,7 @@ namespace llvm {
 
 namespace jitlink {
 class EHFrameRegistrar;
+class Symbol;
 } // namespace jitlink
 
 namespace object {
@@ -59,10 +60,14 @@ public:
   /// configured.
   class Plugin {
   public:
+    using JITLinkSymbolVector = std::vector<const jitlink::Symbol *>;
+    using LocalDependenciesMap = DenseMap<SymbolStringPtr, JITLinkSymbolVector>;
+
     virtual ~Plugin();
     virtual void modifyPassConfig(MaterializationResponsibility &MR,
                                   const Triple &TT,
                                   jitlink::PassConfiguration &Config) {}
+
     virtual void notifyLoaded(MaterializationResponsibility &MR) {}
     virtual Error notifyEmitted(MaterializationResponsibility &MR) {
       return Error::success();
@@ -71,6 +76,15 @@ public:
       return Error::success();
     }
     virtual Error notifyRemovingAllModules() { return Error::success(); }
+
+    /// Return any dependencies that synthetic symbols (e.g. init symbols)
+    /// have on locally scoped jitlink::Symbols. This is used by the
+    /// ObjectLinkingLayer to update the dependencies for the synthetic
+    /// symbols.
+    virtual LocalDependenciesMap
+    getSyntheticSymbolLocalDependencies(MaterializationResponsibility &MR) {
+      return LocalDependenciesMap();
+    }
   };
 
   using ReturnObjectBufferFunction =

llvm/include/llvm/ExecutionEngine/Orc/SymbolStringPool.h

@@ -53,6 +53,7 @@ class SymbolStringPtr {
 
 public:
   SymbolStringPtr() = default;
+  SymbolStringPtr(nullptr_t) {}
   SymbolStringPtr(const SymbolStringPtr &Other)
     : S(Other.S) {
     if (isRealPoolEntry(S))
@@ -85,6 +86,8 @@ public:
       --S->getValue();
   }
 
+  explicit operator bool() const { return S; }
+
   StringRef operator*() const { return S->first(); }
 
   friend bool operator==(const SymbolStringPtr &LHS,

llvm/lib/ExecutionEngine/Orc/CMakeLists.txt

@@ -12,6 +12,8 @@ add_llvm_component_library(LLVMOrcJIT
   Legacy.cpp
   Layer.cpp
   LLJIT.cpp
+  MachOPlatform.cpp
+  Mangling.cpp
   NullResolver.cpp
   ObjectLinkingLayer.cpp
   ObjectTransformLayer.cpp

llvm/lib/ExecutionEngine/Orc/CompileOnDemandLayer.cpp

@@ -7,6 +7,7 @@
 //===----------------------------------------------------------------------===//
 
 #include "llvm/ExecutionEngine/Orc/CompileOnDemandLayer.h"
+#include "llvm/ExecutionEngine/Orc/ExecutionUtils.h"
 #include "llvm/IR/Mangler.h"
 #include "llvm/IR/Module.h"
 
@@ -68,18 +69,18 @@ namespace orc {
 class PartitioningIRMaterializationUnit : public IRMaterializationUnit {
 public:
   PartitioningIRMaterializationUnit(ExecutionSession &ES,
-                                    const ManglingOptions &MO,
+                                    const IRSymbolMapper::ManglingOptions &MO,
                                     ThreadSafeModule TSM, VModuleKey K,
                                     CompileOnDemandLayer &Parent)
       : IRMaterializationUnit(ES, MO, std::move(TSM), std::move(K)),
         Parent(Parent) {}
 
   PartitioningIRMaterializationUnit(
-      ThreadSafeModule TSM, SymbolFlagsMap SymbolFlags,
-      SymbolNameToDefinitionMap SymbolToDefinition,
+      ThreadSafeModule TSM, VModuleKey K, SymbolFlagsMap SymbolFlags,
+      SymbolStringPtr InitSymbol, SymbolNameToDefinitionMap SymbolToDefinition,
       CompileOnDemandLayer &Parent)
       : IRMaterializationUnit(std::move(TSM), std::move(K),
-                              std::move(SymbolFlags),
+                              std::move(SymbolFlags), std::move(InitSymbol),
                               std::move(SymbolToDefinition)),
         Parent(Parent) {}
 
@@ -172,21 +173,23 @@ CompileOnDemandLayer::getPerDylibResources(JITDylib &TargetD) {
   auto I = DylibResources.find(&TargetD);
   if (I == DylibResources.end()) {
     auto &ImplD =
-        getExecutionSession().createJITDylib(TargetD.getName() + ".impl");
+        getExecutionSession().createBareJITDylib(TargetD.getName() + ".impl");
+    JITDylibSearchOrder NewSearchOrder;
     TargetD.withSearchOrderDo(
         [&](const JITDylibSearchOrder &TargetSearchOrder) {
-          auto NewSearchOrder = TargetSearchOrder;
-          assert(
-              !NewSearchOrder.empty() &&
-              NewSearchOrder.front().first == &TargetD &&
-              NewSearchOrder.front().second ==
-                  JITDylibLookupFlags::MatchAllSymbols &&
-              "TargetD must be at the front of its own search order and match "
-              "non-exported symbol");
-          NewSearchOrder.insert(std::next(NewSearchOrder.begin()),
-                                {&ImplD, JITDylibLookupFlags::MatchAllSymbols});
-          ImplD.setSearchOrder(std::move(NewSearchOrder), false);
+          NewSearchOrder = TargetSearchOrder;
         });
+
+    assert(
+        !NewSearchOrder.empty() && NewSearchOrder.front().first == &TargetD &&
+        NewSearchOrder.front().second == JITDylibLookupFlags::MatchAllSymbols &&
+        "TargetD must be at the front of its own search order and match "
+        "non-exported symbol");
+    NewSearchOrder.insert(std::next(NewSearchOrder.begin()),
+                          {&ImplD, JITDylibLookupFlags::MatchAllSymbols});
+    ImplD.setSearchOrder(NewSearchOrder, false);
+    TargetD.setSearchOrder(std::move(NewSearchOrder), false);
+
     PerDylibResources PDR(ImplD, BuildIndirectStubsManager());
     I = DylibResources.insert(std::make_pair(&TargetD, std::move(PDR))).first;
   }
@@ -251,8 +254,15 @@ void CompileOnDemandLayer::emitPartition(
   auto &ES = getExecutionSession();
   GlobalValueSet RequestedGVs;
   for (auto &Name : R.getRequestedSymbols()) {
-    assert(Defs.count(Name) && "No definition for symbol");
-    RequestedGVs.insert(Defs[Name]);
+    if (Name == R.getInitializerSymbol())
+      TSM.withModuleDo([&](Module &M) {
+        for (auto &GV : getStaticInitGVs(M))
+          RequestedGVs.insert(&GV);
+      });
+    else {
+      assert(Defs.count(Name) && "No definition for symbol");
+      RequestedGVs.insert(Defs[Name]);
+    }
   }
 
   /// Perform partitioning with the context lock held, since the partition
@@ -272,7 +282,8 @@ void CompileOnDemandLayer::emitPartition(
   // If the partition is empty, return the whole module to the symbol table.
   if (GVsToExtract->empty()) {
     R.replace(std::make_unique<PartitioningIRMaterializationUnit>(
-        std::move(TSM), R.getSymbols(), std::move(Defs), *this));
+        std::move(TSM), R.getVModuleKey(), R.getSymbols(),
+        R.getInitializerSymbol(), std::move(Defs), *this));
     return;
   }
 

llvm/lib/ExecutionEngine/Orc/CompileUtils.cpp

@@ -24,9 +24,9 @@
 namespace llvm {
 namespace orc {
 
-IRMaterializationUnit::ManglingOptions
+IRSymbolMapper::ManglingOptions
 irManglingOptionsFromTargetOptions(const TargetOptions &Opts) {
-  IRMaterializationUnit::ManglingOptions MO;
+  IRSymbolMapper::ManglingOptions MO;
 
   MO.EmulatedTLS = Opts.EmulatedTLS;
 

llvm/lib/ExecutionEngine/Orc/Core.cpp

@@ -11,7 +11,6 @@
 #include "llvm/ADT/STLExtras.h"
 #include "llvm/Config/llvm-config.h"
 #include "llvm/ExecutionEngine/Orc/OrcError.h"
-#include "llvm/IR/Mangler.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Debug.h"
 #include "llvm/Support/Format.h"
@@ -421,12 +420,6 @@ void AsynchronousSymbolQuery::detach() {
   QueryRegistrations.clear();
 }
 
-MaterializationResponsibility::MaterializationResponsibility(
-    JITDylib &JD, SymbolFlagsMap SymbolFlags, VModuleKey K)
-    : JD(JD), SymbolFlags(std::move(SymbolFlags)), K(std::move(K)) {
-  assert(!this->SymbolFlags.empty() && "Materializing nothing?");
-}
-
 MaterializationResponsibility::~MaterializationResponsibility() {
   assert(SymbolFlags.empty() &&
          "All symbols should have been explicitly materialized or failed");
@@ -501,9 +494,13 @@ void MaterializationResponsibility::failMaterialization() {
 
 void MaterializationResponsibility::replace(
     std::unique_ptr<MaterializationUnit> MU) {
+
   for (auto &KV : MU->getSymbols())
     SymbolFlags.erase(KV.first);
 
+  if (MU->getInitializerSymbol() == InitSymbol)
+    InitSymbol = nullptr;
+
   LLVM_DEBUG(JD.getExecutionSession().runSessionLocked([&]() {
     dbgs() << "In " << JD.getName() << " replacing symbols with " << *MU
            << "\n";
@@ -519,6 +516,7 @@ MaterializationResponsibility::delegate(const SymbolNameSet &Symbols,
   if (NewKey == VModuleKey())
     NewKey = K;
 
+  SymbolStringPtr DelegatedInitSymbol;
   SymbolFlagsMap DelegatedFlags;
 
   for (auto &Name : Symbols) {
@@ -528,10 +526,14 @@ MaterializationResponsibility::delegate(const SymbolNameSet &Symbols,
            "instance");
 
     DelegatedFlags[Name] = std::move(I->second);
+    if (Name == InitSymbol)
+      std::swap(InitSymbol, DelegatedInitSymbol);
+
     SymbolFlags.erase(I);
   }
 
   return MaterializationResponsibility(JD, std::move(DelegatedFlags),
+                                       std::move(DelegatedInitSymbol),
                                        std::move(NewKey));
 }
 
@@ -550,7 +552,7 @@ void MaterializationResponsibility::addDependenciesForAll(
 
 AbsoluteSymbolsMaterializationUnit::AbsoluteSymbolsMaterializationUnit(
     SymbolMap Symbols, VModuleKey K)
-    : MaterializationUnit(extractFlags(Symbols), std::move(K)),
+    : MaterializationUnit(extractFlags(Symbols), nullptr, std::move(K)),
       Symbols(std::move(Symbols)) {}
 
 StringRef AbsoluteSymbolsMaterializationUnit::getName() const {
@@ -581,7 +583,7 @@ AbsoluteSymbolsMaterializationUnit::extractFlags(const SymbolMap &Symbols) {
 ReExportsMaterializationUnit::ReExportsMaterializationUnit(
     JITDylib *SourceJD, JITDylibLookupFlags SourceJDLookupFlags,
     SymbolAliasMap Aliases, VModuleKey K)
-    : MaterializationUnit(extractFlags(Aliases), std::move(K)),
+    : MaterializationUnit(extractFlags(Aliases), nullptr, std::move(K)),
       SourceJD(SourceJD), SourceJDLookupFlags(SourceJDLookupFlags),
       Aliases(std::move(Aliases)) {}
 
@@ -972,7 +974,7 @@ void JITDylib::addDependencies(const SymbolStringPtr &Name,
       // Assert that this symbol exists and has not reached the ready state
       // already.
       assert(OtherSymI != OtherJITDylib.Symbols.end() &&
-             (OtherSymI->second.getState() != SymbolState::Ready &&
+             (OtherSymI->second.getState() < SymbolState::Ready &&
               "Dependency on emitted/ready symbol"));
 #endif
 
@@ -1101,6 +1103,7 @@ Error JITDylib::resolve(const SymbolMap &Resolved) {
 Error JITDylib::emit(const SymbolFlagsMap &Emitted) {
   AsynchronousSymbolQuerySet CompletedQueries;
   SymbolNameSet SymbolsInErrorState;
+  DenseMap<JITDylib *, SymbolNameVector> ReadySymbols;
 
   ES.runSessionLocked([&, this]() {
     std::vector<SymbolTable::iterator> Worklist;
@@ -1145,6 +1148,7 @@ Error JITDylib::emit(const SymbolFlagsMap &Emitted) {
       // dependencies) then notify any pending queries.
       for (auto &KV : MI.Dependants) {
         auto &DependantJD = *KV.first;
+        auto &DependantJDReadySymbols = ReadySymbols[&DependantJD];
         for (auto &DependantName : KV.second) {
           auto DependantMII =
               DependantJD.MaterializingInfos.find(DependantName);
@@ -1184,6 +1188,7 @@ Error JITDylib::emit(const SymbolFlagsMap &Emitted) {
             // Since this dependant is now ready, we erase its MaterializingInfo
             // and update its materializing state.
             DependantSymEntry.setState(SymbolState::Ready);
+            DependantJDReadySymbols.push_back(DependantName);
 
             for (auto &Q : DependantMI.takeQueriesMeeting(SymbolState::Ready)) {
               Q->notifySymbolMetRequiredState(
@@ -1192,22 +1197,21 @@ Error JITDylib::emit(const SymbolFlagsMap &Emitted) {
                 CompletedQueries.insert(Q);
               Q->removeQueryDependence(DependantJD, DependantName);
             }
-
-            DependantJD.MaterializingInfos.erase(DependantMII);
           }
         }
       }
 
+      auto &ThisJDReadySymbols = ReadySymbols[this];
       MI.Dependants.clear();
       if (MI.UnemittedDependencies.empty()) {
         SymI->second.setState(SymbolState::Ready);
+        ThisJDReadySymbols.push_back(Name);
         for (auto &Q : MI.takeQueriesMeeting(SymbolState::Ready)) {
           Q->notifySymbolMetRequiredState(Name, SymI->second.getSymbol());
           if (Q->isComplete())
             CompletedQueries.insert(Q);
           Q->removeQueryDependence(*this, Name);
         }
-        MaterializingInfos.erase(MII);
       }
     }
   });
@@ -1882,6 +1886,57 @@ void JITDylib::transferEmittedNodeDependencies(
   }
 }
 
+Platform::~Platform() {}
+
+Expected<DenseMap<JITDylib *, SymbolMap>> Platform::lookupInitSymbols(
+    ExecutionSession &ES,
+    const DenseMap<JITDylib *, SymbolLookupSet> &InitSyms) {
+
+  DenseMap<JITDylib *, SymbolMap> CompoundResult;
+  Error CompoundErr = Error::success();
+  std::mutex LookupMutex;
+  std::condition_variable CV;
+  uint64_t Count = InitSyms.size();
+
+  LLVM_DEBUG({
+    dbgs() << "Issuing init-symbol lookup:\n";
+    for (auto &KV : InitSyms)
+      dbgs() << "  " << KV.first->getName() << ": " << KV.second << "\n";
+  });
+
+  for (auto &KV : InitSyms) {
+    auto *JD = KV.first;
+    auto Names = std::move(KV.second);
+    ES.lookup(
+        LookupKind::Static,
+        JITDylibSearchOrder({{JD, JITDylibLookupFlags::MatchAllSymbols}}),
+        std::move(Names), SymbolState::Ready,
+        [&, JD](Expected<SymbolMap> Result) {
+          {
+            std::lock_guard<std::mutex> Lock(LookupMutex);
+            --Count;
+            if (Result) {
+              assert(!CompoundResult.count(JD) &&
+                     "Duplicate JITDylib in lookup?");
+              CompoundResult[JD] = std::move(*Result);
+            } else
+              CompoundErr =
+                  joinErrors(std::move(CompoundErr), Result.takeError());
+          }
+          CV.notify_one();
+        },
+        NoDependenciesToRegister);
+  }
+
+  std::unique_lock<std::mutex> Lock(LookupMutex);
+  CV.wait(Lock, [&] { return Count == 0 || CompoundErr; });
+
+  if (CompoundErr)
+    return std::move(CompoundErr);
+
+  return std::move(CompoundResult);
+}
+
 ExecutionSession::ExecutionSession(std::shared_ptr<SymbolStringPool> SSP)
     : SSP(SSP ? std::move(SSP) : std::make_shared<SymbolStringPool>()) {
 }
@@ -1895,7 +1950,7 @@ JITDylib *ExecutionSession::getJITDylibByName(StringRef Name) {
   });
 }
 
-JITDylib &ExecutionSession::createJITDylib(std::string Name) {
+JITDylib &ExecutionSession::createBareJITDylib(std::string Name) {
   assert(!getJITDylibByName(Name) && "JITDylib with that name already exists");
   return runSessionLocked([&, this]() -> JITDylib & {
     JDs.push_back(
@@ -1904,6 +1959,14 @@ JITDylib &ExecutionSession::createJITDylib(std::string Name) {
   });
 }
 
+Expected<JITDylib &> ExecutionSession::createJITDylib(std::string Name) {
+  auto &JD = createBareJITDylib(Name);
+  if (P)
+    if (auto Err = P->setupJITDylib(JD))
+      return std::move(Err);
+  return JD;
+}
+
 void ExecutionSession::legacyFailQuery(AsynchronousSymbolQuery &Q, Error Err) {
   assert(!!Err && "Error should be in failure state");
 
@@ -2144,11 +2207,11 @@ ExecutionSession::lookup(const JITDylibSearchOrder &SearchOrder,
 
 Expected<JITEvaluatedSymbol>
 ExecutionSession::lookup(const JITDylibSearchOrder &SearchOrder,
-                         SymbolStringPtr Name) {
+                         SymbolStringPtr Name, SymbolState RequiredState) {
   SymbolLookupSet Names({Name});
 
   if (auto ResultMap = lookup(SearchOrder, std::move(Names), LookupKind::Static,
-                              SymbolState::Ready, NoDependenciesToRegister)) {
+                              RequiredState, NoDependenciesToRegister)) {
     assert(ResultMap->size() == 1 && "Unexpected number of results");
     assert(ResultMap->count(Name) && "Missing result for symbol");
     return std::move(ResultMap->begin()->second);
@@ -2157,14 +2220,15 @@ ExecutionSession::lookup(const JITDylibSearchOrder &SearchOrder,
 }
 
 Expected<JITEvaluatedSymbol>
-ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder,
-                         SymbolStringPtr Name) {
-  return lookup(makeJITDylibSearchOrder(SearchOrder), Name);
+ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder, SymbolStringPtr Name,
+                         SymbolState RequiredState) {
+  return lookup(makeJITDylibSearchOrder(SearchOrder), Name, RequiredState);
 }
 
 Expected<JITEvaluatedSymbol>
-ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder, StringRef Name) {
-  return lookup(SearchOrder, intern(Name));
+ExecutionSession::lookup(ArrayRef<JITDylib *> SearchOrder, StringRef Name,
+                         SymbolState RequiredState) {
+  return lookup(SearchOrder, intern(Name), RequiredState);
 }
 
 void ExecutionSession::dump(raw_ostream &OS) {
@@ -2195,17 +2259,5 @@ void ExecutionSession::runOutstandingMUs() {
   }
 }
 
-MangleAndInterner::MangleAndInterner(ExecutionSession &ES, const DataLayout &DL)
-    : ES(ES), DL(DL) {}
-
-SymbolStringPtr MangleAndInterner::operator()(StringRef Name) {
-  std::string MangledName;
-  {
-    raw_string_ostream MangledNameStream(MangledName);
-    Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
-  }
-  return ES.intern(MangledName);
-}
-
 } // End namespace orc.
 } // End namespace llvm.