Quantcast

JIT doens't resolve address

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

JIT doens't resolve address

Daniel Marjamäki via cfe-dev
Hello LLVM-World,

I was following the "Building a JIT in LLVM"-Tutorial and tried to load a normal main. My code is the following:
class Jitter
{
private:
  std::unique_ptr<TargetMachine> TM;
  const DataLayout DL;
  ObjectLinkingLayer<> ObjectLayer;
  IRCompileLayer<decltype(ObjectLayer)> CompileLayer;

public:
  typedef decltype(CompileLayer)::ModuleSetHandleT ModuleHandle;

  Jitter() : TM(EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
      CompileLayer(ObjectLayer, SimpleCompiler(*TM))
  {printf("!");
        llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
  }

  TargetMachine &getTargetMachine() { return *TM; }

  ModuleHandle addModule(std::unique_ptr<Module> &&M) {
  // Build our symbol resolver:
  // Lambda 1: Look back into the JIT itself to find symbols that are part of
  //           the same "logical dylib".
  // Lambda 2: Search for external symbols in the host process.
  auto Resolver = createLambdaResolver(
      [&](const std::string &Name)
      {
                printf("FLUSH :0\n");

        if (auto Sym = CompileLayer.findSymbol(Name, false))
          return Sym;
        return JITSymbol(nullptr);
      },
      [](const std::string &S)
          {
                  printf("PLUSH :0\n");

        if (auto SymAddr =
              RTDyldMemoryManager::getSymbolAddressInProcess(S))
          return JITSymbol(SymAddr, JITSymbolFlags::Exported);
        return JITSymbol(nullptr);
      });

  // Build a singleton module set to hold our module.
  std::vector<std::unique_ptr<Module>> Ms;
  Ms.push_back(std::move(M));

  // Add the set to the JIT with the resolver we created above and a newly
  // created SectionMemoryManager.
  return CompileLayer.addModuleSet(std::move(Ms),
                                   make_unique<SectionMemoryManager>(),
                                   std::move(Resolver));
}

JITSymbol findSymbol(const std::string Name) {
  std::string MangledName;
  raw_string_ostream MangledNameStream(MangledName);
  Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
  printf("Tzearch for: %s\n\n", MangledNameStream.str());
  return CompileLayer.findSymbol(MangledNameStream.str(), false);
}

void removeModule(ModuleHandle H) {
  CompileLayer.removeModuleSet(H);
}

};


And calling from main with:

int main()
{
        llvm::InitializeNativeTarget();
        llvm::InitializeNativeTargetAsmPrinter();
        llvm::InitializeNativeTargetAsmParser();

        llvm::LLVMContext context;
        llvm::SMDiagnostic dia;

        std::unique_ptr<llvm::Module> M = llvm::parseIRFile("./jit_main.ll", dia, context);
        Jitter jit;
        printf("Wuff?");
        Jitter::ModuleHandle h = jit.addModule(std::move(M));
        printf("KNUFF!\n");

        printf("Kuchen! 0x%p\n", jit.findSymbol("main").getAddress());

        system("PAUSE");
        return 0;
}

The Code runs without a fail, but when the programm tries to resolve "main" the address is 0. The strange thing: the printf "FLUSH :0\n" and "PLUSH :0\n" are never called, so did the code never compiled? What I'm doing wrong?

Kind regards
Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JIT doens't resolve address

Daniel Marjamäki via cfe-dev
HI Bjoern,

There are two kinds of symbol lookup in the JIT:

(1) You want to find a symbol defined JIT'd code. This is what "findSymbol" on the JIT class does.

(2) The JIT wants to fix up a module that contains references to symbols defined elsewhere (either in your program, or in other JIT'd code). This is the SymbolResolver's job.

So assuming your main function in main.ll is trivial, e.g.

int main() {
  return 0;
}

then your findSymbol call will return the address of the JIT'd main without ever needing to look anything up in the resolver.

If, on the other hand, your main function contains an external reference, e.g.

int main() {
  printf("Hello, World!"); // Reference to externally defined function printf.
  return 0;
}

Then you'll receive a call back on your resolver looking for the address of printf.

Hope this helps!

Cheers,
Lang.

On Thu, Apr 20, 2017 at 6:43 AM, via cfe-dev <[hidden email]> wrote:
Hello LLVM-World,

I was following the "Building a JIT in LLVM"-Tutorial and tried to load a normal main. My code is the following:
class Jitter
{
private:
  std::unique_ptr<TargetMachine> TM;
  const DataLayout DL;
  ObjectLinkingLayer<> ObjectLayer;
  IRCompileLayer<decltype(ObjectLayer)> CompileLayer;

public:
  typedef decltype(CompileLayer)::ModuleSetHandleT ModuleHandle;

  Jitter() : TM(EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
      CompileLayer(ObjectLayer, SimpleCompiler(*TM))
  {printf("!");
        llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
  }

  TargetMachine &getTargetMachine() { return *TM; }

  ModuleHandle addModule(std::unique_ptr<Module> &&M) {
  // Build our symbol resolver:
  // Lambda 1: Look back into the JIT itself to find symbols that are part of
  //           the same "logical dylib".
  // Lambda 2: Search for external symbols in the host process.
  auto Resolver = createLambdaResolver(
      [&](const std::string &Name)
      {
                printf("FLUSH :0\n");

        if (auto Sym = CompileLayer.findSymbol(Name, false))
          return Sym;
        return JITSymbol(nullptr);
      },
      [](const std::string &S)
          {
                  printf("PLUSH :0\n");

        if (auto SymAddr =
              RTDyldMemoryManager::getSymbolAddressInProcess(S))
          return JITSymbol(SymAddr, JITSymbolFlags::Exported);
        return JITSymbol(nullptr);
      });

  // Build a singleton module set to hold our module.
  std::vector<std::unique_ptr<Module>> Ms;
  Ms.push_back(std::move(M));

  // Add the set to the JIT with the resolver we created above and a newly
  // created SectionMemoryManager.
  return CompileLayer.addModuleSet(std::move(Ms),
                                   make_unique<SectionMemoryManager>(),
                                   std::move(Resolver));
}

JITSymbol findSymbol(const std::string Name) {
  std::string MangledName;
  raw_string_ostream MangledNameStream(MangledName);
  Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
  printf("Tzearch for: %s\n\n", MangledNameStream.str());
  return CompileLayer.findSymbol(MangledNameStream.str(), false);
}

void removeModule(ModuleHandle H) {
  CompileLayer.removeModuleSet(H);
}

};


And calling from main with:

int main()
{
        llvm::InitializeNativeTarget();
        llvm::InitializeNativeTargetAsmPrinter();
        llvm::InitializeNativeTargetAsmParser();

        llvm::LLVMContext context;
        llvm::SMDiagnostic dia;

        std::unique_ptr<llvm::Module> M = llvm::parseIRFile("./jit_main.ll", dia, context);
        Jitter jit;
        printf("Wuff?");
        Jitter::ModuleHandle h = jit.addModule(std::move(M));
        printf("KNUFF!\n");

        printf("Kuchen! 0x%p\n", jit.findSymbol("main").getAddress());

        system("PAUSE");
        return 0;
}

The Code runs without a fail, but when the programm tries to resolve "main" the address is 0. The strange thing: the printf "FLUSH :0\n" and "PLUSH :0\n" are never called, so did the code never compiled? What I'm doing wrong?

Kind regards
Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JIT doens't resolve address

Daniel Marjamäki via cfe-dev
Hi Bjoern,

Sorry - I just noticed that the address for your "main" function has come back as '0'. In this case the answer is even simpler: The JIT never found the function at all, and so definitely didn't have any external symbols to resolve, so it never called the resolver.

A failure to find main has three likely causes:

(1) jit_main.ll does not define main,
(2) jit_main.ll defines main with private/internal linkage (the JIT can't see private/internal symbols in general)
(3) jit_main.ll defines main with external linkage, but a system mangling is applied (e.g. on MacOSX 'main' is mangled to '_main'). The mangler code in your findSymbol function *should* correct for this, but this may fail if the default data layout for your TargetMachine varies from the Module's DataLayout.

Can you share the contents of your jit_main.ll Module?

Cheers,
Lang.
   

On Thu, Apr 20, 2017 at 1:35 PM, Lang Hames <[hidden email]> wrote:
HI Bjoern,

There are two kinds of symbol lookup in the JIT:

(1) You want to find a symbol defined JIT'd code. This is what "findSymbol" on the JIT class does.

(2) The JIT wants to fix up a module that contains references to symbols defined elsewhere (either in your program, or in other JIT'd code). This is the SymbolResolver's job.

So assuming your main function in main.ll is trivial, e.g.

int main() {
  return 0;
}

then your findSymbol call will return the address of the JIT'd main without ever needing to look anything up in the resolver.

If, on the other hand, your main function contains an external reference, e.g.

int main() {
  printf("Hello, World!"); // Reference to externally defined function printf.
  return 0;
}

Then you'll receive a call back on your resolver looking for the address of printf.

Hope this helps!

Cheers,
Lang.

On Thu, Apr 20, 2017 at 6:43 AM, via cfe-dev <[hidden email]> wrote:
Hello LLVM-World,

I was following the "Building a JIT in LLVM"-Tutorial and tried to load a normal main. My code is the following:
class Jitter
{
private:
  std::unique_ptr<TargetMachine> TM;
  const DataLayout DL;
  ObjectLinkingLayer<> ObjectLayer;
  IRCompileLayer<decltype(ObjectLayer)> CompileLayer;

public:
  typedef decltype(CompileLayer)::ModuleSetHandleT ModuleHandle;

  Jitter() : TM(EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
      CompileLayer(ObjectLayer, SimpleCompiler(*TM))
  {printf("!");
        llvm::sys::DynamicLibrary::LoadLibraryPermanently(nullptr);
  }

  TargetMachine &getTargetMachine() { return *TM; }

  ModuleHandle addModule(std::unique_ptr<Module> &&M) {
  // Build our symbol resolver:
  // Lambda 1: Look back into the JIT itself to find symbols that are part of
  //           the same "logical dylib".
  // Lambda 2: Search for external symbols in the host process.
  auto Resolver = createLambdaResolver(
      [&](const std::string &Name)
      {
                printf("FLUSH :0\n");

        if (auto Sym = CompileLayer.findSymbol(Name, false))
          return Sym;
        return JITSymbol(nullptr);
      },
      [](const std::string &S)
          {
                  printf("PLUSH :0\n");

        if (auto SymAddr =
              RTDyldMemoryManager::getSymbolAddressInProcess(S))
          return JITSymbol(SymAddr, JITSymbolFlags::Exported);
        return JITSymbol(nullptr);
      });

  // Build a singleton module set to hold our module.
  std::vector<std::unique_ptr<Module>> Ms;
  Ms.push_back(std::move(M));

  // Add the set to the JIT with the resolver we created above and a newly
  // created SectionMemoryManager.
  return CompileLayer.addModuleSet(std::move(Ms),
                                   make_unique<SectionMemoryManager>(),
                                   std::move(Resolver));
}

JITSymbol findSymbol(const std::string Name) {
  std::string MangledName;
  raw_string_ostream MangledNameStream(MangledName);
  Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
  printf("Tzearch for: %s\n\n", MangledNameStream.str());
  return CompileLayer.findSymbol(MangledNameStream.str(), false);
}

void removeModule(ModuleHandle H) {
  CompileLayer.removeModuleSet(H);
}

};


And calling from main with:

int main()
{
        llvm::InitializeNativeTarget();
        llvm::InitializeNativeTargetAsmPrinter();
        llvm::InitializeNativeTargetAsmParser();

        llvm::LLVMContext context;
        llvm::SMDiagnostic dia;

        std::unique_ptr<llvm::Module> M = llvm::parseIRFile("./jit_main.ll", dia, context);
        Jitter jit;
        printf("Wuff?");
        Jitter::ModuleHandle h = jit.addModule(std::move(M));
        printf("KNUFF!\n");

        printf("Kuchen! 0x%p\n", jit.findSymbol("main").getAddress());

        system("PAUSE");
        return 0;
}

The Code runs without a fail, but when the programm tries to resolve "main" the address is 0. The strange thing: the printf "FLUSH :0\n" and "PLUSH :0\n" are never called, so did the code never compiled? What I'm doing wrong?

Kind regards
Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev




_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JIT doens't resolve address

Daniel Marjamäki via cfe-dev
Hello Lang,

the Code for my jit_main is a normal printf-HelloWorld program. I opended the file with notepad and found the following:

; Function Attrs: noinline norecurse uwtable
define i32 @main() #0 {
entry:
  %retval = alloca i32, align 4
  store i32 0, i32* %retval, align 4
  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([35 x i8], [35 x i8]* @"\01??_C@_0CD@CMJEAMCD@Planschbeckilein?5sagt?5Halloilein@", i32 0, i32 0))
  %call1 = call i32 @system(i8* getelementptr inbounds ([6 x i8], [6 x i8]* @"\01??_C@_05DIAHPDGL@PAUSE?$AA@", i32 0, i32 0))
  ret i32 0
}


So... there is a main... Anyway! The code for the main is this one:
#include <stdio.h>
#include <stdlib.h>

int main()
{
        printf("Planschbeckilein sagt Halloilein!\n");
        system("PAUSE");

        return 0;
}


Kind regards
Björn



From:        Lang Hames <[hidden email]>
To:        [hidden email]
Cc:        Clang Dev <[hidden email]>
Date:        20.04.2017 22:54
Subject:        Re: [cfe-dev] JIT doens't resolve address




Hi Bjoern,

Sorry - I just noticed that the address for your "main" function has come back as '0'. In this case the answer is even simpler: The JIT never found the function at all, and so definitely didn't have any external symbols to resolve, so it never called the resolver.

A failure to find main has three likely causes:

(1) jit_main.ll does not define main,
(2) jit_main.ll defines main with private/internal linkage (the JIT can't see private/internal symbols in general)
(3) jit_main.ll defines main with external linkage, but a system mangling is applied (e.g. on MacOSX 'main' is mangled to '_main'). The mangler code in your findSymbol function *should* correct for this, but this may fail if the default data layout for your TargetMachine varies from the Module's DataLayout.

Can you share the contents of your jit_main.ll Module?

Cheers,
Lang.
   

On Thu, Apr 20, 2017 at 1:35 PM, Lang Hames <lhames@...> wrote:
HI Bjoern,

There are two kinds of symbol lookup in the JIT:

(1) You want to find a symbol defined JIT'd code. This is what "findSymbol" on the JIT class does.

(2) The JIT wants to fix up a module that contains references to symbols defined elsewhere (either in your program, or in other JIT'd code). This is the SymbolResolver's job.

So assuming your main function in main.ll is trivial, e.g.

int main() {
  return 0;
}

then your findSymbol call will return the address of the JIT'd main without ever needing to look anything up in the resolver.

If, on the other hand, your main function contains an external reference, e.g.

int main() {
  printf("Hello, World!"); // Reference to externally defined function printf.
  return 0;
}

Then you'll receive a call back on your resolver looking for the address of printf.

Hope this helps!

Cheers,
Lang.

On Thu, Apr 20, 2017 at 6:43 AM, via cfe-dev <[hidden email]> wrote:
Hello LLVM-World,

I was following the "Building a JIT in LLVM"-Tutorial and tried to load a normal main. My code is the following:
class Jitter
{

private
:
  std::
unique_ptr<TargetMachine> TM;
 
const DataLayout DL;
 
ObjectLinkingLayer<> ObjectLayer;
 
IRCompileLayer<decltype(ObjectLayer)> CompileLayer;

public
:
 
typedef decltype(CompileLayer)::ModuleSetHandleT ModuleHandle;

  Jitter() : TM(
EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
      CompileLayer(ObjectLayer,
SimpleCompiler(*TM))
  {printf(
"!");
        llvm::sys::
DynamicLibrary::LoadLibraryPermanently(nullptr);
  }


 
TargetMachine &getTargetMachine() { return *TM; }

 
ModuleHandle addModule(std::unique_ptr<Module> &&M) {
 
// Build our symbol resolver:
 
// Lambda 1: Look back into the JIT itself to find symbols that are part of
 
//           the same "logical dylib".
 
// Lambda 2: Search for external symbols in the host process.
 
auto Resolver = createLambdaResolver(
      [&](
const std::string &Name)
      {

                printf(
"FLUSH :0\n");

       
if (auto Sym = CompileLayer.findSymbol(Name, false))
         
return Sym;
       
return JITSymbol(nullptr);
      },

      [](
const std::string &S)
          {

                  printf(
"PLUSH :0\n");

       
if (auto SymAddr =
             
RTDyldMemoryManager::getSymbolAddressInProcess(S))
         
return JITSymbol(SymAddr, JITSymbolFlags::Exported);
       
return JITSymbol(nullptr);
      });


 
// Build a singleton module set to hold our module.
  std::
vector<std::unique_ptr<Module>> Ms;
  Ms.push_back(std::move(
M));

 
// Add the set to the JIT with the resolver we created above and a newly
 
// created SectionMemoryManager.
 
return CompileLayer.addModuleSet(std::move(Ms),
                                   make_unique<
SectionMemoryManager>(),
                                   std::move(Resolver));

}


JITSymbol
findSymbol(const std::string Name) {
  std::
string MangledName;
 
raw_string_ostream MangledNameStream(MangledName);
 
Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
  printf(
"Tzearch for: %s\n\n", MangledNameStream.str());
 
return CompileLayer.findSymbol(MangledNameStream.str(), false);
}


void
removeModule(ModuleHandle H) {
  CompileLayer.removeModuleSet(
H);
}


};



And calling from main with:
int main()
{

        llvm::InitializeNativeTarget();

        llvm::InitializeNativeTargetAsmPrinter();

        llvm::InitializeNativeTargetAsmParser();


        llvm::
LLVMContext context;
        llvm::
SMDiagnostic dia;

        std::
unique_ptr<llvm::Module> M = llvm::parseIRFile("./jit_main.ll", dia, context);
       
Jitter jit;
        printf(
"Wuff?");
       
Jitter::ModuleHandle h = jit.addModule(std::move(M));
        printf(
"KNUFF!\n");

        printf(
"Kuchen! 0x%p\n", jit.findSymbol("main").getAddress());

        system(
"PAUSE");
       
return 0;
}


The Code runs without a fail, but when the programm tries to resolve "main" the address is 0. The strange thing: the printf "
FLUSH :0\n" and "PLUSH :0\n" are never called, so did the code never compiled? What I'm doing wrong?

Kind regards

Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list

[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev





Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JIT doens't resolve address

Daniel Marjamäki via cfe-dev
Hmm, I tried using your code (copy+pasted and then mass-dumped headers to fix includes), and it seems to work fine for me:


Do you get different results when using your LLVM? If not, can you identify the differences between your code and what I constructed from your snippets?

I noticed "PAUSE", are you on Windows? I'm not sure what the functionality/status of the JIT on Windows is, perhaps someone else can comment on that.

~Will

On Fri, Apr 21, 2017 at 1:03 AM, via cfe-dev <[hidden email]> wrote:
Hello Lang,

the Code for my jit_main is a normal printf-HelloWorld program. I opended the file with notepad and found the following:

; Function Attrs: noinline norecurse uwtable
define i32 @main() #0 {
entry:
  %retval = alloca i32, align 4
  store i32 0, i32* %retval, align 4
  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([35 x i8], [35 x i8]* @"\01??_C@_0CD@CMJEAMCD@Planschbeckilein?5sagt?5Halloilein@", i32 0, i32 0))
  %call1 = call i32 @system(i8* getelementptr inbounds ([6 x i8], [6 x i8]* @"\01??_C@_05DIAHPDGL@PAUSE?$AA@", i32 0, i32 0))
  ret i32 0
}


So... there is a main... Anyway! The code for the main is this one:
#include <stdio.h>
#include <stdlib.h>

int main()
{
        printf("Planschbeckilein sagt Halloilein!\n");
        system("PAUSE");

        return 0;
}


Kind regards
Björn



From:        Lang Hames <[hidden email]>
To:        [hidden email]
Cc:        Clang Dev <[hidden email]>
Date:        20.04.2017 22:54
Subject:        Re: [cfe-dev] JIT doens't resolve address




Hi Bjoern,

Sorry - I just noticed that the address for your "main" function has come back as '0'. In this case the answer is even simpler: The JIT never found the function at all, and so definitely didn't have any external symbols to resolve, so it never called the resolver.

A failure to find main has three likely causes:

(1) jit_main.ll does not define main,
(2) jit_main.ll defines main with private/internal linkage (the JIT can't see private/internal symbols in general)
(3) jit_main.ll defines main with external linkage, but a system mangling is applied (e.g. on MacOSX 'main' is mangled to '_main'). The mangler code in your findSymbol function *should* correct for this, but this may fail if the default data layout for your TargetMachine varies from the Module's DataLayout.

Can you share the contents of your jit_main.ll Module?

Cheers,
Lang.
   

On Thu, Apr 20, 2017 at 1:35 PM, Lang Hames <[hidden email]> wrote:
HI Bjoern,

There are two kinds of symbol lookup in the JIT:

(1) You want to find a symbol defined JIT'd code. This is what "findSymbol" on the JIT class does.

(2) The JIT wants to fix up a module that contains references to symbols defined elsewhere (either in your program, or in other JIT'd code). This is the SymbolResolver's job.

So assuming your main function in main.ll is trivial, e.g.

int main() {
  return 0;
}

then your findSymbol call will return the address of the JIT'd main without ever needing to look anything up in the resolver.

If, on the other hand, your main function contains an external reference, e.g.

int main() {
  printf("Hello, World!"); // Reference to externally defined function printf.
  return 0;
}

Then you'll receive a call back on your resolver looking for the address of printf.

Hope this helps!

Cheers,
Lang.

On Thu, Apr 20, 2017 at 6:43 AM, via cfe-dev <[hidden email]> wrote:
Hello LLVM-World,

I was following the "Building a JIT in LLVM"-Tutorial and tried to load a normal main. My code is the following:
class Jitter
{

private
:
  std::
unique_ptr<TargetMachine> TM;
 
const DataLayout DL;
 
ObjectLinkingLayer<> ObjectLayer;
 
IRCompileLayer<decltype(ObjectLayer)> CompileLayer;

public
:
 
typedef decltype(CompileLayer)::ModuleSetHandleT ModuleHandle;

  Jitter() : TM(
EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
      CompileLayer(ObjectLayer,
SimpleCompiler(*TM))
  {printf(
"!");
        llvm::sys::
DynamicLibrary::LoadLibraryPermanently(nullptr);
  }


 
TargetMachine &getTargetMachine() { return *TM; }

 
ModuleHandle addModule(std::unique_ptr<Module> &&M) {
 
// Build our symbol resolver:
 
// Lambda 1: Look back into the JIT itself to find symbols that are part of
 
//           the same "logical dylib".
 
// Lambda 2: Search for external symbols in the host process.
 
auto Resolver = createLambdaResolver(
      [&](
const std::string &Name)
      {

                printf(
"FLUSH :0\n");

       
if (auto Sym = CompileLayer.findSymbol(Name, false))
         
return Sym;
       
return JITSymbol(nullptr);
      },

      [](
const std::string &S)
          {

                  printf(
"PLUSH :0\n");

       
if (auto SymAddr =
             
RTDyldMemoryManager::getSymbolAddressInProcess(S))
         
return JITSymbol(SymAddr, JITSymbolFlags::Exported);
       
return JITSymbol(nullptr);
      });


 
// Build a singleton module set to hold our module.
  std::
vector<std::unique_ptr<Module>> Ms;
  Ms.push_back(std::move(
M));

 
// Add the set to the JIT with the resolver we created above and a newly
 
// created SectionMemoryManager.
 
return CompileLayer.addModuleSet(std::move(Ms),
                                   make_unique<
SectionMemoryManager>(),
                                   std::move(Resolver));

}


JITSymbol
findSymbol(const std::string Name) {
  std::
string MangledName;
 
raw_string_ostream MangledNameStream(MangledName);
 
Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
  printf(
"Tzearch for: %s\n\n", MangledNameStream.str());
 
return CompileLayer.findSymbol(MangledNameStream.str(), false);
}


void
removeModule(ModuleHandle H) {
  CompileLayer.removeModuleSet(
H);
}


};



And calling from main with:
int main()
{

        llvm::InitializeNativeTarget();

        llvm::InitializeNativeTargetAsmPrinter();

        llvm::InitializeNativeTargetAsmParser();


        llvm::
LLVMContext context;
        llvm::
SMDiagnostic dia;

        std::
unique_ptr<llvm::Module> M = llvm::parseIRFile("./jit_main.ll", dia, context);
       
Jitter jit;
        printf(
"Wuff?");
       
Jitter::ModuleHandle h = jit.addModule(std::move(M));
        printf(
"KNUFF!\n");

        printf(
"Kuchen! 0x%p\n", jit.findSymbol("main").getAddress());

        system(
"PAUSE");
       
return 0;
}


The Code runs without a fail, but when the programm tries to resolve "main" the address is 0. The strange thing: the printf "
FLUSH :0\n" and "PLUSH :0\n" are never called, so did the code never compiled? What I'm doing wrong?

Kind regards

Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list

[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev





Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev



_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JIT doens't resolve address - Resolve obj-Addresses?

Daniel Marjamäki via cfe-dev
Hello Mr. Dietz,

I tried using your code and now it works! I'm happy and surprised! I will now compare the codes and try to spot the main difference! Thanks for your help!
Do you know if there is a way to directly load .obj-Files and resolve the references with own addresses? Like when somebody used printf in the generated obj-File, but I replace the address with my own printf?

Kind regards
Björn



From:        Will Dietz <[hidden email]>
To:        [hidden email]
Cc:        Lang Hames <[hidden email]>, Clang Dev <[hidden email]>
Date:        21.04.2017 21:50
Subject:        Re: [cfe-dev] JIT doens't resolve address
Sent by:        [hidden email]




Hmm, I tried using your code (copy+pasted and then mass-dumped headers to fix includes), and it seems to work fine for me:

https://gist.github.com/dtzWill/df84b64a73001532e3fcfe73a2cffbb9#file-test-log

Do you get different results when using your LLVM? If not, can you identify the differences between your code and what I constructed from your snippets?

I noticed "PAUSE", are you on Windows? I'm not sure what the functionality/status of the JIT on Windows is, perhaps someone else can comment on that.

~Will

On Fri, Apr 21, 2017 at 1:03 AM, via cfe-dev <[hidden email]> wrote:
Hello Lang,

the Code for my jit_main is a normal printf-HelloWorld program. I opended the file with notepad and found the following:

; Function Attrs: noinline norecurse uwtable
define i32 @main() #0 {

entry:

  %retval = alloca i32, align 4

  store i32 0, i32* %retval, align 4

  %call = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([35 x i8], [35 x i8]* @"\01??_C@_0CD@CMJEAMCD@Planschbeckilein?5sagt?5Halloilein@", i32 0, i32 0))

  %call1 = call i32 @system(i8* getelementptr inbounds ([6 x i8], [6 x i8]* @"\01??_C@_05DIAHPDGL@PAUSE?$AA@", i32 0, i32 0))

  ret i32 0

}



So... there is a main... Anyway! The code for the main is this one:
#include <stdio.h>
#include
<stdlib.h>

int
main()
{

        printf(
"Planschbeckilein sagt Halloilein!\n");
        system(
"PAUSE");

       
return 0;
}



Kind regards

Björn




From:        
Lang Hames <lhames@...>
To:        
bjoern.gaier@...
Cc:        
Clang Dev <[hidden email]>
Date:        
20.04.2017 22:54
Subject:        
Re: [cfe-dev] JIT doens't resolve address





Hi Bjoern,

Sorry - I just noticed that the address for your "main" function has come back as '0'. In this case the answer is even simpler: The JIT never found the function at all, and so definitely didn't have any external symbols to resolve, so it never called the resolver.

A failure to find main has three likely causes:

(1) jit_main.ll does not define main,
(2) jit_main.ll defines main with private/internal linkage (the JIT can't see private/internal symbols in general)
(3) jit_main.ll defines main with external linkage, but a system mangling is applied (e.g. on MacOSX 'main' is mangled to '_main'). The mangler code in your findSymbol function *should* correct for this, but this may fail if the default data layout for your TargetMachine varies from the Module's DataLayout.

Can you share the contents of your jit_main.ll Module?

Cheers,
Lang.
   

On Thu, Apr 20, 2017 at 1:35 PM, Lang Hames <
lhames@...> wrote:
HI Bjoern,

There are two kinds of symbol lookup in the JIT:

(1) You want to find a symbol defined JIT'd code. This is what "findSymbol" on the JIT class does.

(2) The JIT wants to fix up a module that contains references to symbols defined elsewhere (either in your program, or in other JIT'd code). This is the SymbolResolver's job.

So assuming your main function in main.ll is trivial, e.g.

int main() {

  return 0;

}


then your findSymbol call will return the address of the JIT'd main without ever needing to look anything up in the resolver.

If, on the other hand, your main function contains an external reference, e.g.

int main() {

  printf("Hello, World!"); // Reference to externally defined function printf.

  return 0;

}


Then you'll receive a call back on your resolver looking for the address of printf.

Hope this helps!

Cheers,
Lang.

On Thu, Apr 20, 2017 at 6:43 AM, via cfe-dev <
[hidden email]> wrote:
Hello LLVM-World,


I was following the "Building a JIT in LLVM"-Tutorial and tried to load a normal main. My code is the following:
class Jitter
{

private
:
  std::
unique_ptr<TargetMachine> TM;
 
const DataLayout DL;
 
ObjectLinkingLayer<> ObjectLayer;
 
IRCompileLayer<decltype(ObjectLayer)> CompileLayer;

public
:
 
typedef decltype(CompileLayer)::ModuleSetHandleT ModuleHandle;

  Jitter() : TM(
EngineBuilder().selectTarget()), DL(TM->createDataLayout()),
      CompileLayer(ObjectLayer,
SimpleCompiler(*TM))
  {printf(
"!");
        llvm::sys::
DynamicLibrary::LoadLibraryPermanently(nullptr);
  }


 
TargetMachine &getTargetMachine() { return *TM; }

 
ModuleHandle addModule(std::unique_ptr<Module> &&M) {
 
// Build our symbol resolver:
 
// Lambda 1: Look back into the JIT itself to find symbols that are part of
 
//           the same "logical dylib".
 
// Lambda 2: Search for external symbols in the host process.
 
auto Resolver = createLambdaResolver(
      [&](
const std::string &Name)
      {

                printf(
"FLUSH :0\n");

       
if (auto Sym = CompileLayer.findSymbol(Name, false))
         
return Sym;
       
return JITSymbol(nullptr);
      },

      [](
const std::string &S)
          {

                  printf(
"PLUSH :0\n");

       
if (auto SymAddr =
             
RTDyldMemoryManager::getSymbolAddressInProcess(S))
         
return JITSymbol(SymAddr, JITSymbolFlags::Exported);
       
return JITSymbol(nullptr);
      });


 
// Build a singleton module set to hold our module.
  std::
vector<std::unique_ptr<Module>> Ms;
  Ms.push_back(std::move(
M));

 
// Add the set to the JIT with the resolver we created above and a newly
 
// created SectionMemoryManager.
 
return CompileLayer.addModuleSet(std::move(Ms),
                                   make_unique<
SectionMemoryManager>(),
                                   std::move(Resolver));

}


JITSymbol
findSymbol(const std::string Name) {
  std::
string MangledName;
 
raw_string_ostream MangledNameStream(MangledName);
 
Mangler::getNameWithPrefix(MangledNameStream, Name, DL);
  printf(
"Tzearch for: %s\n\n", MangledNameStream.str());
 
return CompileLayer.findSymbol(MangledNameStream.str(), false);
}


void
removeModule(ModuleHandle H) {
  CompileLayer.removeModuleSet(
H);
}


};




And calling from main with:
int main()
{

        llvm::InitializeNativeTarget();

        llvm::InitializeNativeTargetAsmPrinter();

        llvm::InitializeNativeTargetAsmParser();


        llvm::
LLVMContext context;
        llvm::
SMDiagnostic dia;

        std::
unique_ptr<llvm::Module> M = llvm::parseIRFile("./jit_main.ll", dia, context);
       
Jitter jit;
        printf(
"Wuff?");
       
Jitter::ModuleHandle h = jit.addModule(std::move(M));
        printf(
"KNUFF!\n");

        printf(
"Kuchen! 0x%p\n", jit.findSymbol("main").getAddress());

        system(
"PAUSE");
       
return 0;
}



The Code runs without a fail, but when the programm tries to resolve "main" the address is 0. The strange thing: the printf "
FLUSH :0\n" and "PLUSH :0\n" are never called, so did the code never compiled? What I'm doing wrong?

Kind regards

Björn
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list

[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev





Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.


_______________________________________________
cfe-dev mailing list

[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev




Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789
Geschäftsführer: Hiroshi Kawamura, Dr Hiroshi Nakamura, Markus Bode, Heiko Lampert, Takashi Nagano, Takeshi Fukushima.

_______________________________________________
cfe-dev mailing list
[hidden email]
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Loading...