Analysis-based toolchain tools/job actions without output

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Analysis-based toolchain tools/job actions without output

Manas via cfe-dev
Greetings everyone,

I'm using a slightly modified version of Clang and LLVM to build
binaries that are effectively firmware for custom hardware -- code
that gets compiled down to flat executable code which will be dropped
into some flash memory on a device at a known, fixed address. These
files exit the compilation process from LLD as ELF executables before
being validated and converted to binary blobs via objcopy. The
validation process takes place using a custom tool, and all it does is
check to make sure all the final ELF sections are read-only. (There
are probably better ways of doing this, but that's certainly outside
of the scope of this email!)

The validation and extraction processes are accomplished using
individual build steps in a terrible custom build system (in the name
of OS compatibility despite the fact that we only use this on a single
platform) which can't be replaced any time soon. There is a TON of
code involved in these build steps using this horrible internal build
system and its continued use is equivalent to indentured servitude for
technical debt. However, we're stuck with it for the foreseeable
future.

In an effort to make this easier and to reduce the amount of garbage
code my team has to deal with in this mockery of a build system, I
would like to convert the ELF validation process into a custom job
action to Clang's compilation pipeline -- specifically, actions
defined by `clang::Driver::BuildActions()`
(https://github.com/llvm/llvm-project/blob/d12fa33d736d60d419f86b4ec5f3e77e602d4b1e/clang/lib/Driver/Driver.cpp#L3586).
I've done this before for tools such as `llvm-link` to provide
automatic bitcode linking, but that tool produces output which can be
fed back into the compilation pipeline for further actions such as
code generation.

The ELF validation tool produces no output -- it simply checks the
executable and exits with a 0 status if it determines everything is
good, or it prints relevant error messages and exits with a 1 status.
This is where I run into problems. In my attempts to make this tool a
step in Clang's compilation process, I've not found a way to tell
Clang to simply use the output from the preceding step/job action as
the output for this tool. As a bandage fix, I've modified the
validator tool to accept a `-o=<XYZ>` argument and to use that to
create a symlink of the input file. I can then feed that symlink back
to Clang. Clang is happy with this, but it's *not* a good solution.

I see other job actions that don't produce output (such as those for
handling `-fsyntax-only` and `-verify-pch`, but those also act as
terminators for the compilation process.

Is there a way to do what I'm trying to do within Clang without
halting the pipeline?

Respectfully,
Jonathan Smith (jvstech)
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Analysis-based toolchain tools/job actions without output

Manas via cfe-dev
Hi Jonathan,

Sorry for not responding earlier.

What you've described agrees with my limited experience here - the
compiler driver creates Jobs that correspond to various compilation
phases. And the output from one phase is fed into the next one. This is
very nicely illustrated with `-ccc-print-phases`. I've not tried it
myself, but suspect that diverging from this model is going to be rather
tricky.

Perhaps you could/should tweak the job that generates the binary? That
would be the backend job, right? You'd have to abandon the idea of using
a custom tool and implement this in LLVM. But given how powerful LLVM
is, I imagine that it wouldn't be too hard.

And if that's not an option, then I guess that probably you want a dummy
compiler driver phase that doesn't output anything. I don't have much
experience with the API surrounding phase control in the driver. You
could start by checking the most recent patches touching e.g.:
   * include/clang/Driver/Types.def
   * include/clang/Driver/Phases.h
   * lib/Driver/Phases.cpp

I hope that this helps at least a tiny bit!

-Andrzej


On 19/02/2021 10:32, Jonathan Smith via cfe-dev wrote:

> Greetings everyone,
>
> I'm using a slightly modified version of Clang and LLVM to build
> binaries that are effectively firmware for custom hardware -- code
> that gets compiled down to flat executable code which will be dropped
> into some flash memory on a device at a known, fixed address. These
> files exit the compilation process from LLD as ELF executables before
> being validated and converted to binary blobs via objcopy. The
> validation process takes place using a custom tool, and all it does is
> check to make sure all the final ELF sections are read-only. (There
> are probably better ways of doing this, but that's certainly outside
> of the scope of this email!)
>
> The validation and extraction processes are accomplished using
> individual build steps in a terrible custom build system (in the name
> of OS compatibility despite the fact that we only use this on a single
> platform) which can't be replaced any time soon. There is a TON of
> code involved in these build steps using this horrible internal build
> system and its continued use is equivalent to indentured servitude for
> technical debt. However, we're stuck with it for the foreseeable
> future.
>
> In an effort to make this easier and to reduce the amount of garbage
> code my team has to deal with in this mockery of a build system, I
> would like to convert the ELF validation process into a custom job
> action to Clang's compilation pipeline -- specifically, actions
> defined by `clang::Driver::BuildActions()`
> (https://github.com/llvm/llvm-project/blob/d12fa33d736d60d419f86b4ec5f3e77e602d4b1e/clang/lib/Driver/Driver.cpp#L3586).
> I've done this before for tools such as `llvm-link` to provide
> automatic bitcode linking, but that tool produces output which can be
> fed back into the compilation pipeline for further actions such as
> code generation.
>
> The ELF validation tool produces no output -- it simply checks the
> executable and exits with a 0 status if it determines everything is
> good, or it prints relevant error messages and exits with a 1 status.
> This is where I run into problems. In my attempts to make this tool a
> step in Clang's compilation process, I've not found a way to tell
> Clang to simply use the output from the preceding step/job action as
> the output for this tool. As a bandage fix, I've modified the
> validator tool to accept a `-o=<XYZ>` argument and to use that to
> create a symlink of the input file. I can then feed that symlink back
> to Clang. Clang is happy with this, but it's *not* a good solution.
>
> I see other job actions that don't produce output (such as those for
> handling `-fsyntax-only` and `-verify-pch`, but those also act as
> terminators for the compilation process.
>
> Is there a way to do what I'm trying to do within Clang without
> halting the pipeline?
>
> Respectfully,
> Jonathan Smith (jvstech)
> _______________________________________________
> cfe-dev mailing list
> [hidden email]
> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
Reply | Threaded
Open this post in threaded view
|

Re: Analysis-based toolchain tools/job actions without output

Manas via cfe-dev
Hey Andrzej!

Thanks for the insight!

I'd love to get the backend to handle this automatically, and that's
probably the best way of doing this. That's also *probably* how this
will eventually end up going. But for the time being, I'm just trying
not to step on my team's toes.

What I've ended up doing is modifying the tool to perform the check
and then call `objcopy` itself. So yeah, it appears the compilation
pipeline is designed specifically to keep forwarding output to input
or until reaching the end.

Thanks again for the confirmation/second opinion!

Respectfully,
Jonathan Smith (jvstech)

On Wed, Mar 3, 2021 at 1:32 PM Andrzej Warzynski
<[hidden email]> wrote:

>
> Hi Jonathan,
>
> Sorry for not responding earlier.
>
> What you've described agrees with my limited experience here - the
> compiler driver creates Jobs that correspond to various compilation
> phases. And the output from one phase is fed into the next one. This is
> very nicely illustrated with `-ccc-print-phases`. I've not tried it
> myself, but suspect that diverging from this model is going to be rather
> tricky.
>
> Perhaps you could/should tweak the job that generates the binary? That
> would be the backend job, right? You'd have to abandon the idea of using
> a custom tool and implement this in LLVM. But given how powerful LLVM
> is, I imagine that it wouldn't be too hard.
>
> And if that's not an option, then I guess that probably you want a dummy
> compiler driver phase that doesn't output anything. I don't have much
> experience with the API surrounding phase control in the driver. You
> could start by checking the most recent patches touching e.g.:
>    * include/clang/Driver/Types.def
>    * include/clang/Driver/Phases.h
>    * lib/Driver/Phases.cpp
>
> I hope that this helps at least a tiny bit!
>
> -Andrzej
>
>
> On 19/02/2021 10:32, Jonathan Smith via cfe-dev wrote:
> > Greetings everyone,
> >
> > I'm using a slightly modified version of Clang and LLVM to build
> > binaries that are effectively firmware for custom hardware -- code
> > that gets compiled down to flat executable code which will be dropped
> > into some flash memory on a device at a known, fixed address. These
> > files exit the compilation process from LLD as ELF executables before
> > being validated and converted to binary blobs via objcopy. The
> > validation process takes place using a custom tool, and all it does is
> > check to make sure all the final ELF sections are read-only. (There
> > are probably better ways of doing this, but that's certainly outside
> > of the scope of this email!)
> >
> > The validation and extraction processes are accomplished using
> > individual build steps in a terrible custom build system (in the name
> > of OS compatibility despite the fact that we only use this on a single
> > platform) which can't be replaced any time soon. There is a TON of
> > code involved in these build steps using this horrible internal build
> > system and its continued use is equivalent to indentured servitude for
> > technical debt. However, we're stuck with it for the foreseeable
> > future.
> >
> > In an effort to make this easier and to reduce the amount of garbage
> > code my team has to deal with in this mockery of a build system, I
> > would like to convert the ELF validation process into a custom job
> > action to Clang's compilation pipeline -- specifically, actions
> > defined by `clang::Driver::BuildActions()`
> > (https://github.com/llvm/llvm-project/blob/d12fa33d736d60d419f86b4ec5f3e77e602d4b1e/clang/lib/Driver/Driver.cpp#L3586).
> > I've done this before for tools such as `llvm-link` to provide
> > automatic bitcode linking, but that tool produces output which can be
> > fed back into the compilation pipeline for further actions such as
> > code generation.
> >
> > The ELF validation tool produces no output -- it simply checks the
> > executable and exits with a 0 status if it determines everything is
> > good, or it prints relevant error messages and exits with a 1 status.
> > This is where I run into problems. In my attempts to make this tool a
> > step in Clang's compilation process, I've not found a way to tell
> > Clang to simply use the output from the preceding step/job action as
> > the output for this tool. As a bandage fix, I've modified the
> > validator tool to accept a `-o=<XYZ>` argument and to use that to
> > create a symlink of the input file. I can then feed that symlink back
> > to Clang. Clang is happy with this, but it's *not* a good solution.
> >
> > I see other job actions that don't produce output (such as those for
> > handling `-fsyntax-only` and `-verify-pch`, but those also act as
> > terminators for the compilation process.
> >
> > Is there a way to do what I'm trying to do within Clang without
> > halting the pipeline?
> >
> > Respectfully,
> > Jonathan Smith (jvstech)
> > _______________________________________________
> > cfe-dev mailing list
> > [hidden email]
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> >
_______________________________________________
cfe-dev mailing list
[hidden email]
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev