docs/build_system

Build System

There are many ways to build YARP, which means the build system is a bit more complicated than usual.

Requirements

  • It must work to build YARP for all 6 uses-cases below.
  • It must be possible to build YARP without needing ruby/rake/etc. Because once YARP is the single parser in TruffleRuby, JRuby or CRuby there won’t be another Ruby parser around to parse such Ruby code. Most/every Ruby implementations want to avoid depending on another Ruby during the build process as that is very brittle.
  • It is desirable to compile YARP with the same or very similar compiler flags for all use-cases (e.g. optimization level, warning flags, etc). Otherwise, there is the risk YARP does not work correctly with those different compiler flags.

The main solution for the second point seems a Makefile, otherwise many of the usages would have to duplicate the logic to build YARP.

General Design

  1. Templates are generated by templates/template.rb
  2. The Makefile compiles both librubyparser.a and librubyparser.{so,dylib,dll} from the src/**/*.c and include/**/*.h files
  3. The Rakefile :compile task ensures the above prerequisites are done, then calls make, and uses Rake::ExtensionTask to compile the C extension (using its extconf.rb), which uses librubyparser.a

This way there is minimal duplication, and each layer builds on the previous one and has its own responsibilities.

The static library exports no symbols, to avoid any conflict.
The shared library exports some symbols, and this is fine since there should only be one librubyparser shared library
loaded per process (i.e., at most one version of the yarp gem loaded in a process, only the gem uses the shared library).

The various ways to build YARP

Building from ruby/yarp repository with bundle exec rake

rake calls make and then uses Rake::ExtensionTask to compile the C extension (see above).

Building the yarp gem by gem install/bundle install

The gem contains the pre-generated templates.
When installing the gem, extconf.rb is used and that:

  • runs make build/librubyparser.a
  • compiles the C extension with mkmf

When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead of the last step,
there is Ruby code using FFI which uses librubyparser.{so,dylib,dll}
to implement the same methods as the C extension, but using serialization instead of many native calls/accesses
(JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).

Building the yarp gem from git, e.g. gem 'yarp', github: 'ruby/yarp'

The same as above, except the extconf.rb additionally runs first:

  • templates/template.rb to generate the templates

Because of course those files are not part of the git repository.

Building YARP as part of CRuby

This script imports YARP sources in CRuby.

The script generates the templates when importing.

YARP’s Makefile is not used at all in CRuby. Instead, CRuby’s Makefile is used.

Building YARP as part of TruffleRuby

This script imports YARP sources in TruffleRuby.
The script generates the templates when importing.

Then when mx build builds TruffleRuby and the yarp mx project inside, it runs make.

Then the yarp bindings mx project is built, which contains the bindings
and links to librubyparser.a (to avoid exporting symbols, so no conflict when installing the yarp gem).

Building YARP as part of JRuby

TODO, probably similar to TruffleRuby.