mbox series

[bug#49946,00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

Message ID 87h7fztt60.fsf@gmx.com
Headers show
Series Tree-sitter, node-gyp addon support and emacs-tree-sitter | expand

Message

Pierre Langlois Aug. 8, 2021, 10:29 p.m. UTC 
Hi Guix!

Here's a patch series that adds support for the Tree-sitter library,
command-line tool and its support in emacs:
https://tree-sitter.github.io/tree-sitter

The series in split into multiple parts:

1. Add the tree-sitter package, which is both a command-line tool for
   generating parsers and a runtime library.  The runtime is written in
   C but the tooling in Rust.

  * gnu: rust-chunked-transfer: Update to 1.4.0.
  * gnu: Add rust-fdlimit.
  * gnu: Add rust-html-escape.
  * gnu: Add rust-smallbitvec.
  * gnu: rust-tiny-http: Update to 0.8.
  * gnu: rust-spin: Update to 0.7.
  * gnu: Add tree-sitter.

2. Add support for native addons to the node build-system.

  * gnu: node: Patch /usr/bin/env in node-gyp.
  * guix: node-build-system: Support compiling addons with node-gyp.

  I've not touched the node build system before so it'd be great to get
  some review here!

  The idea is that when a package has a `binding.gyp' file then there's
  native code that needs to be compiled against node.  This uses
  node-gyp, which is a build system written in python.
  Normally, there shouldn't be anything to do, the package.json file
  should invoke `node-gyp` tp compile C code.  However, `node-gyp' needs
  node headers to compile against, packaged as a tarball, which it tries
  to download.  Instead, we can run a `node-gyp --tarball <> configure'
  step to manually provide the tarball, which we can package separately
  for any given node version.

  I could have added this into the packages themselves, but I thought it
  made more sense to be a part of the build system, let me know what you
  think!

3. Packaging grammars for multiple languages.

  Tree-sitter grammars are Nodejs packages, they consist of a grammar.js
  file which the tree-sitter command-line program uses to generate C
  code for the parser/scanner.  Using npm as a packaging mechanism for
  this seems overkill, but it's true some grammars depend on each other
  (cpp on c, typescript and javascript).

  * gnu: Add node-nan.
  * gnu: Add tree-sitter-c.
  * gnu: Add tree-sitter-cpp.
  * gnu: Add tree-sitter-css.
  * gnu: Add tree-sitter-go.
  * gnu: Add tree-sitter-html.
  * gnu: Add tree-sitter-java.
  * gnu: Add tree-sitter-javascript.
  * gnu: Add tree-sitter-json.
  * gnu: Add tree-sitter-julia.
  * gnu: Add tree-sitter-php.
  * gnu: Add tree-sitter-python.
  * gnu: Add tree-sitter-rust.
  * gnu: Add tree-sitter-typescript.

4. Finally, emacs support!

  emacs-tree-sitter can be used for syntax highlighting, replacing
  font-lock.  The tree-sitter runtime library is built as a Rust crate
  and exposed to emacs using a module thanks to the rust-emacs crate.

  Then emacs-tree-sitter-core provides just the core APIs as exposed by
  the rust module, then emacs-tree-sitter integrates it with emacs'
  syntax highlighting (and probably other things).

  Finally, we bundle together the various language grammars into a
  single emacs-tree-sitter-langs package so that they can get loaded
  along with their respective major modes.  Note that there are grammars
  left to support which I've not yet managed to package, but what we
  have already is a good start I think :-). So far I've tested C++,
  Rust, Python and JavaScript and JSON.

  * gnu: Add rust-bindgen@0.56.
  * gnu: Add rust-tree-sitter.
  * gnu: rust-emacs-module: Update to 0.16.
  * gnu: rust-emacs-macros: Update to 0.17.
  * gnu: rust-emacs: Update to 0.17.
  * gnu: Add emacs-tree-sitter-core.
  * gnu: Add emacs-tree-sitter.
  * gnu: Add emacs-tree-sitter-langs.


Let me know what you think! This rebuilds all node packages, but there
aren't that many so hopefully that'll be ok on master.

Thanks,
Pierre

Pierre Langlois (31):
  gnu: rust-chunked-transfer: Update to 1.4.0.
  gnu: Add rust-fdlimit.
  gnu: Add rust-html-escape.
  gnu: Add rust-smallbitvec.
  gnu: rust-tiny-http: Update to 0.8.
  gnu: rust-spin: Update to 0.7.
  gnu: Add tree-sitter.
  gnu: node: Patch /usr/bin/env in node-gyp.
  guix: node-build-system: Support compiling addons with node-gyp.
  gnu: Add node-nan.
  gnu: Add tree-sitter-c.
  gnu: Add tree-sitter-cpp.
  gnu: Add tree-sitter-css.
  gnu: Add tree-sitter-go.
  gnu: Add tree-sitter-html.
  gnu: Add tree-sitter-java.
  gnu: Add tree-sitter-javascript.
  gnu: Add tree-sitter-json.
  gnu: Add tree-sitter-julia.
  gnu: Add tree-sitter-php.
  gnu: Add tree-sitter-python.
  gnu: Add tree-sitter-rust.
  gnu: Add tree-sitter-typescript.
  gnu: Add rust-bindgen@0.56.
  gnu: Add rust-tree-sitter.
  gnu: rust-emacs-module: Update to 0.16.
  gnu: rust-emacs-macros: Update to 0.17.
  gnu: rust-emacs: Update to 0.17.
  gnu: Add emacs-tree-sitter-core.
  gnu: Add emacs-tree-sitter.
  gnu: Add emacs-tree-sitter-langs.

 gnu/local.mk                     |   1 +
 gnu/packages/crates-io.scm       | 290 +++++++++++--
 gnu/packages/node-xyz.scm        |  30 ++
 gnu/packages/node.scm            |  29 +-
 gnu/packages/tree-sitter.scm     | 716 +++++++++++++++++++++++++++++++
 guix/build-system/node.scm       |  16 +
 guix/build/node-build-system.scm |  15 +
 7 files changed, 1064 insertions(+), 33 deletions(-)
 create mode 100644 gnu/packages/tree-sitter.scm

Comments

Mekeor Melire May 14, 2022, 10:36 p.m. UTC | #1 
Hello,

what's the current state of this set of patches?

Pierre, do you happen to have a channel or so where I could try out the
patch-series?
Pierre Langlois May 15, 2022, 12:14 p.m. UTC | #2 
Hi,

Mekeor Melire <mekeor@posteo.de> writes:

> Hello,
>
> what's the current state of this set of patches?

I'm still working on it, sorry I've not had too much time for it these
days.

>
> Pierre, do you happen to have a channel or so where I could try out the
> patch-series?

I don't have a channel, however I just pushed the series rebased on top
of master on gitlab here: https://gitlab.com/planglois/guix/-/commits/emacs-tree-sitter

So if you'd like to test, you could use this instead of applying
patches:

  guix pull --url=https://gitlab.com/planglois/guix --branch=emacs-tree-sitter

I suppose I could move to a channel, but that's a little bit more work,
and I'd delete it immediately after we have something upstream. 

Let me know how testing goes!

Pierre
Luis Henrique Gomes Higino Jan. 25, 2023, 10:04 p.m. UTC | #3 
Hi guix,

I was taking a look at this patch series again and wondering if it 
could
be modified to support the new built in treesit module coming in
Emacs 29 instead of emacs-tree-sitter, as the latter recommends 
using
the native integration going forward [1]. Maybe we should leave 
this to when
Emacs 29 is released, but I think working on this in advance could
result on it being delivered earlier, which would be great. What 
do you think Pierre?

[1]: 
https://github.com/emacs-tree-sitter/elisp-tree-sitter/commit/d3eab879e9b0ccc9582113d81a93ad06f3c6bfb1

PS: I'm willing to help

Regards,