diff mbox

[bug#49946,00/31] Tree-sitter, node-gyp addon support and emacs-tree-sitter

Message ID 87edqxri41.fsf@trop.in
State New
Headers show

Commit Message

Andrew Tropin Feb. 10, 2023, 9:21 a.m. UTC 
On 2021-08-08 23:29, Pierre Langlois wrote:

> 4. Finally, emacs support!
>
>   emacs-tree-sitter can be used for syntax highlighting, replacing
>   font-lock.  The tree-sitter runtime library is built as a Rust crate
>   and exposed to emacs using a module thanks to the rust-emacs crate.
>
>   Then emacs-tree-sitter-core provides just the core APIs as exposed by
>   the rust module, then emacs-tree-sitter integrates it with emacs'
>   syntax highlighting (and probably other things).
>
>   Finally, we bundle together the various language grammars into a
>   single emacs-tree-sitter-langs package so that they can get loaded
>   along with their respective major modes.  Note that there are grammars
>   left to support which I've not yet managed to package, but what we
>   have already is a good start I think :-). So far I've tested C++,
>   Rust, Python and JavaScript and JSON.
>
>   * gnu: Add rust-bindgen@0.56.
>   * gnu: Add rust-tree-sitter.
>   * gnu: rust-emacs-module: Update to 0.16.
>   * gnu: rust-emacs-macros: Update to 0.17.
>   * gnu: rust-emacs: Update to 0.17.
>   * gnu: Add emacs-tree-sitter-core.
>   * gnu: Add emacs-tree-sitter.
>   * gnu: Add emacs-tree-sitter-langs.

Hi Pierre!

Implemented a loading of tree-sitter grammars in Emacs using guix
native-search-paths and built-in treesit package.  I did it for emacs
package, but it will work only with Emacs 29 and greater (right now it's
emacs-next and similiar packages), until emacs package updated to
version 29 it won't take any effect on it except providing one more
environment variable in guix profile.

I tested it with emacs-next-pgtk, the patch is attached below, please
let me know what do you think!

Comments

Pierre Langlois Feb. 10, 2023, 4:22 p.m. UTC | #1 
Hi,

Andrew Tropin <andrew@trop.in> writes:

> [[PGP Signed Part:Undecided]]
> On 2021-08-08 23:29, Pierre Langlois wrote:
>
>> 4. Finally, emacs support!
>>
>>   emacs-tree-sitter can be used for syntax highlighting, replacing
>>   font-lock.  The tree-sitter runtime library is built as a Rust crate
>>   and exposed to emacs using a module thanks to the rust-emacs crate.
>>
>>   Then emacs-tree-sitter-core provides just the core APIs as exposed by
>>   the rust module, then emacs-tree-sitter integrates it with emacs'
>>   syntax highlighting (and probably other things).
>>
>>   Finally, we bundle together the various language grammars into a
>>   single emacs-tree-sitter-langs package so that they can get loaded
>>   along with their respective major modes.  Note that there are grammars
>>   left to support which I've not yet managed to package, but what we
>>   have already is a good start I think :-). So far I've tested C++,
>>   Rust, Python and JavaScript and JSON.
>>
>>   * gnu: Add rust-bindgen@0.56.
>>   * gnu: Add rust-tree-sitter.
>>   * gnu: rust-emacs-module: Update to 0.16.
>>   * gnu: rust-emacs-macros: Update to 0.17.
>>   * gnu: rust-emacs: Update to 0.17.
>>   * gnu: Add emacs-tree-sitter-core.
>>   * gnu: Add emacs-tree-sitter.
>>   * gnu: Add emacs-tree-sitter-langs.
>
> Hi Pierre!
>
> Implemented a loading of tree-sitter grammars in Emacs using guix
> native-search-paths and built-in treesit package.  I did it for emacs
> package, but it will work only with Emacs 29 and greater (right now it's
> emacs-next and similiar packages), until emacs package updated to
> version 29 it won't take any effect on it except providing one more
> environment variable in guix profile.
>
> I tested it with emacs-next-pgtk, the patch is attached below, please
> let me know what do you think!

This is cool, thanks! I'm not familiar with how upstream emacs loads
grammars so I can't comment too much on the details. That being said, I
think it'd be good to get it to work with emacs 28 as well, I've been
using it for the past 6 months and it's working nicely. I'm happy to
take care of that part of the work though and let others focus on
emacs-next.

We have the following packages for emacs 28 support:

>>   * gnu: Add rust-bindgen@0.56.
>>   * gnu: Add rust-tree-sitter.
>>   * gnu: rust-emacs-module: Update to 0.16.
>>   * gnu: rust-emacs-macros: Update to 0.17.
>>   * gnu: rust-emacs: Update to 0.17.
>>   * gnu: Add emacs-tree-sitter-core.
>>   * gnu: Add emacs-tree-sitter.
>>   * gnu: Add emacs-tree-sitter-langs.

Most of those would eventually go away I think, except probably
emacs-tree-sitter-langs, which is really useful to provide basic
tree-sitter syntax highlighint support to many language modes that may
not need to be changed themselves.

For instance, there is a PR to eventually move over to using the core
emacs 29 treesit module: https://github.com/emacs-tree-sitter/tree-sitter-langs/pull/157

>
> From b5ecd4e3734e9dd0bc76ebe95cab9c43aa85a3fe Mon Sep 17 00:00:00 2001
> From: Andrew Tropin <andrew@trop.in>
> Date: Fri, 10 Feb 2023 12:32:12 +0400
> Subject: [PATCH] gnu: emacs: Add TREE_SITTER_GRAMMAR_PATH support.
>
> gnu/packages/emacs.scm (emacs)[native-search-paths]: Add a search-path for
> tree-sitter grammars.
> gnu/packages/aux-files/emacs/guix-emacs.el: Add directories from
> TREE_SITTER_GRAMMAR_PATH to treesit-extra-load-path.
> ---
>  gnu/packages/aux-files/emacs/guix-emacs.el |  7 +++++++
>  gnu/packages/emacs.scm                     | 10 +++++++++-
>  2 files changed, 16 insertions(+), 1 deletion(-)
>
> diff --git a/gnu/packages/aux-files/emacs/guix-emacs.el b/gnu/packages/aux-files/emacs/guix-emacs.el
> index 56dbcb8d67..708093267d 100644
> --- a/gnu/packages/aux-files/emacs/guix-emacs.el
> +++ b/gnu/packages/aux-files/emacs/guix-emacs.el
> @@ -76,6 +76,13 @@ The files in the list do not have extensions (.el, .elc)."
>                     (when (file-directory-p pkg-dir)
>                       (package-load-descriptor pkg-dir)))))))))))
>  
> +;; If emacs built with tree-sitter, read the value of the environment variable
> +;; to make tree-sitter grammars available in emacs out-of-the-box.
> +(with-eval-after-load 'treesit
> +  (when-let ((grammar-path (getenv "TREE_SITTER_GRAMMAR_PATH")))
> +    (mapcar (lambda (x) (add-to-list 'treesit-extra-load-path x))
> +            (split-string grammar-path ":"))))
> +
>  (provide 'guix-emacs)
>  
>  ;;; guix-emacs.el ends here
> diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
> index 4ce41deb88..a3cc9d2e22 100644
> --- a/gnu/packages/emacs.scm
> +++ b/gnu/packages/emacs.scm
> @@ -367,7 +367,15 @@ (define* (emacs-byte-compile-directory dir)
>              (files '("lib/emacs/native-site-lisp")))
>             (search-path-specification
>              (variable "INFOPATH")
> -            (files '("share/info")))))
> +            (files '("share/info")))
> +           ;; tree-sitter support is not yet available in emacs 28, but this
> +           ;; search path won't harm and also will be benifitial for
> +           ;; emacs-next and other emacs-* packages, which have tree-sitter
> +           ;; support enabled.  Please, remove this comment, when emacs
> +           ;; package is updated to 29.
> +           (search-path-specification
> +            (variable "TREE_SITTER_GRAMMAR_PATH")
> +            (files '("lib/tree-sitter")))))
>  
>      (home-page "https://www.gnu.org/software/emacs/")
>      (synopsis "The extensible, customizable, self-documenting text editor")
> -- 
> 2.39.1

LGTM!


Thanks,
Pierre
Liliana Marie Prikler Feb. 10, 2023, 6:02 p.m. UTC | #2 
Am Freitag, dem 10.02.2023 um 13:21 +0400 schrieb Andrew Tropin:

> +           ;; tree-sitter support is not yet available in emacs 28,
> but this
> +           ;; search path won't harm and also will be benifitial for
> +           ;; emacs-next and other emacs-* packages, which have
> tree-sitter
> +           ;; support enabled.  Please, remove this comment, when
> emacs
> +           ;; package is updated to 29.
IIRC it's "beneficial".  Otherwise LGTM
Andrew Tropin Feb. 11, 2023, 8:12 a.m. UTC | #3 
On 2023-02-10 19:02, Liliana Marie Prikler wrote:

> Am Freitag, dem 10.02.2023 um 13:21 +0400 schrieb Andrew Tropin:
>
>> +           ;; tree-sitter support is not yet available in emacs 28,
>> but this
>> +           ;; search path won't harm and also will be benifitial for
>> +           ;; emacs-next and other emacs-* packages, which have
>> tree-sitter
>> +           ;; support enabled.  Please, remove this comment, when
>> emacs
>> +           ;; package is updated to 29.
> IIRC it's "beneficial".  Otherwise LGTM

Nice catch, thank you very much, pushed as
c77a3d4554175ee5320ccc713aa21b1ba3e6b8a3
diff mbox

Patch

From b5ecd4e3734e9dd0bc76ebe95cab9c43aa85a3fe Mon Sep 17 00:00:00 2001
From: Andrew Tropin <andrew@trop.in>
Date: Fri, 10 Feb 2023 12:32:12 +0400
Subject: [PATCH] gnu: emacs: Add TREE_SITTER_GRAMMAR_PATH support.

gnu/packages/emacs.scm (emacs)[native-search-paths]: Add a search-path for
tree-sitter grammars.
gnu/packages/aux-files/emacs/guix-emacs.el: Add directories from
TREE_SITTER_GRAMMAR_PATH to treesit-extra-load-path.
---
 gnu/packages/aux-files/emacs/guix-emacs.el |  7 +++++++
 gnu/packages/emacs.scm                     | 10 +++++++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/aux-files/emacs/guix-emacs.el b/gnu/packages/aux-files/emacs/guix-emacs.el
index 56dbcb8d67..708093267d 100644
--- a/gnu/packages/aux-files/emacs/guix-emacs.el
+++ b/gnu/packages/aux-files/emacs/guix-emacs.el
@@ -76,6 +76,13 @@  The files in the list do not have extensions (.el, .elc)."
                    (when (file-directory-p pkg-dir)
                      (package-load-descriptor pkg-dir)))))))))))
 
+;; If emacs built with tree-sitter, read the value of the environment variable
+;; to make tree-sitter grammars available in emacs out-of-the-box.
+(with-eval-after-load 'treesit
+  (when-let ((grammar-path (getenv "TREE_SITTER_GRAMMAR_PATH")))
+    (mapcar (lambda (x) (add-to-list 'treesit-extra-load-path x))
+            (split-string grammar-path ":"))))
+
 (provide 'guix-emacs)
 
 ;;; guix-emacs.el ends here
diff --git a/gnu/packages/emacs.scm b/gnu/packages/emacs.scm
index 4ce41deb88..a3cc9d2e22 100644
--- a/gnu/packages/emacs.scm
+++ b/gnu/packages/emacs.scm
@@ -367,7 +367,15 @@  (define* (emacs-byte-compile-directory dir)
             (files '("lib/emacs/native-site-lisp")))
            (search-path-specification
             (variable "INFOPATH")
-            (files '("share/info")))))
+            (files '("share/info")))
+           ;; tree-sitter support is not yet available in emacs 28, but this
+           ;; search path won't harm and also will be benifitial for
+           ;; emacs-next and other emacs-* packages, which have tree-sitter
+           ;; support enabled.  Please, remove this comment, when emacs
+           ;; package is updated to 29.
+           (search-path-specification
+            (variable "TREE_SITTER_GRAMMAR_PATH")
+            (files '("lib/tree-sitter")))))
 
     (home-page "https://www.gnu.org/software/emacs/")
     (synopsis "The extensible, customizable, self-documenting text editor")
-- 
2.39.1