mbox series

[bug#69637,mesa-updates,0/6] gnu: mesa: Update to 24.0.2.

Message ID cover.1709899566.git.aurtzy@gmail.com
Headers show
Series gnu: mesa: Update to 24.0.2. | expand

Message

aurtzy March 8, 2024, 12:32 p.m. UTC
Hello!

This series of patches updates Mesa to the latest stable version with the NVK
Vulkan driver for NVIDIA GPUs enabled for x86_64 systems.

Mesa 24.0.2 depends on a newer version of libdrm (>= 2.4.119), which is
pending in another issue (https://issues.guix.gnu.org/68845).

Some packages (rust-syn-2.0.39, rust-proc-macro2-1.0.70, rust-quote-1.0.33)
were added specifically to be used as sources that satisfy explicit versions
of dependencies required by Mesa (for NVK).  Would it be better to inline
origins in the change-subproject-sources phase?  The same question also
applies for the rust-unicode-ident-1 source.

Currently, the Rust dependencies are found in the build by substituting the
fallback https://* URLs in the relevant ./subprojects/*.wrap files with
file://*, then relying on the fallback to "download" from the store.  It
seemed a bit hacky doing it this way, but I'm not aware of a better solution.

A newer Meson version is also required for NVK (>= 1.3.0), which this patch
series adds a new variable for.

x86_64-linux and i686-linux have been tested to successfully build.

Cheers,

aurtzy

aurtzy (6):
  gnu: Add meson-1.3.
  gnu: Add rust-syn-2.0.39.
  gnu: Add rust-proc-macro2-1.0.70.
  gnu: Add rust-quote-1.0.33.
  gnu: Add rust-bindgen-cli-0.69.
  gnu: mesa: Update to 24.0.2.

 gnu/packages/build-tools.scm | 14 ++++++
 gnu/packages/crates-io.scm   | 95 ++++++++++++++++++++++++++++++++++++
 gnu/packages/gl.scm          | 68 +++++++++++++++++++-------
 3 files changed, 160 insertions(+), 17 deletions(-)


base-commit: 0b5cbca689b2f91949e191ca256ffc9bb48c1141

Comments

Efraim Flashner March 12, 2024, 8:11 a.m. UTC | #1
On Fri, Mar 08, 2024 at 07:32:39AM -0500, aurtzy wrote:
> Hello!
> 
> This series of patches updates Mesa to the latest stable version with the NVK
> Vulkan driver for NVIDIA GPUs enabled for x86_64 systems.
> 
> Mesa 24.0.2 depends on a newer version of libdrm (>= 2.4.119), which is
> pending in another issue (https://issues.guix.gnu.org/68845).
> 
> Some packages (rust-syn-2.0.39, rust-proc-macro2-1.0.70, rust-quote-1.0.33)
> were added specifically to be used as sources that satisfy explicit versions
> of dependencies required by Mesa (for NVK).  Would it be better to inline
> origins in the change-subproject-sources phase?  The same question also
> applies for the rust-unicode-ident-1 source.

Are there other architectures which have rust based drivers? x86_64
isn't the only architecture which has rust building on it.

> Currently, the Rust dependencies are found in the build by substituting the
> fallback https://* URLs in the relevant ./subprojects/*.wrap files with
> file://*, then relying on the fallback to "download" from the store.  It
> seemed a bit hacky doing it this way, but I'm not aware of a better solution.

The crates are also available in
%output/share/cargo/registry/name-version.crate, although I can't think
of a good way to address them by name without using find-files.

I would personally replace the versions requested by mesa with whatever
version we happen to have in guix so that we don't have to add special
versions just for mesa.

> A newer Meson version is also required for NVK (>= 1.3.0), which this patch
> series adds a new variable for.
> 
> x86_64-linux and i686-linux have been tested to successfully build.
> 
> Cheers,
> 
> aurtzy
> 
> aurtzy (6):
>   gnu: Add meson-1.3.
>   gnu: Add rust-syn-2.0.39.
>   gnu: Add rust-proc-macro2-1.0.70.
>   gnu: Add rust-quote-1.0.33.
>   gnu: Add rust-bindgen-cli-0.69.
>   gnu: mesa: Update to 24.0.2.
> 
>  gnu/packages/build-tools.scm | 14 ++++++
>  gnu/packages/crates-io.scm   | 95 ++++++++++++++++++++++++++++++++++++
>  gnu/packages/gl.scm          | 68 +++++++++++++++++++-------
>  3 files changed, 160 insertions(+), 17 deletions(-)
> 
> 
> base-commit: 0b5cbca689b2f91949e191ca256ffc9bb48c1141
> -- 
> 2.41.0
> 
> 
> 
>
aurtzy March 21, 2024, 1:52 a.m. UTC | #2
On 3/12/24 04:11, Efraim Flashner wrote:
> Are there other architectures which have rust based drivers? x86_64
> isn't the only architecture which has rust building on it.
There doesn't appear to be any official documentation stating 
architecture requirements for Mesa with NVK/Rust, however I have added 
few new inputs other than rust to get NVK working. I've only done 
extensive testing with x86_64 so I was unsure of potential issues with 
including this on other architectures (other than i686 not building rust).
> The crates are also available in
> %output/share/cargo/registry/name-version.crate, although I can't think
> of a good way to address them by name without using find-files.
>
> I would personally replace the versions requested by mesa with whatever
> version we happen to have in guix so that we don't have to add special
> versions just for mesa.

I have/had tried a few approaches to use the crates already available in 
Guix with no success so far. I've outlined the approaches below; still 
looking into solutions, but perhaps there's something I'm missing or 
haven't tried yet?

- Simply including crates as (native-)inputs does not make them 
discoverable by meson.

- Mesa uses these *.wrap files which specify the rust dependency 
versions, source URLs, and tar hashes. I currently get the build working 
by relying on meson to fall back to "downloading" from a patched source 
URL (pointing to store), although it still has to match the hash.

- I recently discovered a way to disable the hash requirement so I could 
use a different input version (i.e. one from Guix), but doing it causes 
"File src/lib.rs does not exist" errors. I'm still looking into this 
right now, as it seems promising.

- Old IRC logs point to projects like newsboat and librsvg which also 
mix cargo with with another build system, but these start with 
cargo-build-system with phases added/replaced from the second build 
system. Cargo.toml doesn't exist in Mesa either (which 
cargo-build-system seems to depend on), so experimenting with using 
cargo-build-system didn't yield much.

I wanted to look more into the third bullet before responding, but I 
felt it would be unfortunate to have this information rot while trying 
to make time for hacking - hopefully it's still useful.

Cheers,

aurtzy
John Kehayias March 21, 2024, 4:39 a.m. UTC | #3
Hi aurtzy and Efraim,

On Wed, Mar 20, 2024 at 09:52 PM, aurtzy wrote:

> On 3/12/24 04:11, Efraim Flashner wrote:
>
>  Are there other architectures which have rust based drivers? x86_64
> isn't the only architecture which has rust building on it.
>
> There doesn't appear to be any official documentation stating architecture requirements for Mesa with
> NVK/Rust, however I have added few new inputs other than rust to get NVK working. I've only done extensive
> testing with x86_64 so I was unsure of potential issues with including this on other architectures (other than
> i686 not building rust).
>

I guess this is the main question, if we will try to enable the NVK
driver for other architectures, if that is supported. We could also
leave it for the known working x86_64 to start. I would say we could
provide a mesa variant package for testing, but that might be
difficult as far as I know (e.g. trying to get Xorg to use a different
mesa package looked difficult from what I saw others try).

We could try just checking for where rust is available (is that just
supported-systems for the rust package, or do we have other logic?)
and building with NVK to see what fails... Though with how long it can
take us to build on other architectures, that might take a while to
find out and then correct.

>  The crates are also available in
> %output/share/cargo/registry/name-version.crate, although I can't think
> of a good way to address them by name without using find-files.
>
> I would personally replace the versions requested by mesa with whatever
> version we happen to have in guix so that we don't have to add special
> versions just for mesa.
>
> I have/had tried a few approaches to use the crates already available in Guix with no success so far. I've
> outlined the approaches below; still looking into solutions, but perhaps there's something I'm missing or
> haven't tried yet?
>
> - Simply including crates as (native-)inputs does not make them discoverable by meson.
>
> - Mesa uses these *.wrap files which specify the rust dependency versions, source URLs, and tar hashes. I
> currently get the build working by relying on meson to fall back to "downloading" from a patched source URL
> (pointing to store), although it still has to match the hash.
>
> - I recently discovered a way to disable the hash requirement so I could use a different input version (i.e. one
> from Guix), but doing it causes "File src/lib.rs does not exist" errors. I'm still looking into this right now, as it
> seems promising.
>
> - Old IRC logs point to projects like newsboat and librsvg which also mix cargo with with another build
> system, but these start with cargo-build-system with phases added/replaced from the second build system.
> Cargo.toml doesn't exist in Mesa either (which cargo-build-system seems to depend on), so experimenting
> with using cargo-build-system didn't yield much.
>
> I wanted to look more into the third bullet before responding, but I felt it would be unfortunate to have this
> information rot while trying to make time for hacking - hopefully it's still useful.
>
> Cheers,
>
> aurtzy

Thanks for this additional info, aurtzy, and your work on this!

Efraim, any thoughts on the rust related stuff based on these other
attempts? I'm not familiar enough with rust, rust packaging, or what
mesa is doing in the meson builds to comment right now.

I would like to get the build farm cranking on the updates I have
queued for mesa-updates (cairo, libdrm, mesa, vulkan). We could also
do just the version update of mesa to start, or just NVK on x86_64,
leaving future changes for the next round. I don't have a preference
myself, other than wanting to get this branch moving with these
updates.

Thoughts?

And thanks both of you again!
John
Efraim Flashner March 21, 2024, 10:58 a.m. UTC | #4
On Thu, Mar 21, 2024 at 04:39:29AM +0000, John Kehayias wrote:
> Hi aurtzy and Efraim,
> 
> On Wed, Mar 20, 2024 at 09:52 PM, aurtzy wrote:
> 
> > On 3/12/24 04:11, Efraim Flashner wrote:
> >
> >  Are there other architectures which have rust based drivers? x86_64
> > isn't the only architecture which has rust building on it.
> >
> > There doesn't appear to be any official documentation stating architecture requirements for Mesa with
> > NVK/Rust, however I have added few new inputs other than rust to get NVK working. I've only done extensive
> > testing with x86_64 so I was unsure of potential issues with including this on other architectures (other than
> > i686 not building rust).
> >
> 
> I guess this is the main question, if we will try to enable the NVK
> driver for other architectures, if that is supported. We could also
> leave it for the known working x86_64 to start. I would say we could
> provide a mesa variant package for testing, but that might be
> difficult as far as I know (e.g. trying to get Xorg to use a different
> mesa package looked difficult from what I saw others try).
> 
> We could try just checking for where rust is available (is that just
> supported-systems for the rust package, or do we have other logic?)
> and building with NVK to see what fails... Though with how long it can
> take us to build on other architectures, that might take a while to
> find out and then correct.

We have (supported-package? package) to check if a package is supported.
Then we can wrap the phase in #$@(if (assoc-ref inputs "rust") ...) and
it should all just work.

> >  The crates are also available in
> > %output/share/cargo/registry/name-version.crate, although I can't think
> > of a good way to address them by name without using find-files.
> >
> > I would personally replace the versions requested by mesa with whatever
> > version we happen to have in guix so that we don't have to add special
> > versions just for mesa.
> >
> > I have/had tried a few approaches to use the crates already available in Guix with no success so far. I've
> > outlined the approaches below; still looking into solutions, but perhaps there's something I'm missing or
> > haven't tried yet?
> >
> > - Simply including crates as (native-)inputs does not make them discoverable by meson.
> >
> > - Mesa uses these *.wrap files which specify the rust dependency versions, source URLs, and tar hashes. I
> > currently get the build working by relying on meson to fall back to "downloading" from a patched source URL
> > (pointing to store), although it still has to match the hash.
> >
> > - I recently discovered a way to disable the hash requirement so I could use a different input version (i.e. one
> > from Guix), but doing it causes "File src/lib.rs does not exist" errors. I'm still looking into this right now, as it
> > seems promising.
> >
> > - Old IRC logs point to projects like newsboat and librsvg which also mix cargo with with another build
> > system, but these start with cargo-build-system with phases added/replaced from the second build system.
> > Cargo.toml doesn't exist in Mesa either (which cargo-build-system seems to depend on), so experimenting
> > with using cargo-build-system didn't yield much.
> >
> > I wanted to look more into the third bullet before responding, but I felt it would be unfortunate to have this
> > information rot while trying to make time for hacking - hopefully it's still useful.

I also tried a couple of different options. The one that I most want
involved using with-output-to-file to rewrite the wrap file and
replacing all the fields. I borrowed the file-sha256 function from
guix/build/cargo-utils.scm to get the source_hash.  In the end I wasn't
able to get the gexp and un-gexp bits working to actually get the file
written.

When I kept a failed build I saw that the 'directory' field is the
directory into which meson writes the meson.build file, which is why
using a different version of the rust crate caused problems with
src/lib.rs not existing.  I suppose we could start from your patch and
then, after running substitute, extract the tarball into either a
hardcoded path (determined after manually reading the sources) or we can
extract the 'directory' field by reading the sources and then untar the
source there.

> > Cheers,
> >
> > aurtzy
> 
> Thanks for this additional info, aurtzy, and your work on this!
> 
> Efraim, any thoughts on the rust related stuff based on these other
> attempts? I'm not familiar enough with rust, rust packaging, or what
> mesa is doing in the meson builds to comment right now.
> 
> I would like to get the build farm cranking on the updates I have
> queued for mesa-updates (cairo, libdrm, mesa, vulkan). We could also
> do just the version update of mesa to start, or just NVK on x86_64,
> leaving future changes for the next round. I don't have a preference
> myself, other than wanting to get this branch moving with these
> updates.
> 
> Thoughts?

Looking at qa.guix.gnu.org I believe that gnome-team is going to merge
soon, and then the emacs-team is up next.

I would prefer to use the already packaged crates but we can go ahead
and use the ones from the patchset for now and work out swapping them
later.  As far as which architectures, I think I'd start with x86_64
only first.  And sprinkle a couple of TODOs around.

> And thanks both of you again!
> John
>