| Message ID | cover.1678049134.git.felix.lechner@lease-up.com |
|---|---|
| Headers | show |
| Series | Adding Gocryptfs (feature branch) | expand |
On Sun, Mar 05, 2023 at 12:52:11PM -0800, Felix Lechner via Guix-patches via wrote: > After a discussion on #guix, someone asked me to request a feature branch > here, as well as an associated jobset for ci.guix.gnu.org. That was me! I've gone ahead and started the process of adding the jobset on ci.guix.gnu.org. It should appear in the next day or two. > I think we are hoping to use this patch series as a proof-of-concept for the > idea of feature branches. Thanks! Pushed to Savannah as 'wip-go-updates'. The 'wip-' prefix stands for "work in progress", and indicates that history may be rewritten on this branch. Concretely, we are able to rebase the branch. I also applied updates for Go 1.19 and 1.20. https://git.savannah.gnu.org/cgit/guix.git/log/?h=wip-go-updates
On Mon, Mar 06, 2023 at 11:42:22AM -0500, Leo Famulari wrote: > That was me! I've gone ahead and started the process of adding the > jobset on ci.guix.gnu.org. It should appear in the next day or two. The jobset is live: https://ci.guix.gnu.org/jobset/go-team Please follow up with a summary of the results. If any code changes are needed, send them as part of the full patch series, versioned with the '--reroll-count=N' Git option, and I'll push them on your behalf. Please don't send updates to individual patches.
Hi Leo, On Tue, Mar 7, 2023 at 3:06 PM Leo Famulari <leo@famulari.name> wrote: > > The jobset is live: Thanks so much! I browsed around a bit (but am new to Cuirass) and most of the failures seem to come from this one-hour timeout in pandoc. [1] Is that something I can address with an updated patch series? Kind regards Felix Lechner [1] https://ci.guix.gnu.org/build/478772/log/raw
On Tue, Mar 07, 2023 at 04:02:42PM -0800, Felix Lechner wrote: > Thanks so much! I browsed around a bit (but am new to Cuirass) and > most of the failures seem to come from this one-hour timeout in > pandoc. [1] I see. Looks like it's actually GHC 9.2.5 that's failing, but pandoc depends on GHC so all the rest of the dependency graph will fail too. > Is that something I can address with an updated patch series? For a while, setting the timeout and max-silent-time in a package definition didn't work here, because Cuirass didn't respect those options. I don't know if that's changed. Can you ask on IRC or guix-devel about this?
Hi Leo, On Wed, Mar 8, 2023 at 7:10 AM Leo Famulari <leo@famulari.name> wrote: > > Can you ask on IRC or guix-devel about this? Thank you for taking the initiative on guix-devel! [1] I did mention the Cuirass question on IRC as you had asked, but no one seemed to remember. Then I addressed the Haskell failure by responding to a recent thread about that upload. [2] By your reply, I know that you saw my message. As a newbie, I am not sure how to proceed. Please let me know if there is anything I can do to help. Kind regards Felix Lechner [1] https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00094.html [2] https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00089.html
On Thu, Mar 09, 2023 at 04:21:20PM -0800, Felix Lechner wrote: > I did mention the Cuirass question on IRC as you had asked, but no one > seemed to remember. Then I addressed the Haskell failure by responding > to a recent thread about that upload. [2] By your reply, I know that > you saw my message. Thanks for working on those leads! In my reply to your message regarding Haskell, I CC-ed Mathieu Othacehe, who is relatively active developing Cuirass and assists with operations on ci.guix.gnu.org. Let's see if he replies. I do see that on ci.guix.gnu.org, we are running Cuirass 1.1.0-13.1341725, which should respect the max-silent-time property. According to NEWS, this was added in Cuirass 1.0.0. One might consider trying to debug a local installation of Cuirass, to see if it actually respects this property. Imagine a test package that sleeps for 10 seconds in a build phase but has a max-silent-time of 5. In the meantime, I would check that gocryptfs is working satisfactorily based on the branch. I was surprised that Cuirass tried to build GHC and all these R packages as a result of these patches. Do you know if these packages (e.g. GHC) depend on Go somehow?
On Thu, Mar 09, 2023 at 08:54:28PM -0500, Leo Famulari wrote: > I was surprised that Cuirass tried to build GHC and all these R packages > as a result of these patches. Do you know if these packages (e.g. GHC) > depend on Go somehow? In the meantime, I rebased the branch and pushed again. Maybe the building of all these non-Go packages was spurious (this can happen with Cuirass), and we'll see a more clear result this time.
Hi Leo, On Sat, Mar 11, 2023 at 8:17 AM Leo Famulari <leo@famulari.name> wrote: > > I rebased the branch and pushed again. Maybe the > building of all these non-Go packages was spurious (this can happen with > Cuirass), and we'll see a more clear result this time. Thanks for doing that! The results look much better, but I believe some of the failures are still unrelated to our feature branch. I cloned the wip-go-updates branch and successfully built pigx-rnaseq on my own equipment even though Cuirass showed failing tests. [1] A partial log is attached. Kind regards Felix Lechner [1] https://ci.guix.gnu.org/build/533474/details
On Thu, Mar 09, 2023 at 08:54:28PM -0500, Leo Famulari wrote: > In the meantime, I would check that gocryptfs is working satisfactorily > based on the branch. Were you able to test this?
Hi Leo, On Mon, Mar 13, 2023 at 10:53 AM Leo Famulari <leo@famulari.name> wrote: > > Were you able to test this? Yes, I did. It works great! Kind regards Felix Lechner
Hi Leo, On Sat, Mar 11, 2023 at 8:17 AM Leo Famulari <leo@famulari.name> wrote: > > In the meantime, I rebased the branch and pushed again. I am not sure how to read the Cuirass derivations, but the file missing for powerpc64le-linux [1] called libgcc_s.so.1 is present on x86_64: gcc-10.3.0-lib gcc-12.1.0-lib gcc-12.2.0-lib gcc-mesboot-4.9.4 gfortran-10.3.0-lib libgccjit-10.3.0 The results were by courtesy of juix.org: Kind regards Felix [1] https://ci.guix.gnu.org/build/547772/log/raw
On Mon, Mar 13, 2023 at 07:05:25PM -0700, Felix Lechner wrote: > I am not sure how to read the Cuirass derivations, but the file > missing for powerpc64le-linux [1] called libgcc_s.so.1 is present on > x86_64: > > gcc-10.3.0-lib > gcc-12.1.0-lib > gcc-12.2.0-lib > gcc-mesboot-4.9.4 > gfortran-10.3.0-lib > libgccjit-10.3.0 Are you saying that something is broken on powerpc64le-linux? Is it a regression from this branch? Or a pre-existing problem?
Hi Leo, On Wed, Mar 15, 2023 at 6:21 PM Leo Famulari <leo@famulari.name> wrote: > > On Mon, Mar 13, 2023 at 07:05:25PM -0700, Felix Lechner wrote: > > I am not sure how to read the Cuirass derivations, but the file > > missing for powerpc64le-linux [1] called libgcc_s.so.1 is present on > > x86_64: > > > > gcc-10.3.0-lib > > gcc-12.1.0-lib > > gcc-12.2.0-lib > > gcc-mesboot-4.9.4 > > gfortran-10.3.0-lib > > libgccjit-10.3.0 > > Are you saying that something is broken on powerpc64le-linux? Is it a > regression from this branch? Or a pre-existing problem? I do not know the answers to your questions. With my message, I merely tried to address what seemed like a remaining issue in this jobset: https://ci.guix.gnu.org/jobset/go-team The failed build #547772 I references was part of the latest evaluation #283979, but I am not sure how to read those reports. Most significantly, I do not know if the reports are cumulative—meaning that fewer errors later signify that earlier errors have been resolved. Kind regards Felix Lechner
On Thu, Mar 16, 2023, at 15:44, Felix Lechner wrote: > Hi Leo, > > On Wed, Mar 15, 2023 at 6:21 PM Leo Famulari <leo@famulari.name> wrote: >> >> On Mon, Mar 13, 2023 at 07:05:25PM -0700, Felix Lechner wrote: >> > I am not sure how to read the Cuirass derivations, but the file >> > missing for powerpc64le-linux [1] called libgcc_s.so.1 is present on >> > x86_64: >> > >> > gcc-10.3.0-lib >> > gcc-12.1.0-lib >> > gcc-12.2.0-lib >> > gcc-mesboot-4.9.4 >> > gfortran-10.3.0-lib >> > libgccjit-10.3.0 >> >> Are you saying that something is broken on powerpc64le-linux? Is it a >> regression from this branch? Or a pre-existing problem? > > I do not know the answers to your questions. With my message, I merely > tried to address what seemed like a remaining issue in this jobset: > > https://ci.guix.gnu.org/jobset/go-team > > The failed build #547772 I references was part of the latest > evaluation #283979, but I am not sure how to read those reports. > > Most significantly, I do not know if the reports are > cumulative—meaning that fewer errors later signify that earlier errors > have been resolved. Okay. Then your next steps should be identifying Guix developers who have a history of working on the PowerPC port and asking them to take a look. This kind of outreach to the Guix developer community is a critical part of landing big changes like this one. If you can't find anyone to respond, then it's safe to say we can ignore this problem.
Hi Leo, On Thu, Mar 16, 2023 at 5:00 PM Leo Famulari <leo@famulari.name> wrote: > > your next steps should be identifying Guix developers who have a history > of working on the PowerPC port and asking them to take a look. Based on this blog post [1] I wrote to Efraim on IRC. I mentioned this bug and the connection to the powerpc64le port. Then I asked the group about the whereabouts of the two blog authors, namely Chris Marusich and Léo Le Bouter. Unfortunately I did not receive a response to either message. if a side note is permitted, I am not sure how a Golang update can affect the availability for a low-level GCC library for a new architecture. Could the error be unrelated? Kind regards, Felix Lechner [1] https://guix.gnu.org/en/blog/2021/new-supported-platform-powerpc64le-linux/
On Fri, Mar 17, 2023 at 12:02:55PM -0700, Felix Lechner via Guix-patches via wrote: > Hi Leo, > > On Thu, Mar 16, 2023 at 5:00 PM Leo Famulari <leo@famulari.name> wrote: > > > > your next steps should be identifying Guix developers who have a history > > of working on the PowerPC port and asking them to take a look. > > Based on this blog post [1] I wrote to Efraim on IRC. I mentioned this > bug and the connection to the powerpc64le port. Then I asked the group > about the whereabouts of the two blog authors, namely Chris Marusich > and Léo Le Bouter. Unfortunately I did not receive a response to > either message. To the best of my knowledge both Léo and Chris have both left the project. As it currently stands on master, go-1.19 doesn't build for ppc64le. I mentioned on IRC that it looks like the 'patch-gcc:lib phase in go-1.17 but for ppc64le, but after trying a few different combinations it looks to me more like the test failures on aarch64 (which I haven't looked at in forever). Looking at it I'm convinced that we are missing patching something on our end, and that these tests are probably supposed to pass. My suggestion is to skip the tests on go-1.17+ on ppc64le and adjust the note that all the listed architectures should have their test suite reviewed to figure out what we're missing. There's something about the internal linking on non-x86 architectures that needs investigating.
On Sun, Mar 19, 2023 at 10:51:51AM +0200, Efraim Flashner wrote: > As it currently stands on master, go-1.19 doesn't build for ppc64le. I > mentioned on IRC that it looks like the 'patch-gcc:lib phase in go-1.17 > but for ppc64le, but after trying a few different combinations it looks > to me more like the test failures on aarch64 (which I haven't looked at > in forever). > > Looking at it I'm convinced that we are missing patching something on > our end, and that these tests are probably supposed to pass. > > My suggestion is to skip the tests on go-1.17+ on ppc64le and adjust the > note that all the listed architectures should have their test suite > reviewed to figure out what we're missing. There's something about the > internal linking on non-x86 architectures that needs investigating. Okay, thanks for taking a look. Since this failure exists on master, I don't think that fixing it is in scope for this branch, although it certainly should be fixed. Or we disable the support on ppc64le for now.
On Sun, Mar 19, 2023 at 01:17:24PM -0400, Leo Famulari wrote: > Okay, thanks for taking a look. Since this failure exists on master, I > don't think that fixing it is in scope for this branch, although it > certainly should be fixed. Or we disable the support on ppc64le for now. Felix, do you see any other outstanding issues with the branch? Or do you think it's ready for master?
Hi Leo, On Sun, Mar 19, 2023 at 10:22 AM Leo Famulari <leo@famulari.name> wrote: > > Felix, do you see any other outstanding issues with the branch? Or do > you think it's ready for master? I do not know how to deduce from the Cuirass output that the failures from earlier evaluations were cured. The numbers appear additive to me. I would prefer to see about three thousand green packages and two in red. Kind regards Felix
On Sun, Mar 19, 2023 at 10:25:31AM -0700, Felix Lechner wrote: > I do not know how to deduce from the Cuirass output that the failures > from earlier evaluations were cured. The numbers appear additive to > me. I would prefer to see about three thousand green packages and two > in red. Unfortunately, Cuirass can't provide that kind of information. You could use `guix weather` to check if substitutes are available based on the patches.
On Sun, Mar 19, 2023 at 04:38:32PM -0400, Leo Famulari wrote: > Unfortunately, Cuirass can't provide that kind of information. You could > use `guix weather` to check if substitutes are available based on the > patches. I've attached a manifest that returns all packages using go-build-system, as well as Go itself. It can be used with `guix weather`. ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2023 Leo Famulari <leo@famulari.name> ;;; ;;; This file is part of GNU Guix. ;;; ;;; GNU Guix is free software; you can redistribute it and/or modify it ;;; under the terms of the GNU General Public License as published by ;;; the Free Software Foundation; either version 3 of the License, or (at ;;; your option) any later version. ;;; ;;; GNU Guix is distributed in the hope that it will be useful, but ;;; WITHOUT ANY WARRANTY; without even the implied warranty of ;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;;; GNU General Public License for more details. ;;; ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>. (use-modules (guix packages) (guix profiles) (gnu packages) (guix build-system go)) (manifest (map package->manifest-entry (fold-packages (lambda (package result) (if (or (eq? (package-build-system package) go-build-system) (equal? (package-name package) "go")) (cons package result) result)) '())))
Hi Leo, On Sun, Mar 19, 2023 at 1:46 PM Leo Famulari <leo@famulari.name> wrote: > > I've attached a manifest that returns all packages using > go-build-system, as well as Go itself. It can be used with `guix > weather`. Thanks, that's awesome! Does the CI infrastructure publish build output from feature branches? Kind regards Felix
On Sun, Mar 19, 2023 at 01:48:32PM -0700, Felix Lechner wrote: > Thanks, that's awesome! Does the CI infrastructure publish build > output from feature branches? What sort of output are you looking for?
Hi Leo, On Sun, Mar 19, 2023 at 1:51 PM Leo Famulari <leo@famulari.name> wrote: > > What sort of output are you looking for? I suppose the answer is yes. It was not obvious to me that the CI infrastructure would acknowledge the presence of substitutes for feature branches, but it makes sense. Kind regards Felix
On Sun, Mar 19, 2023 at 01:54:00PM -0700, Felix Lechner wrote: > I suppose the answer is yes. It was not obvious to me that the CI > infrastructure would acknowledge the presence of substitutes for > feature branches, but it makes sense. Yeah. Basically, everything that's built on CI goes into the same store and is served from there. Roughly.
Hi Leo, On Sun, Mar 19, 2023 at 1:46 PM Leo Famulari <leo@famulari.name> wrote: > > I've attached a manifest that returns all packages using > go-build-system, as well as Go itself. It can be used with `guix > weather`. For a recent checkout of Guix from master, I saw: computing 585 package derivations for x86_64-linux... looking for 599 store items on https://ci.guix.gnu.org... https://ci.guix.gnu.org ☀ 98.0% substitutes available (587 out of 599) at least 1,893.3 MiB of nars (compressed) 4,188.9 MiB on disk (uncompressed) 0.086 seconds per request (51.4 seconds in total) 11.7 requests per second 0.0% (0 out of 12) of the missing items are queued at least 1,000 queued builds i686-linux: 31 (3.1%) aarch64-linux: 228 (22.8%) x86_64-linux: 737 (73.7%) powerpc64le-linux: 4 (.4%) build rate: 1324.99 builds per hour i686-linux: 1049.02 builds per hour powerpc64le-linux: 86.39 builds per hour x86_64-linux: 239.82 builds per hour looking for 599 store items on https://bordeaux.guix.gnu.org... https://bordeaux.guix.gnu.org ☀ 99.3% substitutes available (595 out of 599) at least 1,090.3 MiB of nars (compressed) 4,199.4 MiB on disk (uncompressed) 0.093 seconds per request (55.8 seconds in total) 10.7 requests per second (continuous integration information unavailable) while a recent pull of wip-go-updates showed this: computing 594 package derivations for x86_64-linux... looking for 608 store items on https://ci.guix.gnu.org... https://ci.guix.gnu.org ☀ 96.9% substitutes available (589 out of 608) at least 1,886.4 MiB of nars (compressed) 4,184.5 MiB on disk (uncompressed) 0.087 seconds per request (14.7 seconds in total) 11.5 requests per second 0.0% (0 out of 19) of the missing items are queued at least 1,000 queued builds powerpc64le-linux: 60 (6.0%) i686-linux: 65 (6.5%) aarch64-linux: 225 (22.5%) x86_64-linux: 650 (65.0%) build rate: 1048.34 builds per hour i686-linux: 704.11 builds per hour powerpc64le-linux: 80.85 builds per hour x86_64-linux: 290.39 builds per hour looking for 608 store items on https://bordeaux.guix.gnu.org... https://bordeaux.guix.gnu.org ☀ 96.4% substitutes available (586 out of 608) at least 836.2 MiB of nars (compressed) 3,398.5 MiB on disk (uncompressed) 0.099 seconds per request (16.1 seconds in total) 10.1 requests per second (continuous integration information unavailable) That looks very respectable to me for such a comprehensive change, but I can ultimately not assess whether it's a good ratio. It looks like perhaps seven items are failing due to our changes (on x86_64, and out of about six hundred) but I do not know how important they are for average Guix users. Kind regards Felix
On Sun, Mar 19, 2023 at 02:31:18PM -0700, Felix Lechner wrote: > That looks very respectable to me for such a comprehensive change, but > I can ultimately not assess whether it's a good ratio. Agreed, I think it looks good. > It looks like perhaps seven items are failing due to our changes (on > x86_64, and out of about six hundred) but I do not know how important > they are for average Guix users. Perhaps, but the Cuirass web interface doesn't make it easy to compare and see which packages are failing as a result of these changes. And qa.guix.gnu.org doesn't yet build changes with this large of an impact. So, I've gone ahead and pushed as c4cca9cb5d3e93ef146acb930a95da9d2da6fb06
Hi Leo, On Tue, Mar 28, 2023 at 1:37 PM Leo Famulari <leo@famulari.name> wrote: > > So, I've gone ahead and pushed Yay, thank you! Kind regards Felix