[bug#39309,WIP] gnu: add stack.

I tried using this patch and it still fails. Could I be missing something?

Hi John,

John Soo <jsoo1@asu.edu> writes:

> I tried using this patch and it still fails. Could I be missing
> something?

I’m not sure, but I got the shadowing to work.  In fact, you don’t need
shadowing at all.  You can just pass “--with-hsc2hs=/gnu/store/...” as a
configure flag:

    (arguments
     `(#:configure-flags
       (let ((hsc2hs (assoc-ref %build-inputs "ghc-hsc2hs")))
         (list (string-append "--with-hsc2hs=" hsc2hs "/bin/hsc2hs")))))

Unfortunately, it doesn’t help.  Looking at the source code, “hsc2hs”
*is* calling GCC with a response file.  AIUI, GCC is supposed to
continue to use response files for all of its subprograms.  Hence,
everything should work.  I’m stumped.  :/

I’ll take another look with a fresh mind sometime, and see if I can make
any progress.


-- Tim

Hello again,

Timothy Sample <samplet@ngyro.com> writes:

> [...]
>
> I’ll take another look with a fresh mind sometime, and see if I can make
> any progress.

I don’t have a solution, but I have a bit more info.  Cabal generates
extremely long argument lists for GCC.  For whatever reason, it repeats
the same flags many times (maybe once for each Haskell dependency).
When building Stack, it passes Gawk as an include path 164 times.  In
fact, there are only a handful of distinct include paths, each passed
164 times.  The linker flags are similar but much worse.

This should be fine since everything is using a response file, right?
Nope.  Like the bug you linked discusses, GCC writes all of its
arguments into an environment variable, the length of which is limited
by “MAX_ARG_STRLEN” (which is 128K).  It would be nice to fix GCC here,
and it’s almost a very small fix, but there is minor complication.  The
normal response file handling in GCC expects “argc” and “argv”
parameters, but the environment variable doesn’t come with an “argc”.
Rather, it has to be parsed (in the same way the shell would do it).
It’s still doable, but a it would be a larger change.  Not to mention
that patching GCC is even less fun than patching GHC.

There was some work to deduplicate the flags that Cabal generates:
<https://github.com/haskell/cabal/pull/5356/>.  This would actually fix
the build, but the call to “hsc2hs” doesn’t do the same deduplication
(see “ppHsc2hs” in “Cabal/Distribution/Simple/PreProcess.hs”).

At this point, it might be worth joining the Nix folks on that issue,
and suggesting a patch to Cabal to call “hsc2hs” with deduplicated
flags.  It will help that the other deduplication code was written by
the same person who filed the bug you linked to earlier.  Once the patch
lands, we can apply it to our Cabal and then Stack will build.  Yay!


-- Tim

Hi Tim,

> At this point, it might be worth joining the Nix folks on that issue,
> and suggesting a patch to Cabal to call “hsc2hs” with deduplicated
> flags.  It will help that the other deduplication code was written by
> the same person who filed the bug you linked to earlier.  Once the patch
> lands, we can apply it to our Cabal and then Stack will build.  Yay!

Sorry I'm a bit confused by all this. I guess if we can rely on a
patch into cabal that would be the least effort, right?
Is there already a patch to deduplicate flags passed to hsc2hs?  Do we
need to comment anywhere or express support?

- John

Hi John,

John Soo <jsoo1@asu.edu> writes:

>> At this point, it might be worth joining the Nix folks on that issue,
>> and suggesting a patch to Cabal to call “hsc2hs” with deduplicated
>> flags.  It will help that the other deduplication code was written by
>> the same person who filed the bug you linked to earlier.  Once the patch
>> lands, we can apply it to our Cabal and then Stack will build.  Yay!
>
> Sorry I'm a bit confused by all this. I guess if we can rely on a
> patch into cabal that would be the least effort, right?

Exactly.  Also, sorry if I’m being a little unclear.  I’ve been kind of
thinking out loud with these messages, as you’ll see below.  ;)

> Is there already a patch to deduplicate flags passed to hsc2hs?  Do we
> need to comment anywhere or express support?

I couldn’t find a patch.  Expressing support was definitely what I was
suggesting.  I managed to find a note saying what I just said:

    https://github.com/NixOS/nixpkgs/issues/49206#issuecomment-470324743

I don’t think anything has become of it since then, though.

There is one other thing we could do, but I’m still thinking it through.
The reason we hit the limit is because of the way we use the
“extra-lib-dirs” and “extra-include-dirs” flags when configuring.  Every
time we pass Cabal a list of directories via these flags, Cabal notes
the directories in the package DB, and uses them when calling GCC.  It
doesn’t just use the directories specified when configuring a given
package.  It uses all the directories specified for all of that
package’s dependencies, too.

Right now, we pass in every “lib” and “include” directory from every
input, including the “standard-packages” from the GNU build system.
This means that Gawk’s “lib” directory (for example) is included every
time we configure any package.  In turn, when GHC calls GCC as part of
compiling a Haskell package, Gawk gets included on the command line once
for every node on that package’s dependency tree.  Packages with a large
dependency tree hit the 128K limit imposed by the kernel Linux.

What’s funny is that not a single Haskell package even uses shared
libraries from Gawk!

The other thing that’s funny is that most Haskell packages build just
fine without “extra-lib-dirs” and “extra-include-dirs”.  Even those that
call out to C code work, because Guix sets “LIBRARY_PATH” for us.
Unfortunately, packages that depend on packages that call out to C code
fail.  This is because Guix doesn’t set “LIBRARY_PATH” for the
transitive input, but GHC still wants to make use of the shared library.
This can be solved by adding back the “extra-lib-dirs” and
“extra-include-dirs” flags as-needed.

With this approach, I was able to compile Stack (the tests failed,
though).

The part that I’m still thinking through is how to make this whole
system work.  I would really like something that works automatically,
but I don’t see a way to do it.  This means we would have to add the
flags manually to the handful of packages that need them.  The part that
I really don’t like is that we don’t know a package needs the flags
until a package that depends on it fails to build.  Having everything
one step removed like that is not ideal.  On the other hand, there are
probably less than a dozen Haskell packages that need flags – it’s not
that common to mix Haskell and C code.  I will make a patch that does
this and see what the damages are.

This feels like a step in the right direction, because we end up with a
cleaner package graph as a result.  The added work and delayed feedback
will be tricky, but it should only come up infrequently.  What do you
think?


-- Tim