Add per-file options to extra source files

[andreabedini]

Adds per-file options to the five extra-source fields

This is one of the last things the Cabal library needs before it can build GHC; Hadrian makes the same per-file split today. This patch has been developed for https://github.com/stable-haskell/ghc.

cabal-version: 3.20

library
  c-sources:   foo.c (-DFOO -O2) bar.c
  cmm-sources: Jumps_V32.cmm (-mavx2)

The options are applied only to the relative file.

Based on #12275 which needs to go in first.

Design

ExtraSourceKind tags each source with its kind, and addExtraSourceOpts maps that kind to the one GhcOptions field GHC consults for it:

field	GhcOptions field	GHC flag
c-sources	ghcOptCcOptions	-optc
cxx-sources	ghcOptCxxOptions	-optcxx
asm-sources	ghcOptAsmOptions	-opta
cmm-sources	ghcOptExtra	plain GHC options
js-sources	ghcOptJSppOptions	-optJSP

Only C-- options are plain GHC options, because GHC compiles C-- itself and there is no -optcmm. The other four are options for a tool GHC invokes, so they cannot share ghcOptExtra: ghc -DFOO -c foo.c defines a Haskell CPP macro instead of passing -DFOO to the C compiler.

Keeping the kind — rather than a string the user has already prefixed — leaves the spelling to Cabal at render time, which it has to be, because the prefix depends on the compiler version: -optcxx only exists since GHC 8.10 (below that, C++ options go through -optc), -optJSP only since 9.12 (below that they are dropped, with a warning).

Each extra source is already compiled by its own ghc -c run, so per-file options are just appended to that run's component-wide ones. They come last, so they win over cc-options and its siblings.

Interface change

The five fields hold ExtraSource (a path plus its options) instead of SymbolicPath Pkg File. extraSourceFromPath builds one with no options. Distribution.Simple.SetupHooks re-exports both, so a Hooks package that sets cSources needs no direct Cabal-syntax dependency.

Why the cabal-version gate is an error, and why not 3.18

Gated behind cabal-version: 3.20 per #9331. Worth flagging explicitly: an older Cabal does not reject this syntax, it misreads it. A path is any non-space token, so Cabal-syntax 3.14.1.0 reads

c-sources: foo.c (-DFOO -O2) bar.c

as four source files — foo.c, (-DFOO, -O2), bar.c — with no diagnostic, failing only later when it tries to compile (-DFOO. That rules out a warning, and it rules out the already-published 3.18 as the gate.

Tests

• Parser regression test over all five kinds at 3.20, with quoted options and the pretty-print round trip.
• Error test pinning the rejection below 3.20.
• QuickCheck round-trip property for ExtraSource, with a generator that produces options needing quotes.
• renderGhcOptions unit tests: -optJSP rendered at 9.12 and dropped below it; C-- options plain at every version.
• ExtraSources setup test with a C source that only compiles if its per-file -D arrives.

QA Notes

With cabal-version: 3.20:

• Build a package whose C source #errors unless a macro is defined, and define it per-file: c-sources: greet.c (-DGREETING=hi) other.c. It should build, and -DGREETING must not reach other.c.
• An option containing whitespace has to be quoted: c-sources: greet.c ("-DGREETING=\"hi there\""). Unquoted, it should be read as two options.
• The same file with cabal-version: 3.18 should fail with a parse error naming 3.20 — not a missing-file error about (-DGREETING=hi.
• cabal sdist on a package using per-file options should ship the sources and round-trip the options through the regenerated .cabal.
• With GHC < 9.12: js-sources: foo.js (-DBAR) should warn that the options are ignored and still build; jspp-options: -DBAR should now build rather than failing with an unrecognised-flag error from GHC.

Checklist

Template A: this PR modifies behaviour or interface.

• Patches conform to the coding conventions.
• Changes relevant to users have been recorded in the changelog (significance: significant for the feature).
• The documentation has been updated (doc/cabal-package-description-file.rst, doc/file-format-changelog.rst).
• Manual QA notes have been included.
• Tests have been added.

[Copilot]

Pull request overview

This PR extends Cabal’s .cabal file format and internal APIs to support per-file options on the five “extra source” fields (c-sources, cxx-sources, asm-sources, cmm-sources, js-sources) gated behind cabal-version: 3.20, enabling per-source compilation flags (e.g. foo.c (-DFOO)).

Changes:

• Introduces ExtraSource (path + per-file opts) and updates parsing/pretty-printing/field grammars so the five extra-source fields carry per-file options (guarded at cabal-version >= 3.20).
• Wires per-file options into GHC invocation by routing options into the appropriate GhcOptions fields (including version-gating -optJSP to GHC ≥ 9.12).
• Adds/updates documentation, changelog entries, and tests (parser regression + error test, QuickCheck roundtrip, rendering unit tests, and a setup testsuite case).

Reviewed changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
doc/file-format-changelog.rst	Documents the 3.20 format change for per-file extra-source options.
doc/cabal-package-description-file.rst	Adds user-facing documentation for per-file options and updates field docs.
changelog.d/per-file-extra-source-options.md	Changelog entry describing the new per-file options feature and API change.
changelog.d/jspp-options-ghc-912.md	Changelog entry for omitting -optJSP on GHC < 9.12.
changelog.d/cabal-spec-latest-3-20.md	Changelog entry for making 3.20 the latest known spec (PR #12275).
Cabal/src/Distribution/Simple/SrcDist.hs	Ensures sdist source collection includes ExtraSource file paths.
Cabal/src/Distribution/Simple/SetupHooks/Internal.hs	Updates autogen extra-source handling to extract ExtraSource file paths.
Cabal/src/Distribution/Simple/Program/GHC.hs	Gates -optJSP rendering to GHC ≥ 9.12 and updates related documentation.
Cabal/src/Distribution/Simple/GHCJS.hs	Adapts GHCJS build/link handling to ExtraSource-typed extra sources.
Cabal/src/Distribution/Simple/GHC/Internal.hs	Adds ExtraSourceKind + routing of per-file opts into correct GhcOptions.
Cabal/src/Distribution/Simple/GHC/Build/ExtraSources.hs	Switches extra-source builds to ExtraSource, applies opts per file, warns when JS opts are dropped (< 9.12).
Cabal/src/Distribution/Simple/BuildTarget.hs	Updates component file listing to extract file paths from ExtraSource.
Cabal/src/Distribution/Simple/Build.hs	Updates build-info augmentation helpers to wrap paths with extraSourceFromPath.
Cabal/src/Distribution/PackageDescription/Check/Target.hs	Updates path well-formedness checks to validate extraSourceFile.
Cabal-tree-diff/src/Data/TreeDiff/Instances/Cabal.hs	Adds ToExpr ExtraSource instance for tree-diff output.
cabal-testsuite/PackageTests/SetupHooks/SetupHooksNonHs/SetupHooks.hs	Updates setup hooks test to supply ExtraSource values.
cabal-testsuite/PackageTests/ExtraSources/src/MyLib.hs	Adds a testsuite library module for the new extra-sources setup test.
cabal-testsuite/PackageTests/ExtraSources/setup.test.hs	Adds a setup-based integration test for compiling extra sources.
cabal-testsuite/PackageTests/ExtraSources/setup.out	Expected output for the ExtraSources setup test.
cabal-testsuite/PackageTests/ExtraSources/extra-sources.cabal	Test package exercising per-file options in c-sources / cmm-sources.
cabal-testsuite/PackageTests/ExtraSources/cbits/testcmm.cmm	Test C-- file requiring a macro defined via per-file options.
cabal-testsuite/PackageTests/ExtraSources/cbits/test.c	Test C file requiring a macro defined via per-file options.
Cabal-tests/tests/UnitTests/Distribution/Utils/Structured.hs	Updates structured-hash golden values after type/API changes.
Cabal-tests/tests/UnitTests/Distribution/Types/ExtraSource.hs	Adds QuickCheck roundtrip property for ExtraSource parse/pretty.
Cabal-tests/tests/UnitTests/Distribution/Simple/Program/GHC.hs	Adds unit tests for -optJSP gating and C-- option rendering behavior.
Cabal-tests/tests/UnitTests.hs	Registers new ExtraSource unit tests.
Cabal-tests/tests/ParserTests/regressions/libpq2.expr	Updates expected parse AST output for cSources now being ExtraSource.
Cabal-tests/tests/ParserTests/regressions/libpq1.expr	Updates expected parse AST output for cSources now being ExtraSource.
Cabal-tests/tests/ParserTests/regressions/extra-source-opts.format	Adds formatting regression case for per-file options across all five kinds.
Cabal-tests/tests/ParserTests/regressions/extra-source-opts.expr	Adds AST expectation for the per-file extra-source options regression test.
Cabal-tests/tests/ParserTests/regressions/extra-source-opts.cabal	Adds .cabal input exercising per-file options parsing/printing.
Cabal-tests/tests/ParserTests/errors/extra-source-opts-old-spec.errors	Adds expected error output when using per-file opts below spec 3.20.
Cabal-tests/tests/ParserTests/errors/extra-source-opts-old-spec.cabal	Adds error fixture using per-file opts with cabal-version: 3.4.
Cabal-tests/tests/ParserTests.hs	Registers new parser regression and error tests.
Cabal-tests/tests/NoThunks.hs	Adds NoThunks ExtraSource instance for no-thunks testing.
Cabal-tests/Cabal-tests.cabal	Adds new unit test module to the Cabal-tests test-suite stanza.
Cabal-syntax/src/Distribution/Types/ExtraSource.hs	Introduces the ExtraSource type plus parsing/pretty-printing and version guard.
Cabal-syntax/src/Distribution/Types/BuildInfo/Lens.hs	Updates build-info lenses to expose extra-source fields as [ExtraSource].
Cabal-syntax/src/Distribution/Types/BuildInfo.hs	Updates BuildInfo extra-source fields to [ExtraSource] with clarified comments.
Cabal-syntax/src/Distribution/SPDX/LicenseListVersion.hs	Maps CabalSpecV3_20 to SPDX license list 3.28.
Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs	Updates field grammars to parse/format extra-source fields as ExtraSource.
Cabal-syntax/src/Distribution/PackageDescription.hs	Re-exports Distribution.Types.ExtraSource from Distribution.PackageDescription.
Cabal-syntax/src/Distribution/CabalSpecVersion.hs	Sets cabalSpecLatest to CabalSpecV3_20.
Cabal-syntax/Cabal-syntax.cabal	Exposes the new Distribution.Types.ExtraSource module.
Cabal-QuickCheck/src/Test/QuickCheck/Instances/Cabal.hs	Adds Arbitrary ExtraSource generator/shrinker.
cabal-install/tests/IntegrationTests2.hs	Updates integration test fixture construction for cSources :: [ExtraSource].
cabal-install/src/Distribution/Client/TargetSelector.hs	Updates known-component file collection to extract extraSourceFile.
cabal-install/src/Distribution/Client/SourceFiles.hs	Updates source file existence tracking to extract extraSourceFile.
Cabal-hooks/src/Distribution/Simple/SetupHooks.hs	Re-exports ExtraSource and extraSourceFromPath for hooks packages.
Cabal-described/src/Distribution/Described.hs	Adds Described ExtraSource and updates regex charsets for options syntax.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[andreabedini]

Thank you for the review @zlonast

[zlonast]

I don't like the current syntax, so I'll suggest two other options that I think fit better into the ecosystem design.

Option A — file as a section with a nested field

library
  c-source: cbits/foo.c
    c-options: -DFOO -O2

  c-source: cbits/bar.c

Option B — options as a separate map field

library
  c-sources:
    cbits/foo.c
    cbits/bar.c

  c-options:
    cbits/foo.c: -DFOO -O2

[andreabedini]

I don't think either of those can work, but the objection did make me change the syntax — just not to A or B.

A isn't expressible. There are no nested fields in .cabal files. fieldLayout in Cabal-syntax/src/Distribution/Fields/Parser.hs reads a field as an optional first line plus many (do _ <- indentOfAtLeast ilevel; fieldContent), so every more-indented line is more content of the same field. Your example parses as one field c-source whose value is the two lines cbits/foo.c and c-options: -DFOO -O2 — the colon is just text there. That needs a new production in the grammar, not a new field.

B parses, but it splits something that is currently one value. An ExtraSource is self-contained, so [ExtraSource] merges monoidally and there is nothing to keep in sync. A separate map keyed by a path gives two independently merged fields:

library
  c-sources: cbits/foo.c
  if os(windows)
    c-sources: cbits/win.c
    c-options: cbits/foo.c: -DWIN

Both are monoidalFields and accumulate separately, so nothing guarantees a key names a file that is in c-sources in that configuration, and a typo in the path silently does nothing. common stanzas make it worse. Keeping the options on the file makes that unrepresentable.

What I did change: the text between the parentheses is now taken verbatim and split by splitArgs — the same function --PROG-options uses, so the quoting rules are the command line's. Only ( and ) are structural; they nest as long as they balance, and \ escapes (, ) and \. A backslash anywhere else stands for itself, so Windows paths need no doubling.

c-sources:
  cbits/wide.c ("-DMESSAGE=hello there")
  cbits/size.c (-DSIZE=f(1))
  cbits/smiley.c (-DSMILEY=\))

That also killed a bug: (-DX="a b") used to parse as two options, -DX= and a b, and pretty-printing rendered it back as (-DX= "a b") — making the misreading permanent.

[AndreasPK]

The (-DSMILEY=\)) syntax seems very unintuitive to me given how cabal syntax works in general. If there is a need for this feature some of the options @zlonast offered seem far more reasonable.

A isn't expressible. There are no nested fields in .cabal files.

We already have nested blocks, and even if there were no nested blocks we we could still just add them anyway if everyone agrees this is the best way to do it.

[sheaf]

I think this change requires buy-in from the community and the Cabal maintainers before we can proceed to technical review. I say this for two main reasons:

• We need to get some kind of consensus on the syntax. I know this is painful as it will invite a lot of bike-shedding, but I don't think we can skip it.
• It's not clear to me that we want this feature at all. To me it looks like trying to replicate parts of Make in Cabal and that seems wrong to me. Can't we solve the issues Hadrian faces by using build-type: Configure or build-type: Hooks to programmatically change the flags passed when compiling extra sources instead of requiring new syntax in Cabal files?

The Cabal proposal repository can be found here.

[sheaf]

I would like to apologise, as I did not intend my previous reply to be discouraging. I definitely appreciate the effort invested in splitting off the work to build GHC with Cabal into individually reviewable parts.

My concern is that there is an endless rabbit hole of C build system complexity that we do not want to drag into Cabal, which is why I liked the idea of keeping that complexity outside of Cabal itself. However, I recognise that this is anachronistic: from its inception, Cabal has functioned as build system for "Haskell + supporting low-level C", and it seems pragmatically reasonable to continue in that direction.

I think the framing (which you do give in the OP!) that this is about adding "just enough" C build system feature support into Cabal to build GHC's RTS is key. Do you think you would be able to provide a summary of the user-facing changes that this requires beyond this change? I think that context would be very helpful, as it would allow us to see what the end-state would look like.

[andreabedini]

A isn't expressible. There are no nested fields in .cabal files.

We already have nested blocks, and even if there were no nested blocks we we could still just add them anyway if everyone agrees this is the best way to do it.

Let's stick to the names used by cabal's grammar, there are no "block", there are fields (like build-depends:) and sections (like library or custom-setup).

data Field ann
  = Field   !(Name ann) [FieldLine ann]
  | Section !(Name ann) [SectionArg ann] [Field ann]

Option A used nested fields, which are not part of the grammar. Of course we could add them if that is what we really want to do, but notice that the fields grammar has not changed since the introduction of sections ~20 years ago. You would agree with me that trying to fix into the existing grammar should be tried first.

On cbits/smiley.c (-DSMILEY=\)). Ok, fair to say, this was because I thought it would be better to make ( and ) special and leave " as a normal character, so that you could write (-DMESSAGE="hello there") rather than ("-DMESSAGE=\"hello there\"")[1]. I have no problem going back to the original design of the PR description, as said above, I had found a quirk and thought a way to address it.

I think this change requires buy-in from the community and the Cabal maintainers before we can proceed to technical review.

Absolutely agree. I can open a proposal if needed. Admittedly, I do not see the point, since it would be the exact same discussion as we can have here, never the less, the extra effort is minimal, so I am up for it.

@sheaf No, need to apologise :-)

This change is one of the very few changes to Cabal needed for stable-haskell/ghc. There we use cabal-install to build GHC and building the rts requires passing different compiler options for different c source files.

I would argue there is nothing dynamic here. The set of flags and the set of files is entirely static.

Can't we solve the issues Hadrian faces by using build-type: Configure or build-type: Hooks to programmatically change the flags passed when compiling extra sources instead of requiring new syntax in Cabal files?

Good question.

The funny thing is that rts.cabal already uses build-type: Configure but cannot express this either. The configure script can contribute more build-info but those build-info have the same shape and form of the build-info in the cabal file, so the same single c-options for all c-sources. On other words, build-type: Configure adds dynamism to build-info, but what build-info can express stays the same.

Of the top of my head SetupHooks would have the same issue.

Do you think you would be able to provide a summary of the user-facing changes that this requires beyond this change? I think that context would be very helpful, as it would allow us to see what the end-state would look like.

The "end state" is here: rts. To be fair, I do not like it :D there is still too much autoconf complexity in there and we all spent effort in reducing it. Admittedly, I haven't looked at the rts side for a while, but I would be happy to look again if you have any suggestion.

[1]: The outside quotes are necessary to prevent the space from splitting the input into two options. The inside quotes, now escaped, are the ones passed on.

[dcoutts]

Thanks @andreabedini. It's good to have per-feature PRs like this. It's a good approach towards the goal of getting upstream Cabal support for building the rts and ghc.

As @sheaf asks, it would be very useful for people reviewing these to have some sense of the roadmap: what features overall you think need to be added. Knowing what's coming may legitimately influence design choices.

On the very important bike shedding issue of syntax...

So we have these existing .cabal fields that cover files and options:

sources field	options field
c-sources	cc-options
cxx-sources	cxx-options
asm-sources	asm-options
cmm-sources	cmm-options
js-sources	jspp-options
(Haskell modules)	cpp-options

I think we can probably omit per-file support for Haskell modules since for these we have per-file control anyway via {-# OPTIONS_GHC -opta-blah-blah #-} etc. It's the "foreign" languages where we do not have that control (and cmm, but whatever) and so desire some extra fine-grained per-file control over options.

So that leaves just

sources field	options field
c-sources	cc-options
cxx-sources	cxx-options
cmm-sources	cmm-options
js-sources	jspp-options

I share people distaste for making the syntax of the field contents more complex. I think we should be able to express maps in our existing syntax in some reasonable way.

I think @zlonast makes a good suggestion

library
  c-sources:
    cbits/foo.c
    cbits/bar.c

  c-options:
    cbits/foo.c: -DFOO -O2

though, a couple minor issues:

• the c-options should really be cc-options to match the existing options name; and
• it needs to be cc-options as a section (which can have fields beneath it), not cc-options: as a field (which cannot).

Thus it would be

library
  c-sources:
    cbits/foo.c
    cbits/bar.c

  cc-options: -O1
  cc-options
    cbits/foo.c: -DFOO -O2

Note the reuse of the name between the field (that applies to all files) and the section which contains a map of per-file options. The per-file options obviously needs to be appended to the all files options so that the specific overrides the general.

I'm not sure I totally understand the concern you expressed about this.

Both are monoidalFields and accumulate separately, so nothing guarantees a key names a file that is in c-sources in that configuration, and a typo in the path silently does nothing.

Yes, the cc-options field and my proposed cc-options section map are monoidal fields, separate from the c-sources. Yes there is a risk that a user may provide options for a file that does not exist. But this is easily verified after all the monoidal merging is done. And there is additional flexibility in a separate monoidal field: allowing some options to be conditional as in your example (recast in my proposed syntax).

library
  c-sources: cbits/foo.c
  if os(windows)
    c-sources: cbits/win.c
    cc-options
      cbits/foo.c: -DWIN

I get your point that a list of pairs might be nicer than a pair of lists for "statically" avoiding mistakes. But the Cabal syntax is not terribly good at representing lists of pairs, it's much better at pairs of lists. And doing "dynamic" error checking in practice occurs at the same time (from the users perspective) as static enforcement in the syntax (I mean parse errors are reported at essentially the same time as configuration errors: when one invokes cabal, or hls does it for you).

The other alternative, following the list of pairs approach would be something where you specify files, and can then specify one or more fields for them with extra info. That scales to mentioning many fields relating to one file. But this doesn't seem a great fit here, since we don't have several fields per-source file (of any given type). So this syntax would feel cumbersome most of the time.

[sheaf]

Absolutely agree. I can open a proposal if needed. Admittedly, I do not see the point, since it would be the exact same discussion as we can have here, never the less, the extra effort is minimal, so I am up for it.

The point is that we want to invite the community of Cabal users to discuss the design, instead of confining the discussion to a PR which only technical contributors are going to be looking at.

[dcoutts]

In your real world example:

      if arch(x86_64)
        cmm-sources:
          Jumps_V32.cmm (-mavx2)
          Jumps_V64.cmm (-mavx512f)
          AutoApply_V32.cmm (-mavx2)
          AutoApply_V64.cmm (-mavx512f)
      else
        cmm-sources:
          Jumps_V32.cmm
          Jumps_V64.cmm
          AutoApply_V32.cmm
          AutoApply_V64.cmm

would become:

      cmm-sources:
        Jumps_V32.cmm
        Jumps_V64.cmm
        AutoApply_V32.cmm
        AutoApply_V64.cmm
      if arch(x86_64)
        cmm-options
          Jumps_V32.cmm: -mavx2
          Jumps_V64.cmm: -mavx512f
          AutoApply_V32.cmm: -mavx2
          AutoApply_V64.cmm: -mavx512f

Which I argue is nicer. The extra flexibility of having the field separate allows us to eliminate some repetition.

Yes we have to verify that the named files we end up with for per-file options do exist in the component.

But note that in the first approach, you have another source of silent errors: getting the lists of files in the arch(x86_64) and !arch(x86_64) cases out of sync!

In the latter approach we can list those common files once, and provide the conditional flags which are checkable.

[andreabedini]

Thanks @dcoutts.

Note the reuse of the name between the field (that applies to all files) and the section which contains a map of per-file options. The per-file options obviously needs to be appended to the all files options so that the specific overrides the general.
I'm not sure I totally understand the concern you expressed about this.

I think having a field and a section with the same name, while allowed in the field grammar, would complicate the parser. I am fairly confident it would require substantial changes to the FieldGrammar parser.

Also, this is a matter of taste perhaps, I perceive as a even bigger change that the one I was proposing.

Now, as I was replying (oh god these discussions are useful!) I noticed that we need this for cmm files not for c files and we do control that parser so maybe a GHC solution like # OPTIONS_GHC -opta-blah-blah could be possible. I will investigate.

[dcoutts]

I think having a field and a section with the same name, while allowed in the field grammar, would complicate the parser. I am fairly confident it would require substantial changes to the FieldGrammar parser.

It looks to me like it should be relatively straightforward. The untyped field parser can recognise the syntactic category (field vs section):

ghci> readFields "cc-options: -DBar\ncc-options\n  foo.c: -DFoo"
Right [Field (Name (Position 1 1) "cc-options") [FieldLine (Position 1 13) "-DBar"],Section (Name (Position 2 1) "cc-options") [] [Field (Name (Position 3 3) "foo.c") [FieldLine (Position 3 10) "-DFoo"]]]

and after that, for parsing specific known fields and sections, they are now in different categories (namespaces), Field vs Section, so are handled quite differently.

Seems to me it's worth looking into in more detail to assess the feasibility. You can set your LLM on the task 😜 .