v33.0.0rc1

[AyanSinhaMahapatra]

This is a major release candidate with major changes in packaging,
with the licensedcode-data and licensedcode-index being published
in two seperate wheels. Also adds linux/macos ARM support in
release archives and pypi wheels.

• Remove the licensedcode data and built license indexes from the
  main scancode-toolkit built wheel, and release them as
  seperate wheels which scancode-toolkit depends on.
  This enables to release scancode licensedb data/index wheels
  seperately whenever necessary and also reduces the scancode
  wheel size greatly. For more details see:
  Improve ScanCode packaging and reduce wheel size #3459

• Add macos ARM and linux arm support in release archives and
  pypi wheels through scancode-toolkit-mini which uses
  system provided libraries instead of bundled binaries.
  Also use updated non pure-python dependencies supporting
  linux/macos arm wheels. For more details see:
  Many scancode dependencies are broken for Mac M1 #3205
  Support arm64/aarch64 #3958

• Add new and updates licenses, and license rules, including
  support for the SPDX License List 3.28
  Update licenses #4956

• Add support for the Python UV package manager. Two new package data
  handlers parse pyproject.toml files containing a [tool.uv] table
  and uv.lock lockfiles, including PEP 735 [dependency-groups],
  and the package assembly walks both files together so that the project
  metadata and the resolved transitive dependencies are reported as a
  single Python package.
  Support parsing python UV manifests #4501

What's Changed

• Fix pypi python3.14 wheel release by @AyanSinhaMahapatra in Fix pypi python3.14 wheel release #4686
• Rules for golang related false positives by @CsatariGergely in Rules for golang related false positives #4694
• Pynacl false positive upstream by @CsatariGergely in Pynacl false positive upstream #4695
• Fix dead scancode documentation link by @ShraddhaSharma3 in Fix dead scancode documentation link #4692
• Add new licenses and update SPDX License List to v3.28.0 by @AyanSinhaMahapatra in Add new licenses and update SPDX License List to v3.28.0 #4625
• Update Pixar regression fixture by @kumarasantosh in Update Pixar regression fixture #4878
• Remove duplicate '/project.clj' key in _MANIFEST_ENDS dict by @codewithfourtix in Remove duplicate '/project.clj' key in _MANIFEST_ENDS dict #4781
• Update licenses by @AyanSinhaMahapatra in Update licenses #4956 thanks to @DennisClark @armijnhemel @pombredanne for reporting and adding these.
• Publish licensedcode data/index as seperate pypi wheels and improve arm installation support by @AyanSinhaMahapatra in Publish licensedcode data/index as seperate pypi wheels and improve arm installation support #4847
• Make 'no such option' assertion tolerant of newer click error format by @GuillemSeCa in Make 'no such option' assertion tolerant of newer click error format #5009
• Adding UV package manager support for Python projects by @GuillemSeCa in Adding UV package manager support for Python projects #4980
• Bump version and CHANGELOG to v33.0.0rc1 and add new licenses by @AyanSinhaMahapatra in Bump version and CHANGELOG to v33.0.0rc1 #5041

New Contributors

• @CsatariGergely made their first contribution in Rules for golang related false positives #4694
• @ShraddhaSharma3 made their first contribution in Fix dead scancode documentation link #4692
• @kumarasantosh made their first contribution in Update Pixar regression fixture #4878
• @codewithfourtix made their first contribution in Remove duplicate '/project.clj' key in _MANIFEST_ENDS dict #4781
• @GuillemSeCa made their first contribution in Make 'no such option' assertion tolerant of newer click error format #5009

Full Changelog: v32.5.0...v33.0.0rc1

---

This discussion was created from the release v33.0.0rc1.

[stefan6419846]

For me, the default installation with

pip install --no-cache scancode-toolkit==33.0.0rc1 --index-url https://pypi.org/simple

would break the following snippet:

from licensedcode.models import load_licenses_from_multiple_dirs, licenses_data_dir

load_licenses_from_multiple_dirs(
    builtin_license_data_dir=licenses_data_dir,
    additional_license_data_dirs=[],
    with_deprecated=False,
)

Output:

licensedcode.models.InvalidLicense: No licenses were loaded. Check to see if the license data files are available at "/home/user/lib/python3.11/site-packages/licensedcode/data/licenses".

Installing licensedcode-data solved this, but given the description, it seems like it should have been installed automatically.

[stefan6419846]

By the way: I get a ton of duplicate rules reported, which I have never seen before as well (although I might just have missed this):

licensedcode.index.DuplicateRuleError: Duplicate rules: 
file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0-plus_1.RULE
file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0-plus_3.RULE
file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0-plus_4.RULE

file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0-plus_41.RULE
file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0-plus_43.RULE
file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0-plus_44.RULE

file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0_22.RULE
file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0_24.RULE
file:///home/user/lib/python3.11/site-packages/licensedcode/data/rules/agpl-1.0_25.RULE

...

[AyanSinhaMahapatra]

@stefan6419846 thanks a lot for testing and catching this. I have opened #5053 to address this.

You should install scancode-toolkit[licenses] if you need the data too, see https://github.com/aboutcode-org/scancode-toolkit/blob/develop/pyproject-scancode-toolkit.toml#L113
But this should be indicated, and a proper error raised for this.

The duplicate rule seems super weird, let me check why it's happening.
Is the licensedcode.index.DuplicateRuleError: Duplicate rules: error also coming when you call load_licenses_from_multiple_dirs?

[stefan6419846]

You should install scancode-toolkit[licenses] if you need the data too

This does not work as there seems to be a typo in the package names required by the extra:

ERROR: Could not find a version that satisfies the requirement licensecode-data; extra == "licenses" (from scancode-toolkit[licenses]) (from versions: none)

Regarding the duplicate rules: This is much more complex than anticipated, as I am suddenly seeing my test runtimes being doubled. Turns out that for testing purposes regarding the RC, we strictly need the --pre flag when installing SCTK or specify the RC for licensedcode-index. Otherwise loading the cache file fails with an AttributeError due to structure changes. The fallback is to generate the cache on each call, which is both slow and triggers the duplicate warnings (although I think that these messages about duplicates should be addressed nevertheless).

[stefan6419846]

The duplicates issue seems to be related to the package versions as well: They are only triggered for licensedcode-data<33, id est the current regular release, while licensedcode-data==33.0.0rc1 seems to resolve this as well.