Google Git
Sign in
fuchsia / third_party / github.com / google / licenseclassifier / 944bfc450d2ff7a35690262337efc3a3154cf94f
commit944bfc450d2ff7a35690262337efc3a3154cf94f[log] [tgz]
authorBill Neubauer <wcn@google.com>Thu Aug 05 10:54:58 2021 -0700
committerBill Neubauer <wcn@google.com>Wed Mar 16 15:31:27 2022 -0700
tree921dc8abe87173ed269cc5c0f56bb7fe11fdb270
parentef8812216d51a3a43a3e9409169f8ba06a512766 [diff]
Change the licenseclassifier return type to a summary structure rather than a
raw list of matches. This allows us to return additional metadata about the
original input to the classifier. One datum currently included in this summary
is the number of lines of input presented to the classifier, which is necessary
to identify ranges of text that did not contain a license.

This change consolidates the usages of the third party classifier API around a
common interface for the compliance team and provides a common stub,
eliminating many ad-hoc stub implementations.

This CL was tested via global presubmit to ensure usages outside of the
compliance team were not negatively impacted.  *** Change 388303070

PiperOrigin-RevId: 388973411
2 files changed
tree: 921dc8abe87173ed269cc5c0f56bb7fe11fdb270
  1. commentparser/
  2. internal/
  3. licenses/
  4. serializer/
  5. stringclassifier/
  6. tools/
  7. v2/
  8. .travis.yml
  9. CHANGELOG
  10. classifier.go
  11. classifier_test.go
  12. CONTRIBUTING.md
  13. file_system_resources.go
  14. forbidden.go
  15. go.mod
  16. go.sum
  17. LICENSE
  18. license_type.go
  19. README.md
README.md

License Classifier

Build status

Introduction

The license classifier is a library and set of tools that can analyze text to determine what type of license it contains. It searches for license texts in a file and compares them to an archive of known licenses. These files could be, e.g., LICENSE files with a single or multiple licenses in it, or source code files with the license text in a comment.

A “confidence level” is associated with each result indicating how close the match was. A confidence level of 1.0 indicates an exact match, while a confidence level of 0.0 indicates that no license was able to match the text.

Adding a new license

Adding a new license is straight-forward:

  1. Create a file in licenses/.

    • The filename should be the name of the license or its abbreviation. If the license is an Open Source license, use the appropriate identifier specified at https://spdx.org/licenses/.
    • If the license is the “header” version of the license, append the suffix “.header” to it. See licenses/README.md for more details.

  2. Add the license name to the list in license_type.go.

  3. Regenerate the licenses.db file by running the license serializer:

    $ license_serializer -output licenseclassifier/licenses

  4. Create and run appropriate tests to verify that the license is indeed present.

Tools

Identify license

identify_license is a command line tool that can identify the license(s) within a file.

$ identify_license LICENSE
LICENSE: GPL-2.0 (confidence: 1, offset: 0, extent: 14794)
LICENSE: LGPL-2.1 (confidence: 1, offset: 18366, extent: 23829)
LICENSE: MIT (confidence: 1, offset: 17255, extent: 1059)

License serializer

The license_serializer tool regenerates the licenses.db archive. The archive contains preprocessed license texts for quicker comparisons against unknown texts.

$ license_serializer -output licenseclassifier/licenses

This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.