karl/bucketeer
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ctcloud: CertSpotter-backed CT cloud-resource monitor

Browse Source 
Single stateless Go process that consumes CT logs via embedded (patched)
CertSpotter internals, classifies cloud/database/storage DNS identifiers
in-process with a zero-allocation classifier, enriches via round-robin DNS
resolvers with CNAME following, and batch-writes useful findings to PostgreSQL.

- third_party/certspotter: 2-hunk IdentifierSink patch, no stdout/JSON/files
- internal/classify: AWS/Azure/GCP/Firebase/Cloudflare R2/DigitalOcean/Supabase/OCI
- internal/dns: sharded bounded TTL cache, multi-resolver load balancing + failover
- internal/pipeline: bounded queues, dedup, direct-priority load shedding
- internal/store: pgx batch writer, Postgres CT checkpoints (no cert contents)
- internal/probe: optional non-destructive reachability checks (off by default)
- Docker: distroless non-root, read-only rootfs, no app volume, resource limits
- tests enforce 0 allocs/op on the classifier and sink hot path
This commit is contained in:
StasonJatham committed 2026-09-22 13:13:10 +02:00
1 parent 9e78a114ac
commit e8f7d6bfdf
118 files changed
+16473
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

third_party/certspotter/.github/workflows/test.yml
Vendored
+35
View File
@@ -0,0 +1,35 @@
name: Test and lint Go Code
on:
push:
schedule:
- cron: '42 9 * * *' # Runs daily at 09:42 UTC
workflow_dispatch: # Allows manual triggering
permissions:
contents: read
jobs:
test:
name: Test
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
- name: Run tests
run: CGO_ENABLED=1 go test -race ./...
- name: Install staticcheck
run: go install honnef.co/go/tools/cmd/staticcheck@latest
- name: Run staticcheck
run: staticcheck ./...
third_party/certspotter/CHANGELOG.md
Vendored
+267
View File
@@ -0,0 +1,267 @@
# Change Log
## v0.25.0 (2026-09-13)
- Improve performance monitoring slow RFC 6962 logs whose get-entries
responses are interrupted.
- Add experimental certspotter-metrics command for exporting Prometheus
metrics about certspotter's operation. Please leave feedback in
<https://github.com/SSLMate/certspotter/issues/79>.
- Minor bug fixes and robustness improvements.
## v0.24.2 (2026-06-25)
- Extract all Common Names from Subject instead of first one in each RDN. Note
that modern certificate validators, including those used by all CT-enforcing
clients, ignore the Common Name. Therefore, any certificate missed by
certspotter due to this bug would not work in CT-enforcing clients anyways.
- Support -verbose with -version to print full build info.
- Fix some possible crashes in pathological situations.
## v0.24.1 (2026-05-23)
- Update dependencies.
- Minor bug fixes.
## v0.24.0 (2026-01-12)
- Add certspotter-authorize command, which lets you mark legitimate certificates
as authorized so you don't get notified about them.
- Deduplicate certificates and their corresponding precertificates, so you
receive only one notification when both are logged in CT.
- When `-no_save` is used, write certificates to a temporary location so scripts
can access them via `$CERT_FILENAME`, `$JSON_FILENAME`, and `$TEXT_FILENAME`.
## v0.23.0 (2025-12-05)
- Avoid sending queries to logs in excess of the log's rate limit, if known.
This should improve monitoring of the TrustAsia logs.
## v0.22.0 (2025-09-24)
- Retry failed requests more aggressively so certspotter doesn't fall behind
downloading entries.
- Support `$CERTSPOTTER_CACHE_DIR` for overriding cache directory location.
- Support `$STATE_DIRECTORY`, `$CONFIGURATION_DIRECTORY`, and `$CACHE_DIRECTORY`
when running under systemd.
- Truncate long error responses from logs, to keep size of error files down.
## v0.21.0 (2025-07-02)
- Instead of writing log errors to stderr, save the last 7 days worth in the
state directory, and include recent errors in failed health check notifications.
- Send a meaningful User-Agent string with HTTP requests.
## v0.20.1 (2025-06-19)
- Add resilience against sendmail hanging indefinitely.
- Add resilience against hooks which fork and keep stderr open.
- Upgrade dependencies to latest versions.
- Minor improvements to error handling, code quality, and efficiency.
## v0.20.0 (2025-06-13)
- Remove -batch_size option, which is obsolete due to new parallel download system.
- Only print log errors to stderr if -verbose is specified.
- Fix bug that could cause unverified STHs to be deleted prematurely.
- Fail health check if log has never been successfully contacted.
- Improve -verbose.
- Improve documentation.
## v0.19.1 (2025-05-07)
- Fix panic when retrying failed log requests.
- Properly log failed log requests.
## v0.19.0 (2025-05-07) (RETRACTED)
- Add support for static-ct-api logs, the next generation of CT logs.
- Add support for downloading entries in parallel, to avoid backlogs when
monitoring fast-growing logs.
- Use $EMAIL environment variable to determine sender of emails.
- Remove submitct command.
- Make certspotter more resilient to system crashes such as power failures.
## v0.18.0 (2023-11-13)
- Fix bug with downloading entries that did not materialize in practice
with any of the current logs.
- Include `Message-ID` and `Date` in outbound emails.
## v0.17.0 (2023-10-26)
- Allow sendmail path to be configured with `$SENDMAIL_PATH`.
- Minor improvements to documentation, efficiency.
## v0.16.0 (2023-02-21)
- Write malformed certs and failed healthchecks to filesystem so scripts
can access them.
- Automatically execute scripts under `$CERTSPOTTER_CONFIG_DIR/hooks.d`
if it exists.
- Automatically email addresses listed in `$CERTSPOTTER_CONFIG_DIR/email_recipients`
if it exists.
## v0.15.1 (2023-02-09)
- Fix some typos in help and error messages.
- Allow version to be set via linker flag, to facilitate distro package building.
## v0.15.0 (2023-02-08)
- **Significant behavior change**: certspotter is now intended to run as
a daemon instead of a cron job. Specifically, certspotter no longer
terminates unless it receives SIGTERM or SIGINT or there is a serious error.
You should remove certspotter from your crontab and arrange to run it as a
daemon, passing either the `-email` option or `-script` option to configure
how you want to be notified about certificates.
Reason for this change: although using cron made sense in the early days of
Certificate Transparency, certspotter now needs to run continuously to reliably
keep up with the high growth rate of contemporary CT logs, and to gracefully
handle the many transient errors that can arise when monitoring CT.
See <https://github.com/SSLMate/certspotter/issues/63> for background.
- `-script` is now officially supported and can be used to execute
a command when a certificate is discovered or there is an error. For details,
see the [certspotter-script(8) man page](man/certspotter-script.md).
Note the following changes from the experimental, undocumented `-script`
option found in previous versions:
- The script is also executed when there is an error. Consult the `$EVENT`
variable to determine why the script was executed.
- The `$DNS_NAMES` and `$IP_ADDRESSES` variables have been removed because
the OS limits the size of environment variables and some certificates have
too many identifiers. To determine a certificate's identifiers, you can
read the JSON file specified by the `$JSON_FILENAME` variable, as explained
in the [certspotter-script(8) man page](man/certspotter-script.md).
- The `$CERT_TYPE` variable has been removed because it is almost always
a serious mistake (that can make you miss malicious certificates) to treat
certificates and precertificates differently. If you are currently
using this variable to skip precertificates, stop doing that because
precertificates imply the existence of a corresponding certificate that you
**might not** be separately notified about. For more details, see
<https://github.com/SSLMate/certspotter/commit/cd2bb429fc2f4060a33ec8eb8b71a3eb12e9ba93>.
- New variable `$WATCH_ITEM` contains the first watch list item which
matched the certificate.
- New `-email` option can be used to send an email when a certificate is
discovered or there is an error. Your system must have a working `sendmail`
command.
- (Behavior change) You must specify the `-stdout` option if you want discovered
certificates to be written to stdout. This only makes sense when running
certspotter from the terminal; when running as a daemon you probably want to
use `-email` or `-script` instead.
- Once a day, certspotter will send you a notification (per `-email` or
`-script`) if any problems are preventing it from detecting all certificates.
As in previous versions of certspotter, errors are written to stderr when they
occur, but since most errors are transient, you can now ignore stderr and rely
on the daily health check to notify you about any persistent problems that
require your attention.
- certspotter now saves `.json` and `.txt` files alongside the `.pem` files
containing parsed representations of the certificate.
- `.pem` files no longer have `.cert` or `.precert` in the filename.
- certspotter will save its state periodically, and before terminating due to
SIGTERM or SIGINT, meaning it can resume monitoring without having to
re-download entries it has already processed.
- The experimental "BygoneSSL" feature has been removed due to limited utility.
- The `-num_workers` option has been removed.
- The `-all_time` option has been removed. You can remove the certspotter state
directory if you want to re-download all entries.
- The minimum supported Go version is now 1.19.
## v0.14.0 (2022-06-13)
- Switch to Go module versioning conventions.
## v0.13 (2022-06-13)
- Reduce minimum Go version to 1.17.
- Update install instructions.
## v0.12 (2022-06-07)
- Retry failed log requests. This should make certspotter resilient
to rate limiting by logs.
- Add `-version` flag.
- Eliminate unnecessary dependency. certspotter now depends only on
golang.org/x packages.
- Switch to Go modules.
## v0.11 (2021-08-17)
- Add support for contacting logs via HTTP proxies;
just set the appropriate environment variable as documented at
<https://golang.org/pkg/net/http/#ProxyFromEnvironment>.
- Work around RFC 6962 ambiguity related to consistency proofs
for empty trees.
## v0.10 (2020-04-29)
- Improve speed by processing logs in parallel
- Add `-start_at_end` option to begin monitoring new logs at the end,
which significantly speeds up Cert Spotter, at the cost of missing
certificates that were added to a log before Cert Spotter starts
monitoring it
- (Behavior change) Scan logs in their entirety the first time Cert
Spotter is run, unless `-start_at_end` specified (behavior change)
- The log list is now retrieved from certspotter.org at startup instead
of being embedded in the source. This will allow Cert Spotter to react
more quickly to the frequent changes in logs.
- (Behavior change) the `-logs` option now expects a JSON file in the v2
log list format. See <https://www.certificate-transparency.org/known-logs>
and <https://www.gstatic.com/ct/log_list/v2/log_list_schema.json>.
- `-logs` now accepts an HTTPS URL in addition to a file path.
- (Behavior change) the `-underwater` option has been removed. If you want
its behavior, specify `https://loglist.certspotter.org/underwater.json` to
the `-logs` option.
## v0.9 (2018-04-19)
- Add Cloudflare Nimbus logs
- Remove Google Argon 2017 log
- Remove WoSign and StartCom logs due to disqualification by Chromium
and extended downtime
## v0.8 (2017-12-08)
- Add Symantec Sirius log
- Add DigiCert 2 log
## v0.7 (2017-11-13)
- Add Google Argon logs
- Fix bug that caused crash on 32 bit architectures
## v0.6 (2017-10-19)
- Add Comodo Mammoth and Comodo Sabre logs
- Minor bug fixes and improvements
## v0.5 (2017-05-18)
- Remove PuChuangSiDa 1 log due to excessive downtime and presumptive
disqualification from Chrome
- Add Venafi Gen2 log
- Improve monitoring robustness under certain pathological behavior
by logs
- Minor documentation improvements
## v0.4 (2017-04-03)
- Add PuChuangSiDa 1 log
- Remove Venafi log due to fork and disqualification from Chrome
## v0.3 (2017-02-20)
- Revise `-all_time` flag (behavior change):
- If `-all_time` is specified, scan the entirety of all logs, even
existing logs
- When a new log is added, scan it in its entirety even if `-all_time`
is not specified
- Add new logs:
- Google Icarus
- Google Skydiver
- StartCom
- WoSign
- Overhaul log processing and auditing logic:
- STHs are never deleted unless they can be verified
- Multiple unverified STHs can be queued per log, laying groundwork
for STH pollination support
- New state directory layout; current state directories will be
migrated, but migration will be removed in a future version
- Persist condensed Merkle Tree state between runs, instead of
reconstructing it from consistency proof every time
- Use a lock file to prevent multiple instances of Cert Spotter from
running concurrently (which could clobber the state directory).
- Minor bug fixes and improvements
## v0.2 (2016-08-25)
- Suppress duplicate identifiers in output.
- Fix "EOF" error when running under Go 1.7.
- Fix bug where hook script could fail silently.
- Fix compilation under Go 1.5.
## v0.1 (2016-07-27)
- Initial release.
third_party/certspotter/CONTRIBUTING
Vendored
+44
View File
@@ -0,0 +1,44 @@
All contributors to Cert Spotter must accept the Developer Certificate
of Origin (DCO):
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
To accept the DCO, add the following line to each commit message with
your real name and email address (`git commit -s` will do this for you):
Signed-off-by: Alex Smith <[email protected]>
third_party/certspotter/CTCLOUD_PATCH.md
Vendored
+53
View File
@@ -0,0 +1,53 @@
# ctcloud patch to upstream CertSpotter
This directory is a copy of upstream CertSpotter
(https://github.com/SSLMate/certspotter), pulled in as a local module and
wired up with a `replace` directive in the root `go.mod`.
It is intentionally a copy (not a git submodule) so the patched tree lives in
this repository and is compiled by the Docker build with no network access to a
separate fork. To rebase on a newer upstream release:
1. Replace this directory with the new upstream tree.
2. Re-apply the two hunks below.
## Changes (only two hunks)
### `monitor/config.go`
Adds an exported `IdentifierSink` interface and a `Config.IdentifierSink`
field:
```go
type IdentifierSink interface {
HandleIdentifiers(ctx context.Context, identifiers *certspotter.Identifiers)
}
```
### `monitor/process.go`
In `processCertificate`, immediately after `certInfo.ParseIdentifiers()`, hand
the parsed identifiers to the sink and return, bypassing the watchlist,
certificate-chain retrieval and notification paths:
```go
if config.IdentifierSink != nil {
config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
return nil
}
```
## Why
The sink gives ctcloud parsed DNS identifiers via a direct in-process Go call:
```text
CT log entry -> CertSpotter parse -> IdentifierSink.HandleIdentifiers -> classifier
```
No stdout, no JSON, no files, no hooks, no subprocess. Setting the sink also
skips issuer-chain fetching and the watchlist machinery, which is exactly the
I/O and serialization overhead the design forbids.
Upstream is otherwise unmodified. License and notices are preserved in
`LICENSE`.
third_party/certspotter/LICENSE
Vendored
+373
View File
@@ -0,0 +1,373 @@
Mozilla Public License Version 2.0
==================================
1. Definitions
--------------
1.1. "Contributor"
means each individual or legal entity that creates, contributes to
the creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used
by a Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached
the notice in Exhibit A, the Executable Form of such Source Code
Form, and Modifications of such Source Code Form, in each case
including portions thereof.
1.5. "Incompatible With Secondary Licenses"
means
(a) that the initial Contributor has attached the notice described
in Exhibit B to the Covered Software; or
(b) that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the
terms of a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in
a separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible,
whether at the time of the initial grant or subsequently, any and
all of the rights conveyed by this License.
1.10. "Modifications"
means any of the following:
(a) any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered
Software; or
(b) any new file in Source Code Form that contains any Covered
Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the
License, by the making, using, selling, offering for sale, having
made, import, or transfer of either its Contributions or its
Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU
Lesser General Public License, Version 2.1, the GNU Affero General
Public License, Version 3.0, or any later versions of those
licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that
controls, is controlled by, or is under common control with You. For
purposes of this definition, "control" means (a) the power, direct
or indirect, to cause the direction or management of such entity,
whether by contract or otherwise, or (b) ownership of more than
fifty percent (50%) of the outstanding shares or beneficial
ownership of such entity.
2. License Grants and Conditions
--------------------------------
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
(a) under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
(b) under Patent Claims of such Contributor to make, use, sell, offer
for sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
(a) for any code that a Contributor has removed from Covered Software;
or
(b) for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
(c) under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights
to grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
in Section 2.1.
3. Responsibilities
-------------------
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
(a) such Covered Software must also be made available in Source Code
Form, as described in Section 3.1, and You must inform recipients of
the Executable Form how they can obtain a copy of such Source Code
Form by reasonable means in a timely manner, at a charge no more
than the cost of distribution to the recipient; and
(b) You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter
the recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty,
or limitations of liability) contained within the Source Code Form of
the Covered Software, except that You may alter any license notices to
the extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
---------------------------------------------------
If it is impossible for You to comply with any of the terms of this
License with respect to some or all of the Covered Software due to
statute, judicial order, or regulation then You must: (a) comply with
the terms of this License to the maximum extent possible; and (b)
describe the limitations and the code they affect. Such description must
be placed in a text file included with all distributions of the Covered
Software under this License. Except to the extent prohibited by statute
or regulation, such description must be sufficiently detailed for a
recipient of ordinary skill to be able to understand it.
5. Termination
--------------
5.1. The rights granted under this License will terminate automatically
if You fail to comply with any of its terms. However, if You become
compliant, then the rights granted under this License from a particular
Contributor are reinstated (a) provisionally, unless and until such
Contributor explicitly and finally terminates Your grants, and (b) on an
ongoing basis, if such Contributor fails to notify You of the
non-compliance by some reasonable means prior to 60 days after You have
come back into compliance. Moreover, Your grants from a particular
Contributor are reinstated on an ongoing basis if such Contributor
notifies You of the non-compliance by some reasonable means, this is the
first time You have received notice of non-compliance with this License
from such Contributor, and You become compliant prior to 30 days after
Your receipt of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all
end user license agreements (excluding distributors and resellers) which
have been validly granted by You or Your distributors under this License
prior to termination shall survive termination.
************************************************************************
* *
* 6. Disclaimer of Warranty *
* ------------------------- *
* *
* Covered Software is provided under this License on an "as is" *
* basis, without warranty of any kind, either expressed, implied, or *
* statutory, including, without limitation, warranties that the *
* Covered Software is free of defects, merchantable, fit for a *
* particular purpose or non-infringing. The entire risk as to the *
* quality and performance of the Covered Software is with You. *
* Should any Covered Software prove defective in any respect, You *
* (not any Contributor) assume the cost of any necessary servicing, *
* repair, or correction. This disclaimer of warranty constitutes an *
* essential part of this License. No use of any Covered Software is *
* authorized under this License except under this disclaimer. *
* *
************************************************************************
************************************************************************
* *
* 7. Limitation of Liability *
* -------------------------- *
* *
* Under no circumstances and under no legal theory, whether tort *
* (including negligence), contract, or otherwise, shall any *
* Contributor, or anyone who distributes Covered Software as *
* permitted above, be liable to You for any direct, indirect, *
* special, incidental, or consequential damages of any character *
* including, without limitation, damages for lost profits, loss of *
* goodwill, work stoppage, computer failure or malfunction, or any *
* and all other commercial damages or losses, even if such party *
* shall have been informed of the possibility of such damages. This *
* limitation of liability shall not apply to liability for death or *
* personal injury resulting from such party's negligence to the *
* extent applicable law prohibits such limitation. Some *
* jurisdictions do not allow the exclusion or limitation of *
* incidental or consequential damages, so this exclusion and *
* limitation may not apply to You. *
* *
************************************************************************
8. Litigation
-------------
Any litigation relating to this License may be brought only in the
courts of a jurisdiction where the defendant maintains its principal
place of business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions.
Nothing in this Section shall prevent a party's ability to bring
cross-claims or counter-claims.
9. Miscellaneous
----------------
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides
that the language of a contract shall be construed against the drafter
shall not be used to construe this License against a Contributor.
10. Versions of the License
---------------------------
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses
If You choose to distribute Source Code Form that is Incompatible With
Secondary Licenses under the terms of this version of the License, the
notice described in Exhibit B of this License must be attached.
Exhibit A - Source Code Form License Notice
-------------------------------------------
This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular
file, then You may include the notice in a location (such as a LICENSE
file in a relevant directory) where a recipient would be likely to look
for such a notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
---------------------------------------------------------
This Source Code Form is "Incompatible With Secondary Licenses", as
defined by the Mozilla Public License, v. 2.0.
third_party/certspotter/README.md
Vendored
+183
View File
@@ -0,0 +1,183 @@
# Cert Spotter - Certificate Transparency Monitor - Open Source Edition
**Cert Spotter** is a Certificate Transparency log monitor from SSLMate that
alerts you when an SSL/TLS certificate is issued for one of your domains.
Cert Spotter is easier to use than other open source CT monitors, since it does not require
a database. It's also more robust, since it uses a special certificate parser
that ensures it won't miss certificates.
Cert Spotter is available as a hosted service by SSLMate that also
provides monitoring for expiring and incorrectly-installed certificates,
Slack notifications, and a Web dashboard for tracking all of your
certificates. Visit <https://sslmate.com/certspotter> to sign up.
You can use Cert Spotter to detect:
* Certificates issued to attackers who have compromised your DNS and
are redirecting your visitors to their malicious site.
* Certificates issued to attackers who have taken over an abandoned
sub-domain in order to serve malware under your name.
* Certificates issued to attackers who have compromised a certificate
authority and want to impersonate your site.
* Certificates issued in violation of your corporate policy
or outside of your centralized certificate procurement process.
## Quickstart
The following instructions require you to have [Go version 1.21 or higher](https://go.dev/dl/) installed.
1. Install the certspotter command using the `go` command:
```
go install software.sslmate.com/src/certspotter/cmd/certspotter@latest
```
2. Create a watch list file `$HOME/.certspotter/watchlist` containing the DNS names you want to monitor,
one per line. To monitor an entire domain tree (including the domain itself
and all sub-domains) prefix the domain name with a dot (e.g. `.example.com`).
To monitor a single DNS name only, do not prefix the name with a dot.
3. Place one or more email addresses in the `$HOME/.certspotter/email_recipients`
file (one per line), and/or place one or more executable scripts in the
`$HOME/.certspotter/hooks.d` directory. certspotter will email the listed
addresses (requires your system to have a working sendmail command) and
execute the provided scripts when it detects a certificate for a domain on
your watch list.
4. Configure your system to run `certspotter` as a daemon. You may want to specify
the `-start_at_end` command line option to tell certspotter to start monitoring
new logs at the end instead of the beginning. This saves significant bandwidth, but
you won't be notified about certificates which were logged before you started
using certspotter.
## Documentation
* Command line options and operational details: [certspotter(8) man page](man/certspotter.md)
* The script interface: [certspotter-script(8) man page](man/certspotter-script.md)
* Authorizing known certificates: [certspotter-authorize(8) man page](man/certspotter-authorize.md)
* Monitoring certspotter with Prometheus: [certspotter-metrics(8) man page](man/certspotter-metrics.md)
* [Change Log](CHANGELOG.md)
## Authorizing Known Certificates to Prevent False Alarms
You can use the **certspotter-authorize** command to tell certspotter
about legitimate certificates issued by your certificate authority.
certspotter won't notify you when it discovers an authorized certificate
(or its corresponding precertificate) in Certificate Transparency logs.
To install certspotter-authorize, run:
```
go install software.sslmate.com/src/certspotter/cmd/certspotter-authorize@latest
```
To authorize a certificate, run:
```
certspotter-authorize /path/to/cert.pem
```
For more details, see the [certspotter-authorize(8) man page](man/certspotter-authorize.md).
## What certificates are detected by Cert Spotter?
In the default configuration, any certificate that is logged to a
Certificate Transparency log listed as Qualified, Usable, or ReadOnly
in the Google Chrome or Apple log lists will be detected by Cert Spotter.
By default, Chrome, Safari, Firefox, and Edge only accept certificates that are
logged to at least one such log (Firefox and Edge use the Chrome log list), so any
certificate that works in these browsers will be detected by Cert Spotter.
Cert Spotter monitors both traditional RFC6962 logs and modern static-ct-api logs.
## Security
Cert Spotter assumes an adversarial model in which an attacker produces
a certificate that is accepted by at least some clients but goes
undetected because of an encoding error that prevents CT monitors from
understanding it. To defend against this attack, Cert Spotter uses a
special certificate parser that keeps the certificate unparsed except
for the identifiers. If one of the identifiers matches a domain on your
watchlist, you will be notified, even if other parts of the certificate
are unparsable.
Cert Spotter takes special precautions to ensure identifiers are parsed
correctly, and implements defenses against identifier-based attacks.
For instance, if a DNS identifier contains a null byte, Cert Spotter
interprets it as two identifiers: the complete identifier, and the
identifier formed by truncating at the first null byte. For example, a
certificate for `example.org\0.example.com` will alert the owners of both
`example.org` and `example.com`. This defends against [null prefix attacks](
http://www.thoughtcrime.org/papers/null-prefix-attacks.pdf).
SSLMate continuously monitors CT logs to make sure every certificate's
identifiers can be successfully parsed, and will release updates to
Cert Spotter as necessary to fix parsing failures.
Cert Spotter understands wildcard DNS names, and will alert
you if a wildcard certificate might match an identifier on
your watchlist. For example, a watchlist entry for `sub.example.com` would
match certificates for `*.example.com`.
Cert Spotter is not just a log monitor, but also a log auditor which
checks that the log is obeying its append-only property. A future
release of Cert Spotter will support gossiping with other log monitors
to ensure the log is presenting a single view.
## Testing
Every day at around 12:05 UTC, a certificate is issued and logged to CT
for the DNS name `DAY-OF-WEEK.test.certspotter.org`. You can add one
or more of these DNS names (or `.test.certspotter.org`) to your watch
list to get confirmation that Cert Spotter is working.
| DNS Name | Issuer | Key Type |
| -------------------------------- | ------------- | -------- |
| `monday.test.certspotter.org` | Sectigo | RSA |
| `tuesday.test.certspotter.org` | Let's Encrypt | ECDSA |
| `wednesday.test.certspotter.org` | Google | RSA |
| `thursday.test.certspotter.org` | Sectigo | ECDSA |
| `friday.test.certspotter.org` | Let's Encrypt | RSA |
| `saturday.test.certspotter.org` | Google | ECDSA |
| `sunday.test.certspotter.org` | Let's Encrypt | ECDSA |
## Monitoring certspotter with Prometheus (Experimental)
You can use the experimental **certspotter-metrics** command to output Prometheus metrics
about certspotter's progress monitoring each CT log, such as how many entries
have been downloaded and verified. certspotter-metrics reads the certspotter
state directory and writes metrics to stdout; run it periodically and expose
the output to Prometheus, e.g. with the node_exporter textfile collector.
To install certspotter-metrics, run:
```
go install software.sslmate.com/src/certspotter/cmd/certspotter-metrics@latest
```
Here's a crontab entry that exposes metrics to the node_exporter textfile
collector every five minutes:
```
*/5 * * * * root certspotter-metrics > /var/lib/node_exporter/certspotter.prom.tmp && mv /var/lib/node_exporter/certspotter.prom.tmp /var/lib/node_exporter/certspotter.prom
```
Here's a Prometheus query to determine download backlog (number entries
in the log that haven't been downloaded yet):
```
certspotter_log_max_sth_size-certspotter_log_download_position
```
certspotter-metrics is experimental. Metrics may change in future
versions, and a future version of certspotter might export metrics
directly from a builtin HTTP server rather than using a separate command. Please
provide feedback at <https://github.com/SSLMate/certspotter/issues/79>.
For more details, see the [certspotter-metrics(8) man page](man/certspotter-metrics.md).
## Copyright
Copyright © 2016-2026 Opsmate, Inc.
Licensed under the [Mozilla Public License Version 2.0](LICENSE).
third_party/certspotter/asn1.go
Vendored
+78
View File
@@ -0,0 +1,78 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"bytes"
"encoding/asn1"
"encoding/binary"
"errors"
"unicode/utf8"
)
func stringFromByteSlice(chars []byte) string {
runes := make([]rune, len(chars))
for i, ch := range chars {
runes[i] = rune(ch)
}
return string(runes)
}
func stringFromUint16Slice(chars []uint16) string {
runes := make([]rune, len(chars))
for i, ch := range chars {
runes[i] = rune(ch)
}
return string(runes)
}
func stringFromUint32Slice(chars []uint32) string {
runes := make([]rune, len(chars))
for i, ch := range chars {
runes[i] = rune(ch)
}
return string(runes)
}
func decodeASN1String(value *asn1.RawValue) (string, error) {
if !value.IsCompound && value.Class == 0 {
if value.Tag == 12 {
// UTF8String
if !utf8.Valid(value.Bytes) {
return "", errors.New("malformed UTF8String")
}
return string(value.Bytes), nil
} else if value.Tag == 19 || value.Tag == 22 || value.Tag == 20 || value.Tag == 26 {
// * PrintableString - subset of ASCII
// * IA5String - ASCII
// * TeletexString - 8 bit charset; not quite ISO-8859-1, but often treated as such
// * VisibleString - subset of ASCII
// Don't enforce character set rules. Allow any 8 bit character, since
// CAs routinely mess this up
return stringFromByteSlice(value.Bytes), nil
} else if value.Tag == 30 {
// BMPString - Unicode, encoded in big-endian format using two octets
runes := make([]uint16, len(value.Bytes)/2)
if err := binary.Read(bytes.NewReader(value.Bytes), binary.BigEndian, runes); err != nil {
return "", errors.New("Malformed BMPString: " + err.Error())
}
return stringFromUint16Slice(runes), nil
} else if value.Tag == 28 {
// UniversalString - Unicode, encoded in big-endian format using four octets
runes := make([]uint32, len(value.Bytes)/4)
if err := binary.Read(bytes.NewReader(value.Bytes), binary.BigEndian, runes); err != nil {
return "", errors.New("Malformed UniversalString: " + err.Error())
}
return stringFromUint32Slice(runes), nil
}
}
return "", errors.New("not a string")
}
third_party/certspotter/asn1time.go
Vendored
+257
View File
@@ -0,0 +1,257 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"encoding/asn1"
"errors"
"strconv"
"time"
"unicode"
)
const (
tagUTCTime = 23
tagGeneralizedTime = 24
)
func isDigit(b byte) bool {
return unicode.IsDigit(rune(b))
}
func bytesToInt(bytes []byte) (int, error) {
return strconv.Atoi(string(bytes))
}
func parseUTCTime(bytes []byte) (time.Time, error) {
var err error
var year, month, day int
var hour, min, sec int
var tz *time.Location
// YYMMDDhhmm
if len(bytes) < 10 {
return time.Time{}, errors.New("UTCTime is too short")
}
year, err = bytesToInt(bytes[0:2])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
month, err = bytesToInt(bytes[2:4])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
day, err = bytesToInt(bytes[4:6])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
hour, err = bytesToInt(bytes[6:8])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
min, err = bytesToInt(bytes[8:10])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
bytes = bytes[10:]
// (optional) ss
if len(bytes) >= 2 && isDigit(bytes[0]) {
sec, err = bytesToInt(bytes[0:2])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
bytes = bytes[2:]
}
// timezone (required but allow it to be omitted, since this is a common error)
if len(bytes) >= 1 {
if bytes[0] == 'Z' {
tz = time.UTC
bytes = bytes[1:]
} else if bytes[0] == '+' {
// +hhmm
if len(bytes) < 5 {
return time.Time{}, errors.New("UTCTime positive timezone offset is too short")
}
tzHour, err := bytesToInt(bytes[1:3])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
tzMin, err := bytesToInt(bytes[3:5])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
tz = time.FixedZone("", tzHour*3600+tzMin*60)
bytes = bytes[5:]
} else if bytes[0] == '-' {
// -hhmm
if len(bytes) < 5 {
return time.Time{}, errors.New("UTCTime negative timezone offset is too short")
}
tzHour, err := bytesToInt(bytes[1:3])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
tzMin, err := bytesToInt(bytes[3:5])
if err != nil {
return time.Time{}, errors.New("UTCTime contains invalid integer: " + err.Error())
}
tz = time.FixedZone("", -1*(tzHour*3600+tzMin*60))
bytes = bytes[5:]
}
} else {
tz = time.UTC
}
if len(bytes) > 0 {
return time.Time{}, errors.New("UTCTime has trailing garbage")
}
// https://tools.ietf.org/html/rfc5280#section-4.1.2.5.1
if year >= 50 {
year = 1900 + year
} else {
year = 2000 + year
}
return time.Date(year, time.Month(month), day, hour, min, sec, 0, tz), nil
}
func parseGeneralizedTime(bytes []byte) (time.Time, error) {
var err error
var year, month, day int
var hour, min, sec, ms int
var tz *time.Location
// YYYYMMDDHH
if len(bytes) < 10 {
return time.Time{}, errors.New("GeneralizedTime is too short")
}
year, err = bytesToInt(bytes[0:4])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
month, err = bytesToInt(bytes[4:6])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
day, err = bytesToInt(bytes[6:8])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
hour, err = bytesToInt(bytes[8:10])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
bytes = bytes[10:]
// (optional) MM
if len(bytes) >= 2 && isDigit(bytes[0]) {
min, err = bytesToInt(bytes[0:2])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
bytes = bytes[2:]
// (optional) SS
if len(bytes) >= 2 && isDigit(bytes[0]) {
sec, err = bytesToInt(bytes[0:2])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
bytes = bytes[2:]
// (optional) .fff
if len(bytes) >= 1 && bytes[0] == '.' {
if len(bytes) < 4 {
return time.Time{}, errors.New("GeneralizedTime fractional seconds is too short")
}
ms, err = bytesToInt(bytes[1:4])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
bytes = bytes[4:]
}
}
}
// timezone (Z or +hhmm or -hhmm or nothing)
if len(bytes) >= 1 {
if bytes[0] == 'Z' {
bytes = bytes[1:]
tz = time.UTC
} else if bytes[0] == '+' {
// +hhmm
if len(bytes) < 5 {
return time.Time{}, errors.New("GeneralizedTime positive timezone offset is too short")
}
tzHour, err := bytesToInt(bytes[1:3])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
tzMin, err := bytesToInt(bytes[3:5])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
tz = time.FixedZone("", tzHour*3600+tzMin*60)
bytes = bytes[5:]
} else if bytes[0] == '-' {
// -hhmm
if len(bytes) < 5 {
return time.Time{}, errors.New("GeneralizedTime negative timezone offset is too short")
}
tzHour, err := bytesToInt(bytes[1:3])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
tzMin, err := bytesToInt(bytes[3:5])
if err != nil {
return time.Time{}, errors.New("GeneralizedTime contains invalid integer: " + err.Error())
}
tz = time.FixedZone("", -1*(tzHour*3600+tzMin*60))
bytes = bytes[5:]
}
} else {
tz = time.UTC
}
if len(bytes) > 0 {
return time.Time{}, errors.New("GeneralizedTime has trailing garbage")
}
return time.Date(year, time.Month(month), day, hour, min, sec, ms*1000*1000, tz), nil
}
func decodeASN1Time(value *asn1.RawValue) (time.Time, error) {
if !value.IsCompound && value.Class == 0 {
if value.Tag == tagUTCTime {
return parseUTCTime(value.Bytes)
} else if value.Tag == tagGeneralizedTime {
return parseGeneralizedTime(value.Bytes)
}
}
return time.Time{}, errors.New("not a time value")
}
third_party/certspotter/asn1time_test.go
Vendored
+121
View File
@@ -0,0 +1,121 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"testing"
"time"
)
type timeTest struct {
in string
ok bool
out time.Time
}
var utcTimeTests = []timeTest{
{"9502101525Z", true, time.Date(1995, time.February, 10, 15, 25, 0, 0, time.UTC)},
{"950210152542Z", true, time.Date(1995, time.February, 10, 15, 25, 42, 0, time.UTC)},
{"1502101525Z", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.UTC)},
{"150210152542Z", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.UTC)},
{"1502101525+1000", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.FixedZone("", 10*3600))},
{"1502101525-1000", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.FixedZone("", -1*(10*3600)))},
{"1502101525+1035", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.FixedZone("", 10*3600+35*60))},
{"1502101525-1035", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.FixedZone("", -1*(10*3600+35*60)))},
{"150210152542+1000", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.FixedZone("", 10*3600))},
{"150210152542-1000", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.FixedZone("", -1*(10*3600)))},
{"150210152542+1035", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.FixedZone("", 10*3600+35*60))},
{"150210152542-1035", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.FixedZone("", -1*(10*3600+35*60)))},
{"1502101525", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.UTC)},
{"150210152542", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.UTC)},
{"", false, time.Time{}},
{"123", false, time.Time{}},
{"150210152542-10", false, time.Time{}},
{"150210152542F", false, time.Time{}},
{"150210152542ZF", false, time.Time{}},
}
func TestUTCTime(t *testing.T) {
for i, test := range utcTimeTests {
ret, err := parseUTCTime([]byte(test.in))
if err != nil {
if test.ok {
t.Errorf("#%d: parseUTCTime(%q) failed with error %v", i, test.in, err)
}
continue
}
if !test.ok {
t.Errorf("#%d: parseUTCTime(%q) succeeded, should have failed", i, test.in)
continue
}
if !test.out.Equal(ret) {
t.Errorf("#%d: parseUTCTime(%q) = %v, want %v", i, test.in, ret, test.out)
}
}
}
var generalizedTimeTests = []timeTest{
{"2015021015", true, time.Date(2015, time.February, 10, 15, 0, 0, 0, time.UTC)},
{"201502101525", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.UTC)},
{"20150210152542", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.UTC)},
{"20150210152542.123", true, time.Date(2015, time.February, 10, 15, 25, 42, 123000000, time.UTC)},
{"20150210152542.12", false, time.Time{}},
{"20150210152542.1", false, time.Time{}},
{"20150210152542.", false, time.Time{}},
{"2015021015Z", true, time.Date(2015, time.February, 10, 15, 0, 0, 0, time.UTC)},
{"201502101525Z", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.UTC)},
{"20150210152542Z", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.UTC)},
{"20150210152542.123Z", true, time.Date(2015, time.February, 10, 15, 25, 42, 123000000, time.UTC)},
{"20150210152542.12Z", false, time.Time{}},
{"20150210152542.1Z", false, time.Time{}},
{"20150210152542.Z", false, time.Time{}},
{"2015021015+1000", true, time.Date(2015, time.February, 10, 15, 0, 0, 0, time.FixedZone("", 10*3600))},
{"201502101525+1000", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.FixedZone("", 10*3600))},
{"20150210152542+1000", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.FixedZone("", 10*3600))},
{"20150210152542.123+1000", true, time.Date(2015, time.February, 10, 15, 25, 42, 123000000, time.FixedZone("", 10*3600))},
{"20150210152542.12+1000", false, time.Time{}},
{"20150210152542.1+1000", false, time.Time{}},
{"20150210152542.+1000", false, time.Time{}},
{"2015021015-0835", true, time.Date(2015, time.February, 10, 15, 0, 0, 0, time.FixedZone("", -1*(8*3600+35*60)))},
{"201502101525-0835", true, time.Date(2015, time.February, 10, 15, 25, 0, 0, time.FixedZone("", -1*(8*3600+35*60)))},
{"20150210152542-0835", true, time.Date(2015, time.February, 10, 15, 25, 42, 0, time.FixedZone("", -1*(8*3600+35*60)))},
{"20150210152542.123-0835", true, time.Date(2015, time.February, 10, 15, 25, 42, 123000000, time.FixedZone("", -1*(8*3600+35*60)))},
{"20150210152542.12-0835", false, time.Time{}},
{"20150210152542.1-0835", false, time.Time{}},
{"20150210152542.-0835", false, time.Time{}},
{"", false, time.Time{}},
{"123", false, time.Time{}},
{"2015021015+1000Z", false, time.Time{}},
{"2015021015x", false, time.Time{}},
{"201502101525Zf", false, time.Time{}},
}
func TestGeneralizedTime(t *testing.T) {
for i, test := range generalizedTimeTests {
ret, err := parseGeneralizedTime([]byte(test.in))
if err != nil {
if test.ok {
t.Errorf("#%d: parseGeneralizedTime(%q) failed with error %v", i, test.in, err)
}
continue
}
if !test.ok {
t.Errorf("#%d: parseGeneralizedTime(%q) succeeded, should have failed", i, test.in)
continue
}
if !test.out.Equal(ret) {
t.Errorf("#%d: parseGeneralizedTime(%q) = %v, want %v", i, test.in, ret, test.out)
}
}
}
third_party/certspotter/canonicalize.go
Vendored
+92
View File
@@ -0,0 +1,92 @@
// Copyright (C) 2019 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"encoding/asn1"
)
func canonicalizeRDNString(fromStr string) string {
from := []byte(fromStr)
to := []byte{}
inWhitespace := true
for _, ch := range from {
if ch == ' ' || ch == '\f' || ch == '\n' || ch == '\r' || ch == '\t' || ch == '\v' {
if !inWhitespace {
to = append(to, ' ')
}
inWhitespace = true
} else {
if ch >= 'A' && ch <= 'Z' {
to = append(to, ch+32) // convert to lowercase
} else {
to = append(to, ch)
}
inWhitespace = false
}
}
if inWhitespace && len(to) > 0 {
// whack off the space character that we appended
to = to[:len(to)-1]
}
return string(to)
}
func shouldCanonicalizeASN1String(value *asn1.RawValue) bool {
if !value.IsCompound && value.Class == 0 {
return value.Tag == 12 || value.Tag == 19 || value.Tag == 22 || value.Tag == 20 || value.Tag == 26 || value.Tag == 30 || value.Tag == 28
}
return false
}
func canonicalizeATV(oldATV AttributeTypeAndValue) (AttributeTypeAndValue, error) {
if shouldCanonicalizeASN1String(&oldATV.Value) {
str, err := decodeASN1String(&oldATV.Value)
if err != nil {
return AttributeTypeAndValue{}, err
}
str = canonicalizeRDNString(str)
return AttributeTypeAndValue{
Type: oldATV.Type,
Value: asn1.RawValue{
Class: 0,
Tag: asn1.TagUTF8String,
IsCompound: false,
Bytes: []byte(str),
},
}, nil
} else {
return oldATV, nil
}
}
func canonicalizeRDNSet(oldSet RelativeDistinguishedNameSET) (RelativeDistinguishedNameSET, error) {
newSet := make([]AttributeTypeAndValue, len(oldSet))
for i := range oldSet {
var err error
newSet[i], err = canonicalizeATV(oldSet[i])
if err != nil {
return nil, err
}
}
return newSet, nil
}
func CanonicalizeRDNSequence(oldSequence RDNSequence) (RDNSequence, error) {
newSequence := make([]RelativeDistinguishedNameSET, len(oldSequence))
for i := range oldSequence {
var err error
newSequence[i], err = canonicalizeRDNSet(oldSequence[i])
if err != nil {
return nil, err
}
}
return newSequence, nil
}
third_party/certspotter/canonicalize_test.go
Vendored
+49
View File
@@ -0,0 +1,49 @@
// Copyright (C) 2019 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"testing"
)
type stringCanonTest struct {
in string
out string
}
var stringCanonTests = []stringCanonTest{
{"", ""},
{" ", ""},
{" ", ""},
{"abc", "abc"},
{"aBc", "abc"},
{"ab c", "ab c"},
{"ab c", "ab c"},
{"ab\n c", "ab c"},
{" ab c ", "ab c"},
{" ab c ", "ab c"},
{" ab c", "ab c"},
{"ab c ", "ab c"},
{"abc ", "abc"},
{"abc ", "abc"},
{" abc ", "abc"},
{" abc ", "abc"},
{" abc", "abc"},
{" aBc de f g\n", "abc de f g"},
}
func TestCanonicalizeRDNString(t *testing.T) {
for i, test := range stringCanonTests {
ret := canonicalizeRDNString(test.in)
if test.out != ret {
t.Errorf("#%d: canonicalizeRDNString(%q) = %q, want %q", i, test.in, ret, test.out)
}
}
}
third_party/certspotter/cmd/certspotter-authorize/.gitignore
+1
View File
@@ -0,0 +1 @@
/certspotter-authorize
third_party/certspotter/cmd/certspotter-authorize/main.go
Vendored
+202
View File
@@ -0,0 +1,202 @@
// Copyright (C) 2026 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package main
import (
"crypto/sha256"
"encoding/hex"
"encoding/pem"
"errors"
"flag"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"runtime/debug"
"strconv"
"strings"
"software.sslmate.com/src/certspotter"
)
var programName = os.Args[0]
var Version = "unknown"
var Source = "unknown"
func certspotterVersion() (string, string) {
if buildinfo, ok := debug.ReadBuildInfo(); ok && strings.HasPrefix(buildinfo.Main.Version, "v") {
return strings.TrimPrefix(buildinfo.Main.Version, "v"), buildinfo.Main.Path
} else {
return Version, Source
}
}
func homedir() string {
homedir, err := os.UserHomeDir()
if err != nil {
panic(fmt.Errorf("unable to determine home directory: %w", err))
}
return homedir
}
func startedBySupervisor() bool {
return os.Getenv("SYSTEMD_EXEC_PID") == strconv.Itoa(os.Getpid())
}
func defaultStateDir() string {
if envVar := os.Getenv("CERTSPOTTER_STATE_DIR"); envVar != "" {
return envVar
} else if envVar := os.Getenv("STATE_DIRECTORY"); envVar != "" && startedBySupervisor() {
return envVar
} else {
return filepath.Join(homedir(), ".certspotter")
}
}
func fileExists(filename string) bool {
_, err := os.Lstat(filename)
return err == nil
}
func readCertFile(path string) ([]byte, error) {
if path == "-" {
return io.ReadAll(os.Stdin)
} else {
return os.ReadFile(path)
}
}
func parseCertificate(certBytes []byte) ([]byte, error) {
block, _ := pem.Decode(certBytes)
if block != nil {
if block.Type == "CERTIFICATE" {
return block.Bytes, nil
}
return nil, fmt.Errorf("PEM block type is %q, expected CERTIFICATE", block.Type)
}
return nil, fmt.Errorf("no PEM data found")
}
func computeTBSHash(certDER []byte) ([32]byte, error) {
certInfo, err := certspotter.MakeCertInfoFromRawCert(certDER)
if err != nil {
return [32]byte{}, fmt.Errorf("error parsing certificate: %w", err)
}
precertTBS, err := certspotter.ReconstructPrecertTBS(certInfo.TBS)
if err != nil {
return [32]byte{}, fmt.Errorf("error reconstructing precertificate TBSCertificate: %w", err)
}
return sha256.Sum256(precertTBS.Raw), nil
}
func createNotifiedMarker(stateDir string, tbsHash [32]byte) (string, error) {
tbsHex := hex.EncodeToString(tbsHash[:])
certsDir := filepath.Join(stateDir, "certs")
tbsDir := filepath.Join(certsDir, tbsHex[0:2])
notifiedPath := filepath.Join(tbsDir, "."+tbsHex+".notified")
// Check if already notified
if fileExists(notifiedPath) {
return notifiedPath, nil
}
// Create certs directory if needed
if err := os.Mkdir(certsDir, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return "", fmt.Errorf("error creating certs directory: %w", err)
}
// Create TBS-specific subdirectory if needed
if err := os.Mkdir(tbsDir, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return "", fmt.Errorf("error creating directory: %w", err)
}
// Create marker file
if err := os.WriteFile(notifiedPath, nil, 0666); err != nil {
return "", fmt.Errorf("error creating marker file: %w", err)
}
return notifiedPath, nil
}
func main() {
version, source := certspotterVersion()
var flags struct {
stateDir string
printhash bool
version bool
}
flag.StringVar(&flags.stateDir, "state_dir", defaultStateDir(), "State directory used by certspotter")
flag.BoolVar(&flags.printhash, "printhash", false, "Instead of authorizing certificate, print its TBS hash")
flag.BoolVar(&flags.version, "version", false, "Print version and exit")
flag.Parse()
if flags.version {
fmt.Fprintf(os.Stdout, "certspotter-authorize version %s (%s)\n", version, source)
os.Exit(0)
}
args := flag.Args()
if len(args) == 0 {
fmt.Fprintf(os.Stderr, "Usage: %s [options] -|CERTFILE...\n", programName)
fmt.Fprintf(os.Stderr, "Purpose: suppress future certspotter notifications for a certificate and its corresponding precertificate.\n")
fmt.Fprintf(os.Stderr, "Options:\n")
flag.PrintDefaults()
os.Exit(2)
}
if len(args) > 1 {
for _, arg := range args {
if arg == "-" {
fmt.Fprintf(os.Stderr, "%s: '-' must be the only argument when used\n", programName)
os.Exit(2)
}
}
}
for _, certPath := range args {
certBytes, err := readCertFile(certPath)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: error reading certificate: %s\n", programName, err)
os.Exit(1)
}
certDER, err := parseCertificate(certBytes)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: %s: %s\n", programName, certPath, err)
os.Exit(1)
}
tbsHash, err := computeTBSHash(certDER)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: %s: %s\n", programName, certPath, err)
os.Exit(1)
}
if flags.printhash {
if certPath == "-" {
fmt.Println(hex.EncodeToString(tbsHash[:]))
} else {
fmt.Printf("%s %s\n", hex.EncodeToString(tbsHash[:]), certPath)
}
} else {
_, err = createNotifiedMarker(flags.stateDir, tbsHash)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: %s\n", programName, err)
os.Exit(1)
}
}
}
os.Exit(0)
}
third_party/certspotter/cmd/certspotter-authorize/main_test.go
Vendored
+190
View File
@@ -0,0 +1,190 @@
// Copyright (C) 2026 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package main
import (
"bytes"
"encoding/hex"
"os"
"path/filepath"
"testing"
)
const testCertPEM = `
-----BEGIN CERTIFICATE-----
MIIGuzCCBSOgAwIBAgIRANubk4g/6c+TF8jITzhFX44wDQYJKoZIhvcNAQELBQAw
YDELMAkGA1UEBhMCR0IxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBQdWJsaWMgU2VydmVyIEF1dGhlbnRpY2F0aW9uIENBIERWIFIz
NjAeFw0yNTEyMDIwMDAwMDBaFw0yNjExMjEyMzU5NTlaMBYxFDASBgNVBAMTC3Nz
bG1hdGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA235/3Y/E
4yPAHPa37C7Fgp7KPVjjuTB5vKV9nYIJzfp7NgvDBlf7k5bZFCSsSIj2txhL0hzX
Bwvmy7u7CYR7CApr2Rx2UPOl7Gmlt/DmtfyKac8Iunn2ozuGZDtxq19Go4NL9jl9
e9O3H/lcL/ZFqzbUNlKIOfkOYkOxM3qpQXHTXuhkeI2MJO/S4wX8y8/8uhArWQ9e
h/YrtJlO9fla60kLUlQF7mtJTc+0oB3+N4eF5t2a8Pav00T6lVvH8hMhbY0nZ/tB
CD6/I6yelh8cP094VRJEGWs+zcEuXpz4FsZggkhF/l+AhQ+DfgxZhno4M60kBKC8
Un1BTGX5TjfjJQIDAQABo4IDODCCAzQwHwYDVR0jBBgwFoAUaMASFhgOr872h6Yy
V6NGUV3LBycwHQYDVR0OBBYEFKg2kl8xIzdSxjOEAzlNpdpigAazMA4GA1UdDwEB
/wQEAwIFoDAMBgNVHRMBAf8EAjAAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMEkGA1Ud
IARCMEAwNAYLKwYBBAGyMQECAgcwJTAjBggrBgEFBQcCARYXaHR0cHM6Ly9zZWN0
aWdvLmNvbS9DUFMwCAYGZ4EMAQIBMIGEBggrBgEFBQcBAQR4MHYwTwYIKwYBBQUH
MAKGQ2h0dHA6Ly9jcnQuc2VjdGlnby5jb20vU2VjdGlnb1B1YmxpY1NlcnZlckF1
dGhlbnRpY2F0aW9uQ0FEVlIzNi5jcnQwIwYIKwYBBQUHMAGGF2h0dHA6Ly9vY3Nw
LnNlY3RpZ28uY29tMIIBfQYKKwYBBAHWeQIEAgSCAW0EggFpAWcAdQDXbX0Q0af1
d8LH6V/XAL/5gskzWmXh0LMBcxfAyMVpdwAAAZrg2UujAAAEAwBGMEQCIApVmNqo
gzJBNbcVXezO5sSvOFE5FaVZVz/eaqnCbG+2AiAT/A7XPtOYHwsE0wmTUBCTV/0l
bF7lk573b3rNtBvP3QB2AK9niDtXsE7dj6bZfvYuqOuBCsdxYPAkXlXWDC/nhYc6
AAABmuDZTFoAAAQDAEcwRQIgODruJKtbjW1QJcQP7ARZAw5FgChfI599pJBA2bbQ
suICIQCskIUnzQCD6taycnQCN3zpu+rsz3Vd4AsMeFDM/cDJ5AB2AKyrMHBs6+yE
MfQT0vSRXxEeQiRDsfKmjE88KzunHgLDAAABmuDZS5kAAAQDAEcwRQIgD9IQCPTc
N88jbz5DUILwmDruTo411Ep5M2ZryNjBkywCIQCFwIyqGZEd+PiFv4l+5LOV3yDW
/zUuimFUoAJH5OIiNDBsBgNVHREEZTBjggtzc2xtYXRlLmNvbYIbKi5odHRwLWFw
cHJvdmFsLnNzbG1hdGUuY29tghFjZXJ0cy5zc2xtYXRlLmNvbYITY29uc29sZS5z
c2xtYXRlLmNvbYIPd3d3LnNzbG1hdGUuY29tMA0GCSqGSIb3DQEBCwUAA4IBgQBR
Pjx14qo9PiYYEE1695CHdctA6up8L+n0MRapZcxALN/cetfGeoR00ZEH+7b1X7Ma
F9GGv1OtJXDoCySlAsdwFKHYtKhrUYRuQXdKGkTjdMzKO/+5kXeZIqgsCR10j8nr
Zq0Zcg2ply4j03/0y7+8ZNC1Erp4DB1Tq7ybgXnyURaNQTHsSkDoxMT/bWIrhGD0
C8kN/ExkFvOBQlzdbuwo2d3v0zSM4mYmnqUhUYHprZllOziYgxIqjM/7mfnDkVAi
ov8yNJtn6EPt1wt6Oo3fC+Ft1T/kbSxeZbqWf3Zgbon5ijmNz+xqkb8br2+JdzM+
8gEIqO6mNoMl0tayzb4a5KDaHxhczMGB3ggBwpVcdLtYBBa41thrgRP0VARqFTFG
IIkC9gPMjScf+uv9CQPsNk3kFI8vN4T3x4/g54N8Mc3M4JxvLaOsBj8dMeyq7v2p
1zE9WRngMUWuPgx0O94c0Pteumg/+pSGVeRapIuYZxXvkmLJ5wmwgYepix+cw1w=
-----END CERTIFICATE-----
`
func TestComputeTBSHash(t *testing.T) {
certDER, err := parseCertificate([]byte(testCertPEM))
if err != nil {
t.Fatalf("parseCertificate failed: %v", err)
}
tbsHash, err := computeTBSHash(certDER)
if err != nil {
t.Fatalf("computeTBSHash failed: %v", err)
}
if expected := [...]byte{0x3c, 0xf6, 0xb2, 0x44, 0xc2, 0x95, 0x85, 0xdb, 0xfb, 0xfd, 0x42, 0x0a, 0x6a, 0x4c, 0x62, 0xf7, 0x96, 0x8f, 0xa9, 0x05, 0xb4, 0xd6, 0xa4, 0xf5, 0x9d, 0x4d, 0x3b, 0xc9, 0xfa, 0xcb, 0x0c, 0xc8}; expected != tbsHash {
t.Fatalf("computeTBSHash returned %x; expected %x", tbsHash, expected)
}
}
func TestCreateNotifiedMarker(t *testing.T) {
stateDir := t.TempDir()
certDER, err := parseCertificate([]byte(testCertPEM))
if err != nil {
t.Fatalf("parseCertificate failed: %v", err)
}
tbsHash, err := computeTBSHash(certDER)
if err != nil {
t.Fatalf("computeTBSHash failed: %v", err)
}
// First call should create the marker
notifiedPath, err := createNotifiedMarker(stateDir, tbsHash)
if err != nil {
t.Fatalf("createNotifiedMarker failed: %v", err)
}
// Verify marker file exists
if !fileExists(notifiedPath) {
t.Fatalf("marker file does not exist: %s", notifiedPath)
}
// Verify path structure is correct
tbsHex := hex.EncodeToString(tbsHash[:])
expectedPath := filepath.Join(stateDir, "certs", tbsHex[0:2], "."+tbsHex+".notified")
if notifiedPath != expectedPath {
t.Fatalf("unexpected marker path: got %s, expected %s", notifiedPath, expectedPath)
}
// Second call should succeed (idempotency)
notifiedPath2, err := createNotifiedMarker(stateDir, tbsHash)
if err != nil {
t.Fatalf("createNotifiedMarker second call failed: %v", err)
}
if notifiedPath != notifiedPath2 {
t.Fatalf("second call returned different path: got %s, expected %s", notifiedPath2, notifiedPath)
}
}
func TestReadCertFile(t *testing.T) {
// Test reading from a file
tmpDir := t.TempDir()
certPath := filepath.Join(tmpDir, "cert.pem")
if err := os.WriteFile(certPath, []byte(testCertPEM), 0644); err != nil {
t.Fatalf("failed to write test cert: %v", err)
}
certBytes, err := readCertFile(certPath)
if err != nil {
t.Fatalf("readCertFile failed: %v", err)
}
if !bytes.Equal(certBytes, []byte(testCertPEM)) {
t.Fatal("readCertFile returned different content")
}
}
func TestFileExists(t *testing.T) {
tmpDir := t.TempDir()
// Test with non-existent file
if fileExists(filepath.Join(tmpDir, "nonexistent")) {
t.Fatal("fileExists returned true for non-existent file")
}
// Test with existing file
existingFile := filepath.Join(tmpDir, "existing")
if err := os.WriteFile(existingFile, []byte("test"), 0644); err != nil {
t.Fatalf("failed to write test file: %v", err)
}
if !fileExists(existingFile) {
t.Fatal("fileExists returned false for existing file")
}
}
func TestEndToEnd(t *testing.T) {
stateDir := t.TempDir()
certDER, err := parseCertificate([]byte(testCertPEM))
if err != nil {
t.Fatalf("parseCertificate failed: %v", err)
}
tbsHash, err := computeTBSHash(certDER)
if err != nil {
t.Fatalf("computeTBSHash failed: %v", err)
}
notifiedPath, err := createNotifiedMarker(stateDir, tbsHash)
if err != nil {
t.Fatalf("createNotifiedMarker failed: %v", err)
}
// Verify the marker file structure matches what monitor/fsstate.go expects
tbsHex := hex.EncodeToString(tbsHash[:])
expectedDir := filepath.Join(stateDir, "certs", tbsHex[0:2])
expectedFile := filepath.Join(expectedDir, "."+tbsHex+".notified")
if notifiedPath != expectedFile {
t.Fatalf("unexpected marker path: got %s, expected %s", notifiedPath, expectedFile)
}
if !fileExists(expectedFile) {
t.Fatalf("marker file does not exist: %s", expectedFile)
}
// Verify file is empty (as expected by certspotter)
stat, err := os.Stat(expectedFile)
if err != nil {
t.Fatalf("failed to stat marker file: %v", err)
}
if stat.Size() != 0 {
t.Fatalf("marker file should be empty, but has size %d", stat.Size())
}
}
third_party/certspotter/cmd/certspotter-metrics/.gitignore
+1
View File
@@ -0,0 +1 @@
/certspotter-metrics
third_party/certspotter/cmd/certspotter-metrics/main.go
Vendored
+245
View File
@@ -0,0 +1,245 @@
// Copyright (C) 2026 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package main
import (
"bufio"
"context"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"runtime/debug"
"slices"
"strconv"
"strings"
"software.sslmate.com/src/certspotter/monitor"
)
var programName = os.Args[0]
var Version = "unknown"
var Source = "unknown"
func certspotterVersion() (string, string) {
if buildinfo, ok := debug.ReadBuildInfo(); ok && strings.HasPrefix(buildinfo.Main.Version, "v") {
return strings.TrimPrefix(buildinfo.Main.Version, "v"), buildinfo.Main.Path
} else {
return Version, Source
}
}
func homedir() string {
homedir, err := os.UserHomeDir()
if err != nil {
panic(fmt.Errorf("unable to determine home directory: %w", err))
}
return homedir
}
func startedBySupervisor() bool {
return os.Getenv("SYSTEMD_EXEC_PID") == strconv.Itoa(os.Getpid())
}
func defaultStateDir() string {
if envVar := os.Getenv("CERTSPOTTER_STATE_DIR"); envVar != "" {
return envVar
} else if envVar := os.Getenv("STATE_DIRECTORY"); envVar != "" && startedBySupervisor() {
return envVar
} else {
return filepath.Join(homedir(), ".certspotter")
}
}
// logMetrics is the set of metric values for a single log.
type logMetrics struct {
logID monitor.LogID
maxSTHExists bool
maxSTHSize uint64
maxSTHTimestamp uint64
downloadPosition uint64
verifiedPosition uint64
verifiedSTHExists bool
verifiedSTHTimestamp uint64
unverifiedSTHs uint64
malformedEntries uint64
healthCheckFailures uint64
}
// metricFamily describes one of the metrics output by certspotter-metrics.
// All metrics are gauges labeled with the ID of the log they pertain to.
type metricFamily struct {
name string
help string
value func(*logMetrics) (uint64, bool)
}
var metricFamilies = []metricFamily{
{
name: "certspotter_log_max_sth_size",
help: "Tree size of largest known STH.",
value: func(m *logMetrics) (uint64, bool) { return m.maxSTHSize, m.maxSTHExists },
},
{
name: "certspotter_log_max_sth_timestamp",
help: "Timestamp (in milliseconds since the Epoch) of largest known STH.",
value: func(m *logMetrics) (uint64, bool) { return m.maxSTHTimestamp, m.maxSTHExists },
},
{
name: "certspotter_log_download_position",
help: "Number of entries downloaded from the log, including entries not yet verified.",
value: func(m *logMetrics) (uint64, bool) { return m.downloadPosition, true },
},
{
name: "certspotter_log_verified_position",
help: "Number of entries downloaded from the log and verified against an STH.",
value: func(m *logMetrics) (uint64, bool) { return m.verifiedPosition, true },
},
{
name: "certspotter_log_verified_sth_timestamp",
help: "Timestamp (in milliseconds since the Epoch) of the largest verified STH (the STH whose size is verified_position).",
value: func(m *logMetrics) (uint64, bool) { return m.verifiedSTHTimestamp, m.verifiedSTHExists },
},
{
name: "certspotter_log_unverified_sths",
help: "Number of STHs pending verification.",
value: func(m *logMetrics) (uint64, bool) { return m.unverifiedSTHs, true },
},
{
name: "certspotter_log_malformed_entries",
help: "Number of malformed entries recorded for the log.",
value: func(m *logMetrics) (uint64, bool) { return m.malformedEntries, true },
},
{
name: "certspotter_log_health_check_failures",
help: "Number of health check failures recorded for the log.",
value: func(m *logMetrics) (uint64, bool) { return m.healthCheckFailures, true },
},
}
// gatherMetrics computes the metrics for every log that has a state directory.
// The returned slice is sorted by log ID.
func gatherMetrics(ctx context.Context, state *monitor.FilesystemState) ([]logMetrics, error) {
logIDs, err := state.ListLogs(ctx)
if err != nil {
return nil, fmt.Errorf("error listing logs in state directory: %w", err)
}
slices.SortFunc(logIDs, monitor.LogID.Compare)
allMetrics := make([]logMetrics, 0, len(logIDs))
for _, logID := range logIDs {
m := logMetrics{logID: logID}
logState, err := state.LoadLogState(ctx, logID)
if err != nil {
return nil, fmt.Errorf("error loading state of log %s: %w", logID.Base64String(), err)
}
largestSTH, err := state.LoadLargestSTH(ctx, logID)
if err != nil {
return nil, fmt.Errorf("error loading STHs of log %s: %w", logID.Base64String(), err)
}
if largestSTH != nil {
m.maxSTHExists = true
m.maxSTHSize = largestSTH.TreeSize
m.maxSTHTimestamp = largestSTH.Timestamp
} else if logState != nil && logState.VerifiedSTH != nil {
m.maxSTHExists = true
m.maxSTHSize = logState.VerifiedSTH.TreeSize
m.maxSTHTimestamp = logState.VerifiedSTH.Timestamp
}
if logState != nil {
if logState.DownloadPosition != nil {
m.downloadPosition = logState.DownloadPosition.Size()
}
if logState.VerifiedPosition != nil {
m.verifiedPosition = logState.VerifiedPosition.Size()
}
if logState.VerifiedSTH != nil {
m.verifiedSTHExists = true
m.verifiedSTHTimestamp = logState.VerifiedSTH.Timestamp
}
}
if unverifiedSTHs, err := state.CountSTHs(ctx, logID); err != nil {
return nil, fmt.Errorf("error counting STHs of log %s: %w", logID.Base64String(), err)
} else {
m.unverifiedSTHs = uint64(unverifiedSTHs)
}
if malformedEntries, err := state.CountMalformedEntries(ctx, logID); err != nil {
return nil, fmt.Errorf("error counting malformed entries of log %s: %w", logID.Base64String(), err)
} else {
m.malformedEntries = uint64(malformedEntries)
}
if healthCheckFailures, err := state.CountHealthCheckFailures(ctx, logID); err != nil {
return nil, fmt.Errorf("error counting health check failures of log %s: %w", logID.Base64String(), err)
} else {
m.healthCheckFailures = uint64(healthCheckFailures)
}
allMetrics = append(allMetrics, m)
}
return allMetrics, nil
}
// writeMetrics writes allMetrics to w in the Prometheus text exposition format.
func writeMetrics(w io.Writer, allMetrics []logMetrics) error {
buf := bufio.NewWriter(w)
for _, family := range metricFamilies {
if _, err := fmt.Fprintf(buf, "# HELP %s %s\n", family.name, family.help); err != nil {
return err
}
if _, err := fmt.Fprintf(buf, "# TYPE %s gauge\n", family.name); err != nil {
return err
}
for i := range allMetrics {
value, include := family.value(&allMetrics[i])
if !include {
continue
}
if _, err := fmt.Fprintf(buf, "%s{log_id=\"%s\"} %d\n", family.name, allMetrics[i].logID.Base64String(), value); err != nil {
return err
}
}
}
return buf.Flush()
}
func main() {
version, source := certspotterVersion()
var flags struct {
stateDir string
version bool
}
flag.StringVar(&flags.stateDir, "state_dir", defaultStateDir(), "Directory where certspotter stores state. This should be the same directory used by certspotter(8)")
flag.BoolVar(&flags.version, "version", false, "Print version and exit")
flag.Parse()
if flags.version {
fmt.Fprintf(os.Stdout, "certspotter-metrics version %s (%s)\n", version, source)
os.Exit(0)
}
state := &monitor.FilesystemState{
StateDir: flags.stateDir,
}
allMetrics, err := gatherMetrics(context.Background(), state)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: %s\n", programName, err)
os.Exit(1)
}
if err := writeMetrics(os.Stdout, allMetrics); err != nil {
fmt.Fprintf(os.Stderr, "%s: error writing metrics: %s\n", programName, err)
os.Exit(1)
}
}
third_party/certspotter/cmd/certspotter/.gitignore
+1
View File
@@ -0,0 +1 @@
/certspotter
third_party/certspotter/cmd/certspotter/main.go
Vendored
+290
View File
@@ -0,0 +1,290 @@
// Copyright (C) 2016, 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package main
import (
"bufio"
"context"
"errors"
"flag"
"fmt"
"io/fs"
"os"
"os/signal"
"path/filepath"
"runtime"
"runtime/debug"
"strconv"
"strings"
"syscall"
"time"
"software.sslmate.com/src/certspotter/ctclient"
"software.sslmate.com/src/certspotter/loglist"
"software.sslmate.com/src/certspotter/monitor"
)
var programName = os.Args[0]
var Version = "unknown"
var Source = "unknown"
const defaultLogList = "https://loglist.certspotter.org/monitor.json"
func certspotterVersion() (string, string) {
if buildinfo, ok := debug.ReadBuildInfo(); ok && strings.HasPrefix(buildinfo.Main.Version, "v") {
return strings.TrimPrefix(buildinfo.Main.Version, "v"), buildinfo.Main.Path
} else {
return Version, Source
}
}
func fileExists(filename string) bool {
_, err := os.Lstat(filename)
return err == nil
}
func homedir() string {
homedir, err := os.UserHomeDir()
if err != nil {
panic(fmt.Errorf("unable to determine home directory: %w", err))
}
return homedir
}
func userCacheDir() string {
userCacheDir, err := os.UserCacheDir()
if err != nil {
panic(fmt.Errorf("unable to determine user cache directory: %w", err))
}
return userCacheDir
}
func startedBySupervisor() bool {
return os.Getenv("SYSTEMD_EXEC_PID") == strconv.Itoa(os.Getpid())
}
func defaultStateDir() string {
if envVar := os.Getenv("CERTSPOTTER_STATE_DIR"); envVar != "" {
return envVar
} else if envVar := os.Getenv("STATE_DIRECTORY"); envVar != "" && startedBySupervisor() {
return envVar
} else {
return filepath.Join(homedir(), ".certspotter")
}
}
func defaultConfigDir() string {
if envVar := os.Getenv("CERTSPOTTER_CONFIG_DIR"); envVar != "" {
return envVar
} else if envVar := os.Getenv("CONFIGURATION_DIRECTORY"); envVar != "" && startedBySupervisor() {
return envVar
} else {
return filepath.Join(homedir(), ".certspotter")
}
}
func defaultCacheDir() string {
if envVar := os.Getenv("CERTSPOTTER_CACHE_DIR"); envVar != "" {
return envVar
} else if envVar := os.Getenv("CACHE_DIRECTORY"); envVar != "" && startedBySupervisor() {
return envVar
} else {
return filepath.Join(userCacheDir(), "certspotter")
}
}
func defaultWatchListPath() string {
return filepath.Join(defaultConfigDir(), "watchlist")
}
func defaultWatchListPathIfExists() string {
if fileExists(defaultWatchListPath()) {
return defaultWatchListPath()
} else {
return ""
}
}
func defaultScriptDir() string {
return filepath.Join(defaultConfigDir(), "hooks.d")
}
func defaultEmailFile() string {
return filepath.Join(defaultConfigDir(), "email_recipients")
}
func simplifyError(err error) error {
var pathErr *fs.PathError
if errors.As(err, &pathErr) {
return pathErr.Err
}
return err
}
func readWatchListFile(filename string) (monitor.WatchList, error) {
file, err := os.Open(filename)
if err != nil {
return nil, simplifyError(err)
}
defer file.Close()
return monitor.ReadWatchList(file)
}
func readEmailFile(filename string) ([]string, error) {
file, err := os.Open(filename)
if err != nil {
return nil, simplifyError(err)
}
defer file.Close()
var emails []string
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := scanner.Text()
if line == "" {
continue
}
emails = append(emails, line)
}
return emails, scanner.Err()
}
func appendFunc(slice *[]string) func(string) error {
return func(value string) error {
*slice = append(*slice, value)
return nil
}
}
func main() {
version, source := certspotterVersion()
ctclient.UserAgent = fmt.Sprintf("certspotter/%s (%s; %s; %s; %s; +https://github.com/SSLMate/certspotter)", version, source, runtime.Version(), runtime.GOOS, runtime.GOARCH)
loglist.UserAgent = ctclient.UserAgent
var flags struct {
email []string
healthcheck time.Duration
logs string
noSave bool
script string
startAtEnd bool
stateDir string
stdout bool
verbose bool
version bool
watchlist string
}
flag.Func("email", "Email address to contact when matching certificate is discovered (repeatable)", appendFunc(&flags.email))
flag.DurationVar(&flags.healthcheck, "healthcheck", 24*time.Hour, "How frequently to perform a health check")
flag.StringVar(&flags.logs, "logs", defaultLogList, "File path or URL of JSON list of logs to monitor")
flag.BoolVar(&flags.noSave, "no_save", false, "Do not save a copy of matching certificates in state directory")
flag.StringVar(&flags.script, "script", "", "Program to execute when a matching certificate is discovered")
flag.BoolVar(&flags.startAtEnd, "start_at_end", false, "Start monitoring new logs from the end rather than the beginning (saves considerable bandwidth)")
flag.StringVar(&flags.stateDir, "state_dir", defaultStateDir(), "Directory for storing log position and discovered certificates")
flag.BoolVar(&flags.stdout, "stdout", false, "Write matching certificates to stdout")
flag.BoolVar(&flags.verbose, "verbose", false, "Print detailed information about certspotter's operation to stderr")
flag.BoolVar(&flags.version, "version", false, "Print version and exit")
flag.StringVar(&flags.watchlist, "watchlist", defaultWatchListPathIfExists(), "File containing domain names to watch")
flag.Parse()
if flags.version {
fmt.Fprintf(os.Stdout, "certspotter version %s (%s)\n", version, source)
if flags.verbose {
if buildinfo, ok := debug.ReadBuildInfo(); ok {
fmt.Fprint(os.Stdout, buildinfo)
} else {
fmt.Fprint(os.Stdout, "build info not available\n")
}
}
os.Exit(0)
}
if flags.watchlist == "" {
fmt.Fprintf(os.Stderr, "%s: watch list not found: please create %s or specify alternative path using -watchlist\n", programName, defaultWatchListPath())
os.Exit(2)
}
fsstate := &monitor.FilesystemState{
StateDir: flags.stateDir,
CacheDir: defaultCacheDir(),
SaveCerts: !flags.noSave,
Script: flags.script,
ScriptDir: defaultScriptDir(),
Email: flags.email,
Stdout: flags.stdout,
}
config := &monitor.Config{
LogListSource: flags.logs,
State: fsstate,
StartAtEnd: flags.startAtEnd,
Verbose: flags.verbose,
HealthCheckInterval: flags.healthcheck,
}
emailFileExists := false
if emailRecipients, err := readEmailFile(defaultEmailFile()); err == nil {
emailFileExists = true
fsstate.Email = append(fsstate.Email, emailRecipients...)
} else if !errors.Is(err, fs.ErrNotExist) {
fmt.Fprintf(os.Stderr, "%s: error reading email recipients file %q: %s\n", programName, defaultEmailFile(), err)
os.Exit(1)
}
if len(fsstate.Email) == 0 && !emailFileExists && fsstate.Script == "" && !fileExists(fsstate.ScriptDir) && fsstate.Stdout == false {
fmt.Fprintf(os.Stderr, "%s: no notification methods were specified\n", programName)
fmt.Fprintf(os.Stderr, "Please specify at least one of the following notification methods:\n")
fmt.Fprintf(os.Stderr, " - Place one or more email addresses in %s (one address per line)\n", defaultEmailFile())
fmt.Fprintf(os.Stderr, " - Place one or more executable scripts in the %s directory\n", fsstate.ScriptDir)
fmt.Fprintf(os.Stderr, " - Specify an email address using the -email flag\n")
fmt.Fprintf(os.Stderr, " - Specify the path to an executable script using the -script flag\n")
fmt.Fprintf(os.Stderr, " - Specify the -stdout flag\n")
os.Exit(2)
}
if flags.watchlist == "-" {
watchlist, err := monitor.ReadWatchList(os.Stdin)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: error reading watchlist from standard in: %s\n", programName, err)
os.Exit(1)
}
config.WatchList = watchlist
} else {
watchlist, err := readWatchListFile(flags.watchlist)
if err != nil {
fmt.Fprintf(os.Stderr, "%s: error reading watchlist from %q: %s\n", programName, flags.watchlist, err)
os.Exit(1)
}
config.WatchList = watchlist
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
go func() {
ticker := time.NewTicker(24*time.Hour)
defer ticker.Stop()
for {
fsstate.PruneOldErrors()
select {
case <-ctx.Done():
return
case <-ticker.C:
}
}
}()
if startedBySupervisor() {
// Prevent any process that we spawn from erroneously writing to our directories
os.Unsetenv("CACHE_DIRECTORY")
os.Unsetenv("CONFIGURATION_DIRECTORY")
os.Unsetenv("STATE_DIRECTORY")
}
if err := monitor.Run(ctx, config); ctx.Err() == context.Canceled && errors.Is(err, context.Canceled) {
if flags.verbose {
fmt.Fprintf(os.Stderr, "%s: exiting due to SIGINT or SIGTERM\n", programName)
}
os.Exit(0)
} else {
fmt.Fprintf(os.Stderr, "%s: %s\n", programName, err)
os.Exit(1)
}
}
third_party/certspotter/ctclient/client.go
Vendored
+204
View File
@@ -0,0 +1,204 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
// Package ctclient implements a client for monitoring RFC6962 and static-ct-api Certificate Transparency logs
package ctclient
import (
"bytes"
"context"
"crypto/tls"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strconv"
"time"
"software.sslmate.com/src/certspotter/cttypes"
)
var UserAgent = ""
// Create an HTTP client suitable for communicating with CT logs. dialContext, if non-nil, is used for dialing.
func NewHTTPClient(dialContext func(context.Context, string, string) (net.Conn, error)) *http.Client {
return &http.Client{
Transport: &http.Transport{
Proxy: http.ProxyFromEnvironment,
TLSHandshakeTimeout: 15 * time.Second,
ResponseHeaderTimeout: 30 * time.Second,
MaxIdleConnsPerHost: 10,
IdleConnTimeout: 90 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
TLSClientConfig: &tls.Config{
// We have to disable TLS certificate validation because because several logs
// (WoSign, StartCom, GDCA) use certificates that are not widely trusted.
// Since we verify that every response we receive from the log is signed
// by the log's CT public key (either directly, or indirectly via the Merkle Tree),
// TLS certificate validation is not actually necessary. (We don't want to manage
// our own trust store because that adds undesired complexity and would require
// updating should a log ever change to a different CA.)
InsecureSkipVerify: true,
},
DialContext: dialContext,
ForceAttemptHTTP2: true,
},
CheckRedirect: func(*http.Request, []*http.Request) error {
return errors.New("redirects not followed")
},
Timeout: 60 * time.Second,
}
}
var defaultHTTPClient = NewHTTPClient(nil)
// maxResponseBytes is a generous upper bound on the size of an HTTP response
// body that we are willing to buffer in memory.
const maxResponseBytes = 64 << 20 // 64 MiB
// maxErrorResponseBytes is longest error response that we'll read
const maxErrorResponseBytes = 8 << 10 // 8 KiB
// readResponseBody reads a response body, returning an error if the
// body is larger than maxBytes.
func readResponseBody(r io.Reader, maxBytes int64) ([]byte, error) {
body, err := io.ReadAll(io.LimitReader(r, maxBytes+1))
if err != nil {
return nil, fmt.Errorf("error reading response: %w", err)
}
if int64(len(body)) > maxBytes {
return nil, fmt.Errorf("response body exceeds maximum allowed size of %d bytes", maxBytes)
}
return body, nil
}
func get(ctx context.Context, httpClient *http.Client, fullURL string) (io.ReadCloser, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, fullURL, nil)
if err != nil {
return nil, err
}
request.Header.Set("User-Agent", UserAgent)
if httpClient == nil {
httpClient = defaultHTTPClient
}
response, err := httpClient.Do(request)
if err != nil {
return nil, err
}
if response.StatusCode != 200 {
defer response.Body.Close()
responseBody, _ := io.ReadAll(io.LimitReader(response.Body, maxErrorResponseBytes))
return nil, fmt.Errorf("Get %q: %s (%s)", fullURL, response.Status, formatResponseBody(responseBody))
}
return response.Body, nil
}
func getBytes(ctx context.Context, httpClient *http.Client, fullURL string) ([]byte, error) {
body, err := get(ctx, httpClient, fullURL)
if err != nil {
return nil, err
}
defer body.Close()
bodyBytes, err := readResponseBody(body, maxResponseBytes)
if err != nil {
return nil, fmt.Errorf("Get %q: %w", fullURL, err)
}
return bodyBytes, nil
}
func getJSON(ctx context.Context, httpClient *http.Client, fullURL string, response any) error {
responseBytes, err := getBytes(ctx, httpClient, fullURL)
if err != nil {
return err
}
if err := json.Unmarshal(responseBytes, response); err != nil {
return fmt.Errorf("Get %q: error parsing response JSON: %w", fullURL, err)
}
return nil
}
func getRoots(ctx context.Context, httpClient *http.Client, logURL *url.URL) ([][]byte, error) {
fullURL := logURL.JoinPath("/ct/v1/get-roots").String()
var parsedResponse struct {
Certificates [][]byte `json:"certificates"`
}
if err := getJSON(ctx, httpClient, fullURL, &parsedResponse); err != nil {
return nil, err
}
return parsedResponse.Certificates, nil
}
func addChainOrPreChain(ctx context.Context, httpClient *http.Client, logURL *url.URL, isPreChain bool, chain [][]byte) (*cttypes.SignedCertificateTimestamp, error) {
var endpoint string
if isPreChain {
endpoint = "/ct/v1/add-pre-chain"
} else {
endpoint = "/ct/v1/add-chain"
}
fullURL := logURL.JoinPath(endpoint).String()
requestBody, err := json.Marshal(struct {
Chain [][]byte `json:"chain"`
}{
Chain: chain,
})
if err != nil {
return nil, err
}
request, err := http.NewRequestWithContext(ctx, http.MethodPost, fullURL, bytes.NewReader(requestBody))
if err != nil {
return nil, err
}
request.Header.Set("User-Agent", UserAgent)
request.Header.Set("Content-Type", "application/json")
if httpClient == nil {
httpClient = defaultHTTPClient
}
response, err := httpClient.Do(request)
if err != nil {
return nil, err
}
defer response.Body.Close()
responseBody, err := readResponseBody(response.Body, maxResponseBytes)
if err != nil {
return nil, fmt.Errorf("Post %q: %w", fullURL, err)
}
if response.StatusCode != 200 {
return nil, fmt.Errorf("Post %q: %s (%s)", fullURL, response.Status, formatResponseBody(responseBody))
}
sct := new(cttypes.SignedCertificateTimestamp)
if err := json.Unmarshal(responseBody, sct); err != nil {
return nil, fmt.Errorf("Post %q: error parsing response JSON: %w", fullURL, err)
}
return sct, nil
}
func formatResponseBody(body []byte) string {
const maxLen = 200
body = bytes.TrimSpace(body)
if len(body) > maxLen {
return strconv.QuoteToASCII(string(body[:maxLen])) + "..."
} else {
return strconv.QuoteToASCII(string(body))
}
}
third_party/certspotter/ctclient/client_test.go
Vendored
+77
View File
@@ -0,0 +1,77 @@
// Copyright (C) 2026 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package ctclient
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestReadResponseBody(t *testing.T) {
// under the limit
if body, err := readResponseBody(strings.NewReader("hello"), 10); err != nil {
t.Errorf("unexpected error under limit: %v", err)
} else if string(body) != "hello" {
t.Errorf("got %q, want %q", body, "hello")
}
// exactly at the limit
if body, err := readResponseBody(strings.NewReader("0123456789"), 10); err != nil {
t.Errorf("unexpected error at exact limit: %v", err)
} else if len(body) != 10 {
t.Errorf("got %d bytes, want 10", len(body))
}
// one byte over the limit
if _, err := readResponseBody(strings.NewReader("0123456789X"), 10); err == nil {
t.Errorf("expected error for oversize body, got nil")
}
}
// Confirm the cap is actually wired into getBytes(): a body larger than the limit is
// rejected with an error instead of being buffered in full.
func TestGetEnforcesResponseLimit(t *testing.T) {
const bodyLen = maxResponseBytes + 1024
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
buf := make([]byte, 32*1024)
for written := 0; written < bodyLen; written += len(buf) {
if _, err := w.Write(buf); err != nil {
return
}
}
}))
defer server.Close()
if _, err := getBytes(context.Background(), server.Client(), server.URL); err == nil {
t.Errorf("getBytes() accepted an over-limit response body; expected an error")
} else if !strings.Contains(err.Error(), "maximum allowed size") {
t.Errorf("unexpected error: %v", err)
}
}
// A normal-sized response still round-trips correctly.
func TestGetReadsNormalBody(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "tile-data")
}))
defer server.Close()
body, err := getBytes(context.Background(), server.Client(), server.URL)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if string(body) != "tile-data" {
t.Errorf("got %q, want %q", body, "tile-data")
}
}
third_party/certspotter/ctclient/log.go
Vendored
+59
View File
@@ -0,0 +1,59 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package ctclient
import (
"context"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/merkletree"
)
type WritableLog interface {
AddChain(context.Context, [][]byte) (*cttypes.SignedCertificateTimestamp, error)
AddPreChain(context.Context, [][]byte) (*cttypes.SignedCertificateTimestamp, error)
GetRoots(context.Context) ([][]byte, error)
}
type Log interface {
GetSTH(context.Context) (*cttypes.SignedTreeHead, string, error)
GetRoots(context.Context) ([][]byte, error)
GetEntries(ctx context.Context, startInclusive, endInclusive uint64) ([]Entry, error)
ReconstructTree(context.Context, *cttypes.SignedTreeHead) (*merkletree.CollapsedTree, error)
}
// IssuerGetter represents a source of issuer certificates.
//
// If a [Log] also implements IssuerGetter, then it is mandatory to provide
// an IssuerGetter when using [Entry]s returned by the [Log]. The IssuerGetter
// may be the Log itself, or your own implementation which retrieves issuers
// from a different source, such as a cache.
//
// If a Log doesn't implement IssuerGetter, then you may pass a nil IssuerGetter
// when using the Log's Entrys.
type IssuerGetter interface {
GetIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error)
}
type Entry interface {
LeafInput() []byte
// Returns error from IssuerGetter, otherwise infallible
ExtraData(context.Context, IssuerGetter) ([]byte, error)
// Returns an error if this is not a well-formed precert entry
Precertificate() (cttypes.ASN1Cert, error)
// Returns an error if this is not a well-formed x509 or precert entry
ChainFingerprints() ([][32]byte, error)
// Returns an error if this is not a well-formed x509 or precert entry, or if IssuerGetter failed
GetChain(context.Context, IssuerGetter) (cttypes.ASN1CertChain, error)
}
third_party/certspotter/ctclient/rfc6962.go
Vendored
+272
View File
@@ -0,0 +1,272 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package ctclient
import (
"context"
"crypto/sha256"
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"slices"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/merkletree"
)
type RFC6962Log struct {
URL *url.URL
HTTPClient *http.Client // nil to use default client
}
type RFC6962LogEntry struct {
Leaf_input []byte `json:"leaf_input"`
Extra_data []byte `json:"extra_data"`
}
func (ctlog *RFC6962Log) AddChain(ctx context.Context, chain [][]byte) (*cttypes.SignedCertificateTimestamp, error) {
return addChainOrPreChain(ctx, ctlog.HTTPClient, ctlog.URL, false, chain)
}
func (ctlog *RFC6962Log) AddPreChain(ctx context.Context, chain [][]byte) (*cttypes.SignedCertificateTimestamp, error) {
return addChainOrPreChain(ctx, ctlog.HTTPClient, ctlog.URL, true, chain)
}
func (ctlog *RFC6962Log) GetSTH(ctx context.Context) (*cttypes.SignedTreeHead, string, error) {
fullURL := ctlog.URL.JoinPath("/ct/v1/get-sth").String()
sth := new(cttypes.SignedTreeHead)
if err := getJSON(ctx, ctlog.HTTPClient, fullURL, sth); err != nil {
return nil, fullURL, err
}
return sth, fullURL, nil
}
func (ctlog *RFC6962Log) GetRoots(ctx context.Context) ([][]byte, error) {
return getRoots(ctx, ctlog.HTTPClient, ctlog.URL)
}
func (ctlog *RFC6962Log) getEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]RFC6962LogEntry, error) {
fullURL := ctlog.URL.JoinPath("/ct/v1/get-entries").String()
fullURL += fmt.Sprintf("?start=%d&end=%d", startInclusive, endInclusive)
body, err := get(ctx, ctlog.HTTPClient, fullURL)
if err != nil {
return nil, err
}
defer body.Close()
limitedBody := io.LimitReader(body, maxResponseBytes)
dec := json.NewDecoder(limitedBody)
if tok, err := dec.Token(); err != nil {
return nil, fmt.Errorf("Get %q: error reading JSON object token: %w", fullURL, err)
} else if tok != json.Delim('{') {
return nil, fmt.Errorf("Get %q: not a JSON object", fullURL)
}
for {
if !dec.More() {
return nil, fmt.Errorf("Get %q: JSON object missing entries", fullURL)
}
tok, err := dec.Token()
if err != nil {
return nil, fmt.Errorf("Get %q: error reading JSON field name: %w", fullURL, err)
}
if tok == "entries" {
break
}
var rawMessage json.RawMessage
if err := dec.Decode(&rawMessage); err != nil {
return nil, fmt.Errorf("Get %q: error skipping unwanted JSON field: %w", fullURL, err)
}
}
if tok, err := dec.Token(); err != nil {
return nil, fmt.Errorf("Get %q: error reading JSON array token: %w", fullURL, err)
} else if tok != json.Delim('[') {
return nil, fmt.Errorf("Get %q: entries is not a JSON array", fullURL)
}
var entries []RFC6962LogEntry
for dec.More() {
if uint64(len(entries)) == endInclusive-startInclusive+1 {
return entries, fmt.Errorf("Get %q: extraneous entries returned", fullURL)
}
var entry RFC6962LogEntry
if err := dec.Decode(&entry); err != nil {
return entries, fmt.Errorf("Get %q: error reading JSON entry: %w", fullURL, err)
}
entries = append(entries, entry)
}
if tok, err := dec.Token(); err != nil {
return entries, fmt.Errorf("Get %q: error reading end of JSON array: %w", fullURL, err)
} else if tok != json.Delim(']') {
return entries, fmt.Errorf("Get %q: entries array not terminated", fullURL)
}
if len(entries) == 0 {
return entries, fmt.Errorf("Get %q: zero entries returned", fullURL)
}
io.Copy(io.Discard, limitedBody) // So the HTTP connection can be reused
return entries, nil
}
func (ctlog *RFC6962Log) GetEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]Entry, error) {
nativeEntries, err := ctlog.getEntries(ctx, startInclusive, endInclusive)
var entries []Entry
if nativeEntries != nil {
entries = make([]Entry, len(nativeEntries))
for i := range nativeEntries {
entries[i] = &nativeEntries[i]
}
}
return entries, err
}
type entryAndProofResponse struct {
LeafInput []byte `json:"leaf_input"`
ExtraData []byte `json:"extra_data"`
AuditPath []merkletree.Hash `json:"audit_path"`
}
func (ctlog *RFC6962Log) getEntryAndProof(ctx context.Context, leafIndex uint64, treeSize uint64) (*entryAndProofResponse, error) {
fullURL := ctlog.URL.JoinPath("/ct/v1/get-entry-and-proof").String()
fullURL += fmt.Sprintf("?leaf_index=%d&tree_size=%d", leafIndex, treeSize)
response := new(entryAndProofResponse)
if err := getJSON(ctx, ctlog.HTTPClient, fullURL, response); err != nil {
return nil, err
}
return response, nil
}
type proofResponse struct {
LeafIndex uint64 `json:"leaf_index"`
AuditPath []merkletree.Hash `json:"audit_path"`
}
func (ctlog *RFC6962Log) getProofByHash(ctx context.Context, hash *merkletree.Hash, treeSize uint64) (*proofResponse, error) {
fullURL := ctlog.URL.JoinPath("/ct/v1/get-proof-by-hash").String()
fullURL += fmt.Sprintf("?hash=%s&tree_size=%d", url.QueryEscape(hash.Base64String()), treeSize)
response := new(proofResponse)
if err := getJSON(ctx, ctlog.HTTPClient, fullURL, response); err != nil {
return nil, err
}
return response, nil
}
func (ctlog *RFC6962Log) reconstructTree(ctx context.Context, treeSize uint64) (*merkletree.CollapsedTree, error) {
if treeSize == 0 {
return new(merkletree.CollapsedTree), nil
}
if entryAndProof, err := ctlog.getEntryAndProof(ctx, treeSize-1, treeSize); err == nil {
tree := new(merkletree.CollapsedTree)
slices.Reverse(entryAndProof.AuditPath)
if err := tree.Init(entryAndProof.AuditPath, treeSize-1); err != nil {
return nil, fmt.Errorf("log returned invalid audit proof for entry %d to STH %d: %w", treeSize-1, treeSize, err)
}
tree.Add(merkletree.HashLeaf(entryAndProof.LeafInput))
return tree, nil
}
entries, err := ctlog.getEntries(ctx, treeSize-1, treeSize-1)
if err != nil {
return nil, err
}
leafHash := merkletree.HashLeaf(entries[0].Leaf_input)
tree := new(merkletree.CollapsedTree)
if treeSize > 1 {
response, err := ctlog.getProofByHash(ctx, &leafHash, treeSize)
if err != nil {
return nil, err
}
if response.LeafIndex != treeSize-1 {
// This can happen if the leaf hash is present in the tree in more than one place. Unfortunately, we can't reconstruct when tree if this happens. Fortunately, this is really unlikely, and most logs support get-entry-and-proof anyways.
return nil, fmt.Errorf("unable to reconstruct tree because leaf hash %s is present in tree at more than one index (need proof for index %d but get-proof-by-hash returned proof for index %d)", leafHash.Base64String(), treeSize-1, response.LeafIndex)
}
slices.Reverse(response.AuditPath)
if err := tree.Init(response.AuditPath, treeSize-1); err != nil {
return nil, fmt.Errorf("log returned invalid audit proof for hash %s to STH %d: %w", leafHash.Base64String(), treeSize, err)
}
}
tree.Add(leafHash)
return tree, nil
}
func (ctlog *RFC6962Log) ReconstructTree(ctx context.Context, sth *cttypes.SignedTreeHead) (*merkletree.CollapsedTree, error) {
tree, err := ctlog.reconstructTree(ctx, sth.TreeSize)
if err != nil {
return nil, err
}
if rootHash := tree.CalculateRoot(); rootHash != sth.RootHash {
return nil, fmt.Errorf("calculated root hash (%s) does not match STH (%s) at size %d", rootHash.Base64String(), sth.RootHash.Base64String(), sth.TreeSize)
}
return tree, nil
}
func (entry *RFC6962LogEntry) isX509() bool {
return len(entry.Leaf_input) >= 12 && entry.Leaf_input[0] == 0 && entry.Leaf_input[1] == 0 && entry.Leaf_input[10] == 0 && entry.Leaf_input[11] == 0
}
func (entry *RFC6962LogEntry) isPrecert() bool {
return len(entry.Leaf_input) >= 12 && entry.Leaf_input[0] == 0 && entry.Leaf_input[1] == 0 && entry.Leaf_input[10] == 0 && entry.Leaf_input[11] == 1
}
func (entry *RFC6962LogEntry) LeafInput() []byte {
return entry.Leaf_input
}
func (entry *RFC6962LogEntry) ExtraData(context.Context, IssuerGetter) ([]byte, error) {
return entry.Extra_data, nil
}
func (entry *RFC6962LogEntry) Precertificate() (cttypes.ASN1Cert, error) {
if !entry.isPrecert() {
return nil, fmt.Errorf("not a precertificate entry")
}
extraData, err := cttypes.ParseExtraDataForPrecertEntry(entry.Extra_data)
if err != nil {
return nil, fmt.Errorf("error parsing extra_data: %w", err)
}
return extraData.PreCertificate, nil
}
func (entry *RFC6962LogEntry) ChainFingerprints() ([][32]byte, error) {
chain, err := entry.parseChain()
if err != nil {
return nil, err
}
fingerprints := make([][32]byte, len(chain))
for i := range chain {
fingerprints[i] = sha256.Sum256(chain[i])
}
return fingerprints, nil
}
func (entry *RFC6962LogEntry) GetChain(context.Context, IssuerGetter) (cttypes.ASN1CertChain, error) {
return entry.parseChain()
}
func (entry *RFC6962LogEntry) parseChain() (cttypes.ASN1CertChain, error) {
switch {
case entry.isX509():
extraData, err := cttypes.ParseExtraDataForX509Entry(entry.Extra_data)
if err != nil {
return nil, fmt.Errorf("error parsing extra_data for X509 entry: %w", err)
}
return extraData, nil
case entry.isPrecert():
extraData, err := cttypes.ParseExtraDataForPrecertEntry(entry.Extra_data)
if err != nil {
return nil, fmt.Errorf("error parsing extra_data for precert entry: %w", err)
}
return extraData.PrecertificateChain, nil
default:
return nil, fmt.Errorf("unknown entry type")
}
}
third_party/certspotter/ctclient/static.go
Vendored
+401
View File
@@ -0,0 +1,401 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package ctclient
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"net/http"
"net/url"
"slices"
"strconv"
"sync"
"golang.org/x/crypto/cryptobyte"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/merkletree"
)
const (
staticTileHeight = 8
StaticTileWidth = 1 << staticTileHeight
)
func staticSubtreeSize(level uint64) uint64 { return 1 << (level * staticTileHeight) }
type StaticLog struct {
SubmissionURL *url.URL
MonitoringURL *url.URL
ID cttypes.LogID
HTTPClient *http.Client // nil to use default client
}
type StaticLogEntry struct {
timestampedEntry []byte
precertificate []byte // nil iff x509 entry; non-nil iff precert entry
chain [][32]byte
}
func (ctlog *StaticLog) AddChain(ctx context.Context, chain [][]byte) (*cttypes.SignedCertificateTimestamp, error) {
return addChainOrPreChain(ctx, ctlog.HTTPClient, ctlog.SubmissionURL, false, chain)
}
func (ctlog *StaticLog) AddPreChain(ctx context.Context, chain [][]byte) (*cttypes.SignedCertificateTimestamp, error) {
return addChainOrPreChain(ctx, ctlog.HTTPClient, ctlog.SubmissionURL, true, chain)
}
func (ctlog *StaticLog) GetSTH(ctx context.Context) (*cttypes.SignedTreeHead, string, error) {
fullURL := ctlog.MonitoringURL.JoinPath("/checkpoint").String()
responseBody, err := getBytes(ctx, ctlog.HTTPClient, fullURL)
if err != nil {
return nil, fullURL, err
}
sth, err := cttypes.ParseCheckpoint(responseBody, ctlog.ID)
if err != nil {
return nil, fullURL, err
}
return sth, fullURL, nil
}
func (ctlog *StaticLog) GetRoots(ctx context.Context) ([][]byte, error) {
return getRoots(ctx, ctlog.HTTPClient, ctlog.SubmissionURL)
}
func (ctlog *StaticLog) getEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]StaticLogEntry, error) {
var (
tile = startInclusive / StaticTileWidth
skip = startInclusive % StaticTileWidth
tileWidth = min(StaticTileWidth, endInclusive+1-tile*StaticTileWidth)
numEntries = tileWidth - skip
)
data, err := ctlog.getDataTile(ctx, tile, tileWidth)
if err != nil {
return nil, err
}
var skippedEntry StaticLogEntry
for i := range skip {
if rest, err := skippedEntry.parse(data); err != nil {
return nil, fmt.Errorf("error parsing skipped entry %d in tile %d: %w", i, tile, err)
} else {
data = rest
}
}
entries := make([]StaticLogEntry, numEntries)
for i := range numEntries {
if rest, err := entries[i].parse(data); err != nil {
return nil, fmt.Errorf("error parsing entry %d in tile %d: %w", skip+i, tile, err)
} else {
data = rest
}
}
return entries, nil
}
func (ctlog *StaticLog) GetEntries(ctx context.Context, startInclusive uint64, endInclusive uint64) ([]Entry, error) {
nativeEntries, err := ctlog.getEntries(ctx, startInclusive, endInclusive)
if err != nil {
return nil, err
}
entries := make([]Entry, len(nativeEntries))
for i := range nativeEntries {
entries[i] = &nativeEntries[i]
}
return entries, nil
}
func (ctlog *StaticLog) ReconstructTree(ctx context.Context, sth *cttypes.SignedTreeHead) (*merkletree.CollapsedTree, error) {
type job struct {
level uint64
offset uint64
width uint64
}
var jobs []job
for level, size := uint64(0), sth.TreeSize; size > 0; level++ {
fullTiles := size / StaticTileWidth
remainder := size % StaticTileWidth
size = fullTiles
if remainder > 0 {
jobs = append(jobs, job{
level: level,
offset: fullTiles,
width: remainder,
})
}
}
trees := make([]*merkletree.CollapsedTree, len(jobs))
errs := make([]error, len(jobs))
var wg sync.WaitGroup
for i, job := range jobs {
wg.Add(1)
go func() {
trees[i], errs[i] = ctlog.getTileCollapsedTree(ctx, job.level, job.offset, job.width)
wg.Done()
}()
}
wg.Wait()
if err := errors.Join(errs...); err != nil {
return nil, err
}
tree := new(merkletree.CollapsedTree)
for _, subtree := range slices.Backward(trees) {
if err := tree.Append(*subtree); err != nil {
panic(err)
}
}
if rootHash := tree.CalculateRoot(); rootHash != sth.RootHash {
return nil, fmt.Errorf("calculated root hash (%s) does not match STH (%s) at size %d", rootHash.Base64String(), sth.RootHash.Base64String(), sth.TreeSize)
}
return tree, nil
}
func (ctlog *StaticLog) getDataTile(ctx context.Context, tile uint64, width uint64) ([]byte, error) {
if width == 0 || width > StaticTileWidth {
panic("width is out of range")
}
var partialErr error
if width < StaticTileWidth {
fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath("data", tile, width)).String()
if data, err := getBytes(ctx, ctlog.HTTPClient, fullURL); err != nil {
partialErr = err
} else {
return data, nil
}
}
fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath("data", tile, 0)).String()
if data, err := getBytes(ctx, ctlog.HTTPClient, fullURL); err != nil {
if partialErr != nil {
return nil, partialErr
} else {
return nil, err
}
} else {
return data, nil
}
}
// returned slice is numHashes*merkletree.HashLen bytes long
func (ctlog *StaticLog) getTile(ctx context.Context, level uint64, tile uint64, numHashes uint64) ([]byte, error) {
if numHashes == 0 || numHashes > StaticTileWidth {
panic("numHashes is out of range")
}
var partialErr error
if numHashes < StaticTileWidth {
fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath(strconv.FormatUint(level, 10), tile, numHashes)).String()
if data, err := getBytes(ctx, ctlog.HTTPClient, fullURL); err != nil {
partialErr = err
} else if expectedLen := merkletree.HashLen * int(numHashes); len(data) != expectedLen {
return nil, fmt.Errorf("%s returned %d bytes instead of expected %d", fullURL, len(data), expectedLen)
} else {
return data, nil
}
}
fullURL := ctlog.MonitoringURL.JoinPath(formatTilePath(strconv.FormatUint(level, 10), tile, 0)).String()
if data, err := getBytes(ctx, ctlog.HTTPClient, fullURL); err != nil {
if partialErr != nil {
return nil, partialErr
} else {
return nil, err
}
} else if expectedLen := merkletree.HashLen * StaticTileWidth; len(data) != expectedLen {
return nil, fmt.Errorf("%s returned %d bytes instead of expected %d", fullURL, len(data), expectedLen)
} else {
desiredLen := merkletree.HashLen * int(numHashes)
return data[:desiredLen], nil
}
}
func (ctlog *StaticLog) getTileCollapsedTree(ctx context.Context, level uint64, tile uint64, numHashes uint64) (*merkletree.CollapsedTree, error) {
data, err := ctlog.getTile(ctx, level, tile, numHashes)
if err != nil {
return nil, err
}
subtreeSize := staticSubtreeSize(level)
offset := staticSubtreeSize(level+1) * tile
tree := new(merkletree.CollapsedTree)
if err := tree.InitSubtree(offset, nil, 0); err != nil {
panic(err)
}
for i := uint64(0); i < numHashes; i++ {
hash := (merkletree.Hash)(data[i*merkletree.HashLen : (i+1)*merkletree.HashLen])
var subtree merkletree.CollapsedTree
if err := subtree.InitSubtree(offset+i*subtreeSize, []merkletree.Hash{hash}, subtreeSize); err != nil {
panic(err)
}
if err := tree.Append(subtree); err != nil {
panic(err)
}
}
return tree, nil
}
func (ctlog *StaticLog) GetIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error) {
fullURL := ctlog.MonitoringURL.JoinPath("/issuer/" + hex.EncodeToString(fingerprint[:])).String()
data, err := getBytes(ctx, ctlog.HTTPClient, fullURL)
if err != nil {
return nil, err
}
if gotFingerprint := sha256.Sum256(data); gotFingerprint != *fingerprint {
return nil, fmt.Errorf("%s returned incorrect data with fingerprint %x", fullURL, gotFingerprint[:])
}
return data, nil
}
func (entry *StaticLogEntry) parse(input []byte) ([]byte, error) {
var skipped cryptobyte.String
str := cryptobyte.String(input)
// TimestampedEntry.timestamp
if !str.Skip(8) {
return nil, fmt.Errorf("error reading timestamp")
}
// TimestampedEntry.entry_type
var entryType uint16
if !str.ReadUint16(&entryType) {
return nil, fmt.Errorf("error reading entry type")
}
// TimestampedEntry.signed_entry
if entryType == 0 {
if !str.ReadUint24LengthPrefixed(&skipped) {
return nil, fmt.Errorf("error reading certificate")
}
} else if entryType == 1 {
if !str.Skip(32) {
return nil, fmt.Errorf("error reading issuer_key_hash")
}
if !str.ReadUint24LengthPrefixed(&skipped) {
return nil, fmt.Errorf("error reading tbs_certificate")
}
} else {
return nil, fmt.Errorf("invalid entry type %d", entryType)
}
// TimestampedEntry.extensions
if !str.ReadUint16LengthPrefixed(&skipped) {
return nil, fmt.Errorf("error reading extensions")
}
timestampedEntryLen := len(input) - len(str)
entry.timestampedEntry = input[:timestampedEntryLen]
// precertificate
if entryType == 1 {
var precertificate cryptobyte.String
if !str.ReadUint24LengthPrefixed(&precertificate) {
return nil, fmt.Errorf("error reading precertificate")
}
entry.precertificate = precertificate
} else {
entry.precertificate = nil
}
// certificate_chain
var chainBytes cryptobyte.String
if !str.ReadUint16LengthPrefixed(&chainBytes) {
return nil, fmt.Errorf("error reading certificate_chain")
}
entry.chain = make([][32]byte, 0, len(chainBytes)/32)
for !chainBytes.Empty() {
var fingerprint [32]byte
if !chainBytes.CopyBytes(fingerprint[:]) {
return nil, fmt.Errorf("error reading fingerprint in certificate_chain")
}
entry.chain = append(entry.chain, fingerprint)
}
return str, nil
}
func (entry *StaticLogEntry) LeafInput() []byte {
return append([]byte{0, 0}, entry.timestampedEntry...)
}
func (entry *StaticLogEntry) ExtraData(ctx context.Context, issuerGetter IssuerGetter) ([]byte, error) {
b := cryptobyte.NewBuilder(nil)
if entry.precertificate != nil {
b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
b.AddBytes(entry.precertificate)
})
}
b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
for _, fingerprint := range entry.chain {
b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
cert, err := issuerGetter.GetIssuer(ctx, &fingerprint)
if err != nil {
panic(cryptobyte.BuildError{Err: fmt.Errorf("error getting issuer %x: %w", fingerprint, err)})
}
b.AddBytes(cert)
})
}
})
return b.Bytes()
}
func (entry *StaticLogEntry) Precertificate() (cttypes.ASN1Cert, error) {
if entry.precertificate == nil {
return nil, fmt.Errorf("not a precertificate entry")
}
return entry.precertificate, nil
}
func (entry *StaticLogEntry) ChainFingerprints() ([][32]byte, error) {
return entry.chain, nil
}
func (entry *StaticLogEntry) GetChain(ctx context.Context, issuerGetter IssuerGetter) (cttypes.ASN1CertChain, error) {
var (
chain = make(cttypes.ASN1CertChain, len(entry.chain))
errs = make([]error, len(entry.chain))
)
var wg sync.WaitGroup
for i, fingerprint := range entry.chain {
wg.Add(1)
go func() {
defer wg.Done()
chain[i], errs[i] = issuerGetter.GetIssuer(ctx, &fingerprint)
}()
}
wg.Wait()
if err := errors.Join(errs...); err != nil {
return nil, err
}
return chain, nil
}
func formatTilePath(level string, tile uint64, partial uint64) string {
path := "tile/" + level + "/" + formatTileIndex(tile)
if partial != 0 {
path += fmt.Sprintf(".p/%d", partial)
}
return path
}
func formatTileIndex(tile uint64) string {
const base = 1000
str := fmt.Sprintf("%03d", tile%base)
for tile >= base {
tile = tile / base
str = fmt.Sprintf("x%03d/%s", tile%base, str)
}
return str
}
third_party/certspotter/ctclient/static_test.go
Vendored
+38
View File
@@ -0,0 +1,38 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package ctclient
import (
"testing"
)
func TestFormatTileIndex(t *testing.T) {
tests := []struct {
in uint64
out string
}{
{0, "000"},
{1, "001"},
{12, "012"},
{105, "105"},
{1000, "x001/000"},
{1050, "x001/050"},
{52123, "x052/123"},
{999001, "x999/001"},
{1999001, "x001/x999/001"},
{15999001, "x015/x999/001"},
}
for i, test := range tests {
result := formatTileIndex(test.in)
if result != test.out {
t.Errorf("#%d: formatTileIndex(%q) = %q, want %q", i, test.in, result, test.out)
}
}
}
third_party/certspotter/ctcrypto/key.go
Vendored
+67
View File
@@ -0,0 +1,67 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package ctcrypto
import (
"bytes"
"crypto"
"crypto/ecdsa"
"crypto/rsa"
"crypto/x509"
"fmt"
"software.sslmate.com/src/certspotter/tlstypes"
)
type PublicKey []byte
func (key PublicKey) Verify(input SignatureInput, signature tlstypes.DigitallySigned) error {
parsedKey, err := x509.ParsePKIXPublicKey(key)
if err != nil {
return fmt.Errorf("error parsing log key: %w", err)
}
switch key := parsedKey.(type) {
case *rsa.PublicKey:
if signature.Algorithm.Signature != tlstypes.RSA {
return fmt.Errorf("log key is RSA but this is not an RSA signature")
}
if signature.Algorithm.Hash != tlstypes.SHA256 {
return fmt.Errorf("unsupported hash algorithm %v (only SHA-256 is allowed in CT)", signature.Algorithm.Hash)
}
if rsa.VerifyPKCS1v15((*rsa.PublicKey)(key), crypto.SHA256, input[:], signature.Signature) != nil {
return fmt.Errorf("RSA signature is incorrect")
}
return nil
case *ecdsa.PublicKey:
if signature.Algorithm.Signature != tlstypes.ECDSA {
return fmt.Errorf("log key is ECDSA but this is not an ECDSA signature")
}
if signature.Algorithm.Hash != tlstypes.SHA256 {
return fmt.Errorf("unsupported hash algorithm %v (only SHA-256 is allowed in CT)", signature.Algorithm.Hash)
}
if !ecdsa.VerifyASN1((*ecdsa.PublicKey)(key), input[:], signature.Signature) {
return fmt.Errorf("ECDSA signature is incorrect")
}
default:
return fmt.Errorf("unsupported public key type %T (CT only allows RSA and ECDSA)", key)
}
return nil
}
func (key PublicKey) MarshalBinary() ([]byte, error) {
return bytes.Clone(key), nil
}
func (key *PublicKey) UnmarshalBinary(data []byte) error {
*key = bytes.Clone(data)
return nil
}
third_party/certspotter/ctcrypto/signatures.go
Vendored
+55
View File
@@ -0,0 +1,55 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package ctcrypto
import (
"crypto/sha256"
"golang.org/x/crypto/cryptobyte"
"software.sslmate.com/src/certspotter/cttypes"
)
type SignatureInput [32]byte
func MakeSignatureInput(message []byte) SignatureInput {
return sha256.Sum256(message)
}
func SignatureInputForPrecertSCT(sct *cttypes.SignedCertificateTimestamp, precert cttypes.PreCert) SignatureInput {
var builder cryptobyte.Builder
builder.AddValue(sct.SCTVersion)
builder.AddValue(cttypes.CertificateTimestampSignatureType)
builder.AddUint64(sct.Timestamp)
builder.AddValue(cttypes.PrecertEntryType)
builder.AddValue(&precert)
builder.AddValue(sct.Extensions)
return MakeSignatureInput(builder.BytesOrPanic())
}
func SignatureInputForCertSCT(sct *cttypes.SignedCertificateTimestamp, cert cttypes.ASN1Cert) SignatureInput {
var builder cryptobyte.Builder
builder.AddValue(sct.SCTVersion)
builder.AddValue(cttypes.CertificateTimestampSignatureType)
builder.AddUint64(sct.Timestamp)
builder.AddValue(cttypes.X509EntryType)
builder.AddValue(cert)
builder.AddValue(sct.Extensions)
return MakeSignatureInput(builder.BytesOrPanic())
}
func SignatureInputForSTH(sth *cttypes.SignedTreeHead) SignatureInput {
var builder cryptobyte.Builder
builder.AddValue(cttypes.V1)
builder.AddValue(cttypes.TreeHashSignatureType)
builder.AddUint64(sth.Timestamp)
builder.AddUint64(sth.TreeSize)
builder.AddBytes(sth.RootHash[:])
return MakeSignatureInput(builder.BytesOrPanic())
}
third_party/certspotter/cttypes/certs.go
Vendored
+121
View File
@@ -0,0 +1,121 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"fmt"
"golang.org/x/crypto/cryptobyte"
)
type TBSCertificate []byte
type ASN1Cert []byte
type ASN1CertChain []ASN1Cert
// Corresponds to the PreCert structure in RFC 6962. PreCert is a misnomer because this is really a TBSCertificate, not a precertificate.
type PreCert struct {
IssuerKeyHash [32]byte
TBSCertificate TBSCertificate
}
type PrecertChainEntry struct {
PreCertificate ASN1Cert
PrecertificateChain ASN1CertChain
}
func (v *TBSCertificate) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint24LengthPrefixed((*cryptobyte.String)(v))
}
func (v TBSCertificate) Marshal(b *cryptobyte.Builder) error {
b.AddUint24LengthPrefixed(addBytesFunc(v))
return nil
}
func (v *ASN1Cert) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint24LengthPrefixed((*cryptobyte.String)(v))
}
func (v ASN1Cert) Marshal(b *cryptobyte.Builder) error {
b.AddUint24LengthPrefixed(addBytesFunc(v))
return nil
}
func (v *ASN1CertChain) Unmarshal(s *cryptobyte.String) bool {
chainBytes := new(cryptobyte.String)
if !s.ReadUint24LengthPrefixed(chainBytes) {
return false
}
*v = []ASN1Cert{}
for !chainBytes.Empty() {
var cert ASN1Cert
if !cert.Unmarshal(chainBytes) {
return false
}
*v = append(*v, cert)
}
return true
}
func (v ASN1CertChain) Marshal(b *cryptobyte.Builder) error {
b.AddUint24LengthPrefixed(func(b *cryptobyte.Builder) {
for _, cert := range v {
b.AddValue(cert)
}
})
return nil
}
func (precert *PreCert) Unmarshal(s *cryptobyte.String) error {
if !s.CopyBytes(precert.IssuerKeyHash[:]) {
return fmt.Errorf("error reading PreCert issuer_key_hash")
}
if !precert.TBSCertificate.Unmarshal(s) {
return fmt.Errorf("error reading PreCert tbs_certificate")
}
return nil
}
func (v *PreCert) Marshal(b *cryptobyte.Builder) error {
b.AddBytes(v.IssuerKeyHash[:])
b.AddValue(v.TBSCertificate)
return nil
}
func (entry *PrecertChainEntry) Unmarshal(s *cryptobyte.String) error {
if !entry.PreCertificate.Unmarshal(s) {
return fmt.Errorf("error reading PrecertChainEntry pre_certificate")
}
if !entry.PrecertificateChain.Unmarshal(s) {
return fmt.Errorf("error reading PrecertChainEntry preeertificate_chain")
}
return nil
}
func ParseExtraDataForX509Entry(extraData []byte) (ASN1CertChain, error) {
str := cryptobyte.String(extraData)
var chain ASN1CertChain
if !chain.Unmarshal(&str) {
return nil, fmt.Errorf("error reading ASN.1Cert chain")
}
if !str.Empty() {
return nil, fmt.Errorf("trailing garbage after ASN.1Cert chain")
}
return chain, nil
}
func ParseExtraDataForPrecertEntry(extraData []byte) (*PrecertChainEntry, error) {
str := cryptobyte.String(extraData)
entry := new(PrecertChainEntry)
if err := entry.Unmarshal(&str); err != nil {
return nil, err
}
if !str.Empty() {
return nil, fmt.Errorf("trailing garbage after PrecertChainEntry")
}
return entry, nil
}
third_party/certspotter/cttypes/checkpoint.go
Vendored
+127
View File
@@ -0,0 +1,127 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"bytes"
"crypto/sha256"
"encoding/base64"
"encoding/binary"
"errors"
"fmt"
"strconv"
"strings"
"software.sslmate.com/src/certspotter/merkletree"
"software.sslmate.com/src/certspotter/tlstypes"
)
func chompLine(input []byte) (string, []byte, bool) {
newline := bytes.IndexByte(input, '\n')
if newline == -1 {
return "", nil, false
}
return string(input[:newline]), input[newline+1:], true
}
func makeCheckpointKeyID(origin string, logID LogID) [4]byte {
h := sha256.New()
h.Write([]byte(origin))
h.Write([]byte{'\n', 0x05})
h.Write(logID[:])
var digest [sha256.Size]byte
h.Sum(digest[:0])
return [4]byte(digest[:4])
}
func ParseCheckpoint(input []byte, logID LogID) (*SignedTreeHead, error) {
// origin
origin, input, _ := chompLine(input)
// tree size
sizeLine, input, _ := chompLine(input)
treeSize, err := strconv.ParseUint(sizeLine, 10, 64)
if err != nil {
return nil, fmt.Errorf("malformed tree size: %w", err)
}
// root hash
hashLine, input, _ := chompLine(input)
rootHash, err := base64.StdEncoding.DecodeString(hashLine)
if err != nil {
return nil, fmt.Errorf("malformed root hash: %w", err)
}
if len(rootHash) != merkletree.HashLen {
return nil, fmt.Errorf("root hash has wrong length (should be %d bytes long, not %d)", merkletree.HashLen, len(rootHash))
}
// 0 or more non-empty extension lines (ignored)
for {
line, rest, ok := chompLine(input)
if !ok {
return nil, errors.New("signed note ended prematurely")
}
input = rest
if len(line) == 0 {
break
}
}
// signature lines
signaturePrefix := "\u2014 " + origin + " "
keyID := makeCheckpointKeyID(origin, logID)
for {
signatureLine, rest, ok := chompLine(input)
if !ok {
return nil, errors.New("signed note is missing signature from the log")
}
input = rest
if !strings.HasPrefix(signatureLine, signaturePrefix) {
continue
}
signatureBytes, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(signatureLine, signaturePrefix))
if err != nil {
return nil, fmt.Errorf("malformed signature: %w", err)
}
if !bytes.HasPrefix(signatureBytes, keyID[:]) {
continue
}
if len(signatureBytes) < 12 {
return nil, errors.New("malformed signature: too short")
}
timestamp := binary.BigEndian.Uint64(signatureBytes[4:12])
signature, err := tlstypes.ParseDigitallySigned(signatureBytes[12:])
if err != nil {
return nil, fmt.Errorf("malformed signature: %w", err)
}
return &SignedTreeHead{
TreeSize: treeSize,
Timestamp: timestamp,
RootHash: (merkletree.Hash)(rootHash),
Signature: *signature,
}, nil
}
}
func FormatCheckpoint(sth *SignedTreeHead, origin string, logID LogID) []byte {
keyID := makeCheckpointKeyID(origin, logID)
signature := keyID[:]
signature = binary.BigEndian.AppendUint64(signature, sth.Timestamp)
signature = append(signature, sth.Signature.Bytes()...)
buf := new(bytes.Buffer)
fmt.Fprintln(buf, origin)
fmt.Fprintln(buf, sth.TreeSize)
fmt.Fprintln(buf, sth.RootHash.Base64String())
fmt.Fprintln(buf)
fmt.Fprintf(buf, "\u2014 %s %s\n", origin, base64.StdEncoding.EncodeToString(signature))
return buf.Bytes()
}
third_party/certspotter/cttypes/checkpoint_test.go
Vendored
+113
View File
@@ -0,0 +1,113 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"bytes"
"software.sslmate.com/src/certspotter/tlstypes"
"testing"
)
func TestParseCheckpoint(t *testing.T) {
logID := LogID{0x49, 0x4d, 0x90, 0x49, 0xb8, 0xaf, 0x3e, 0x5a, 0xca, 0xba, 0x99, 0x3e, 0x4c, 0x1f, 0x30, 0x56, 0x73, 0x7a, 0xa9, 0xf9, 0x6d, 0x00, 0xf7, 0xb0, 0xb9, 0xb2, 0x51, 0x06, 0xf7, 0xbe, 0x1a, 0x8f}
expected := SignedTreeHead{
TreeSize: 820495916,
Timestamp: 1719511711300,
RootHash: [...]byte{0xc0, 0x3a, 0x6b, 0xe9, 0xf1, 0x1d, 0xc9, 0xcc, 0x20, 0x89, 0xe4, 0x45, 0xa7, 0x3f, 0x61, 0x41, 0xee, 0x82, 0xe8, 0x0d, 0x2d, 0x83, 0xa2, 0xb6, 0x65, 0x53, 0xd4, 0x96, 0xd7, 0x1e, 0xd2, 0x12},
Signature: tlstypes.DigitallySigned{
Algorithm: tlstypes.SignatureAndHashAlgorithm{
Hash: tlstypes.SHA256,
Signature: tlstypes.ECDSA,
},
Signature: []byte{0x30, 0x46, 0x02, 0x21, 0x00, 0xd0, 0x0d, 0x31, 0x91, 0x50, 0x80, 0x62, 0xfa, 0xb0, 0xf9, 0xf7, 0x63, 0x61, 0x78, 0x95, 0x2b, 0x9c, 0x19, 0x22, 0x3a, 0x1d, 0x08, 0xc5, 0x68, 0x0e, 0xd0, 0x8b, 0x3b, 0x79, 0x18, 0x88, 0x86, 0x02, 0x21, 0x00, 0xd5, 0x74, 0xae, 0x8c, 0x1d, 0x1d, 0x7a, 0x4e, 0x80, 0x6c, 0x36, 0x46, 0x81, 0xb4, 0x7c, 0x91, 0x78, 0xc0, 0x3f, 0xdc, 0xc0, 0xab, 0xa5, 0x90, 0x40, 0x8d, 0x0e, 0xf6, 0x2c, 0x83, 0xa9, 0x34},
},
}
sthStrings := []string{
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\nextension line\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\nextension line\nanother extension line\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n\u2014 someoneelse.example c2lnbmF0dXJlCg==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 someoneelse.example c2lnbmF0dXJlCg==\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\nextension line\nanother extension line\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2c2lnbmF0dXJlCg==\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
}
for i, str := range sthStrings {
parsed, err := ParseCheckpoint([]byte(str), logID)
if err != nil {
t.Errorf("%d: Unexpected error: %s", i, err)
return
}
if parsed.TreeSize != expected.TreeSize {
t.Errorf("%d: wrong tree size", i)
return
}
if parsed.Timestamp != expected.Timestamp {
t.Errorf("%d: wrong timestamp", i)
return
}
if !bytes.Equal(parsed.RootHash[:], expected.RootHash[:]) {
t.Errorf("%d: wrong root hash", i)
return
}
if !(parsed.Signature.Algorithm == expected.Signature.Algorithm && bytes.Equal(parsed.Signature.Signature, expected.Signature.Signature)) {
t.Errorf("%d: wrong signature", i)
return
}
}
}
func TestParseCheckpointFailure(t *testing.T) {
logID := LogID{0x49, 0x4d, 0x90, 0x49, 0xb8, 0xaf, 0x3e, 0x5a, 0xca, 0xba, 0x99, 0x3e, 0x4c, 0x1f, 0x30, 0x56, 0x73, 0x7a, 0xa9, 0xf9, 0x6d, 0x00, 0xf7, 0xb0, 0xb9, 0xb2, 0x51, 0x06, 0xf7, 0xbe, 0x1a, 0x8f}
sthStrings := []string{
"",
"\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 oak.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 bm90YXNpZ25hdHVyZQo=\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 notbase64!\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916!\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEd!ycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
"sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n- sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n",
}
for i, str := range sthStrings {
_, err := ParseCheckpoint([]byte(str), logID)
if err == nil {
t.Errorf("%d: Unexpected success", i)
return
}
}
}
func TestFormatCheckpoint(t *testing.T) {
origin := "sycamore.ct.letsencrypt.org/2024h2"
logID := LogID{0x49, 0x4d, 0x90, 0x49, 0xb8, 0xaf, 0x3e, 0x5a, 0xca, 0xba, 0x99, 0x3e, 0x4c, 0x1f, 0x30, 0x56, 0x73, 0x7a, 0xa9, 0xf9, 0x6d, 0x00, 0xf7, 0xb0, 0xb9, 0xb2, 0x51, 0x06, 0xf7, 0xbe, 0x1a, 0x8f}
sth := SignedTreeHead{
TreeSize: 820495916,
Timestamp: 1719511711300,
RootHash: [...]byte{0xc0, 0x3a, 0x6b, 0xe9, 0xf1, 0x1d, 0xc9, 0xcc, 0x20, 0x89, 0xe4, 0x45, 0xa7, 0x3f, 0x61, 0x41, 0xee, 0x82, 0xe8, 0x0d, 0x2d, 0x83, 0xa2, 0xb6, 0x65, 0x53, 0xd4, 0x96, 0xd7, 0x1e, 0xd2, 0x12},
Signature: tlstypes.DigitallySigned{
Algorithm: tlstypes.SignatureAndHashAlgorithm{
Hash: tlstypes.SHA256,
Signature: tlstypes.ECDSA,
},
Signature: []byte{0x30, 0x46, 0x02, 0x21, 0x00, 0xd0, 0x0d, 0x31, 0x91, 0x50, 0x80, 0x62, 0xfa, 0xb0, 0xf9, 0xf7, 0x63, 0x61, 0x78, 0x95, 0x2b, 0x9c, 0x19, 0x22, 0x3a, 0x1d, 0x08, 0xc5, 0x68, 0x0e, 0xd0, 0x8b, 0x3b, 0x79, 0x18, 0x88, 0x86, 0x02, 0x21, 0x00, 0xd5, 0x74, 0xae, 0x8c, 0x1d, 0x1d, 0x7a, 0x4e, 0x80, 0x6c, 0x36, 0x46, 0x81, 0xb4, 0x7c, 0x91, 0x78, 0xc0, 0x3f, 0xdc, 0xc0, 0xab, 0xa5, 0x90, 0x40, 0x8d, 0x0e, 0xf6, 0x2c, 0x83, 0xa9, 0x34},
},
}
expected := "sycamore.ct.letsencrypt.org/2024h2\n820495916\nwDpr6fEdycwgieRFpz9hQe6C6A0tg6K2ZVPUltce0hI=\n\n\u2014 sycamore.ct.letsencrypt.org/2024h2 PdkIagAAAZBa4n5EBAMASDBGAiEA0A0xkVCAYvqw+fdjYXiVK5wZIjodCMVoDtCLO3kYiIYCIQDVdK6MHR16ToBsNkaBtHyReMA/3MCrpZBAjQ72LIOpNA==\n"
got := FormatCheckpoint(&sth, origin, logID)
if string(got) != expected {
t.Errorf("expected %q, got %q", expected, got)
}
}
third_party/certspotter/cttypes/helpers.go
Vendored
+20
View File
@@ -0,0 +1,20 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"golang.org/x/crypto/cryptobyte"
)
func addBytesFunc(v []byte) cryptobyte.BuilderContinuation {
return func(b *cryptobyte.Builder) {
b.AddBytes(v)
}
}
third_party/certspotter/cttypes/logid.go
Vendored
+69
View File
@@ -0,0 +1,69 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"bytes"
"encoding/base64"
"fmt"
"golang.org/x/crypto/cryptobyte"
)
type LogID [32]byte
func (id LogID) Compare(other LogID) int {
return bytes.Compare(id[:], other[:])
}
func (v *LogID) Unmarshal(s *cryptobyte.String) bool {
return s.CopyBytes((*v)[:])
}
func (v LogID) Marshal(b *cryptobyte.Builder) error {
b.AddBytes(v[:])
return nil
}
func (id *LogID) UnmarshalBinary(bytes []byte) error {
if len(bytes) != len(*id) {
return fmt.Errorf("LogID has wrong length (should be %d, not %d)", len(*id), len(bytes))
}
*id = (LogID)(bytes)
return nil
}
func (id LogID) MarshalBinary() ([]byte, error) {
return id[:], nil
}
func (id *LogID) UnmarshalText(textData []byte) error {
if len(textData) != 44 {
return fmt.Errorf("LogID has wrong length (should be %d, not %d)", 44, len(textData))
}
var bytes [33]byte
if n, err := base64.StdEncoding.Decode(bytes[:], textData); err != nil {
return fmt.Errorf("LogID contains invalid base64: %w", err)
} else if n != 32 {
return fmt.Errorf("LogID has wrong length (should be %d bytes, not %d)", 32, n)
}
copy(id[:], bytes[:])
return nil
}
func (id LogID) MarshalText() ([]byte, error) {
encodedBytes := make([]byte, 44)
base64.StdEncoding.Encode(encodedBytes, id[:])
return encodedBytes, nil
}
func (id LogID) Base64String() string {
return base64.StdEncoding.EncodeToString(id[:])
}
func (id LogID) Base64URLString() string {
return base64.RawURLEncoding.EncodeToString(id[:])
}
third_party/certspotter/cttypes/merkleleaf.go
Vendored
+213
View File
@@ -0,0 +1,213 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"fmt"
"golang.org/x/crypto/cryptobyte"
"software.sslmate.com/src/certspotter/merkletree"
)
type MerkleLeafType uint8
const (
TimestampedEntryType MerkleLeafType = 0
)
type LogEntryType uint16
const (
X509EntryType LogEntryType = 0
PrecertEntryType LogEntryType = 1
)
type CTExtensions []byte
type MerkleTreeLeaf struct {
Version Version
LeafType MerkleLeafType
TimestampedEntry *TimestampedEntry
}
type TimestampedEntry struct {
Timestamp uint64
EntryType LogEntryType
SignedEntryASN1Cert *ASN1Cert
SignedEntryPreCert *PreCert
Extensions CTExtensions
}
func (v *MerkleLeafType) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint8((*uint8)(v))
}
func (v MerkleLeafType) Marshal(b *cryptobyte.Builder) error {
b.AddUint8(uint8(v))
return nil
}
func (v *LogEntryType) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint16((*uint16)(v))
}
func (v LogEntryType) Marshal(b *cryptobyte.Builder) error {
b.AddUint16(uint16(v))
return nil
}
func (v *CTExtensions) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint16LengthPrefixed((*cryptobyte.String)(v))
}
func (v CTExtensions) Marshal(b *cryptobyte.Builder) error {
b.AddUint16LengthPrefixed(addBytesFunc(v))
return nil
}
func (leaf *MerkleTreeLeaf) Unmarshal(s *cryptobyte.String) error {
if !leaf.Version.Unmarshal(s) {
return fmt.Errorf("error reading MerkleTreeLeaf version")
}
if leaf.Version != V1 {
return fmt.Errorf("unsupported Version 0x%02x", leaf.Version)
}
if !leaf.LeafType.Unmarshal(s) {
return fmt.Errorf("error reading MerkleTreeLeaf leaf_type")
}
switch leaf.LeafType {
case TimestampedEntryType:
leaf.TimestampedEntry = new(TimestampedEntry)
if err := leaf.TimestampedEntry.Unmarshal(s); err != nil {
return err
}
default:
return fmt.Errorf("unrecognized MerkleLeafType 0x%02x", leaf.LeafType)
}
return nil
}
func (v *MerkleTreeLeaf) Marshal(b *cryptobyte.Builder) error {
b.AddValue(v.Version)
b.AddValue(v.LeafType)
switch v.LeafType {
case TimestampedEntryType:
b.AddValue(v.TimestampedEntry)
}
return nil
}
func (v *MerkleTreeLeaf) Bytes() ([]byte, error) {
var builder cryptobyte.Builder
builder.AddValue(v)
return builder.Bytes()
}
func (v *MerkleTreeLeaf) Hash() merkletree.Hash {
var builder cryptobyte.Builder
builder.AddValue(v)
return merkletree.HashLeaf(builder.BytesOrPanic())
}
func (entry *TimestampedEntry) Unmarshal(s *cryptobyte.String) error {
if !s.ReadUint64(&entry.Timestamp) {
return fmt.Errorf("error reading TimestampedEntry timestamp")
}
if !entry.EntryType.Unmarshal(s) {
return fmt.Errorf("error reading TimestampedEntry entry_type")
}
switch entry.EntryType {
case X509EntryType:
entry.SignedEntryASN1Cert = new(ASN1Cert)
if !entry.SignedEntryASN1Cert.Unmarshal(s) {
return fmt.Errorf("error reading TimestampedEntry signed_entry ASN.1Cert")
}
case PrecertEntryType:
entry.SignedEntryPreCert = new(PreCert)
if err := entry.SignedEntryPreCert.Unmarshal(s); err != nil {
return fmt.Errorf("error reading TimestampedEntryType signed_entry: %w", err)
}
default:
return fmt.Errorf("unrecognized TimestampedEntryType 0x%02x", entry.EntryType)
}
if !entry.Extensions.Unmarshal(s) {
return fmt.Errorf("error reading TimestampedEntry extensions")
}
return nil
}
func (v *TimestampedEntry) Marshal(b *cryptobyte.Builder) error {
b.AddUint64(v.Timestamp)
b.AddValue(v.EntryType)
switch v.EntryType {
case X509EntryType:
b.AddValue(v.SignedEntryASN1Cert)
case PrecertEntryType:
b.AddValue(v.SignedEntryPreCert)
}
b.AddValue(v.Extensions)
return nil
}
func ParseLeafInput(leafInput []byte) (*MerkleTreeLeaf, error) {
str := cryptobyte.String(leafInput)
leaf := new(MerkleTreeLeaf)
if err := leaf.Unmarshal(&str); err != nil {
return nil, err
}
if !str.Empty() {
return nil, fmt.Errorf("trailing garbage after MerkleTreeLeaf")
}
return leaf, nil
}
func MerkleTreeLeafForCert(timestamp uint64, extensions []byte, cert ASN1Cert) *MerkleTreeLeaf {
return &MerkleTreeLeaf{
Version: V1,
LeafType: TimestampedEntryType,
TimestampedEntry: &TimestampedEntry{
Timestamp: timestamp,
EntryType: X509EntryType,
SignedEntryASN1Cert: &cert,
Extensions: extensions,
},
}
}
func MerkleTreeLeafForCertSCT(sct *SignedCertificateTimestamp, cert ASN1Cert) *MerkleTreeLeaf {
return &MerkleTreeLeaf{
Version: sct.SCTVersion,
LeafType: TimestampedEntryType,
TimestampedEntry: &TimestampedEntry{
Timestamp: sct.Timestamp,
EntryType: X509EntryType,
SignedEntryASN1Cert: &cert,
Extensions: sct.Extensions,
},
}
}
func MerkleTreeLeafForPrecert(timestamp uint64, extensions []byte, precert PreCert) *MerkleTreeLeaf {
return &MerkleTreeLeaf{
Version: V1,
LeafType: TimestampedEntryType,
TimestampedEntry: &TimestampedEntry{
Timestamp: timestamp,
EntryType: PrecertEntryType,
SignedEntryPreCert: &precert,
Extensions: extensions,
},
}
}
func MerkleTreeLeafForPrecertSCT(sct *SignedCertificateTimestamp, precert PreCert) *MerkleTreeLeaf {
return &MerkleTreeLeaf{
Version: sct.SCTVersion,
LeafType: TimestampedEntryType,
TimestampedEntry: &TimestampedEntry{
Timestamp: sct.Timestamp,
EntryType: PrecertEntryType,
SignedEntryPreCert: &precert,
Extensions: sct.Extensions,
},
}
}
third_party/certspotter/cttypes/sct.go
Vendored
+73
View File
@@ -0,0 +1,73 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"fmt"
"golang.org/x/crypto/cryptobyte"
"software.sslmate.com/src/certspotter/tlstypes"
)
type SignedCertificateTimestamp struct {
SCTVersion Version `json:"sct_version"`
ID LogID `json:"id"`
Timestamp uint64 `json:"timestamp"`
Extensions CTExtensions `json:"extensions"`
Signature tlstypes.DigitallySigned `json:"signature"`
}
func (sct *SignedCertificateTimestamp) Bytes() ([]byte, error) {
var builder cryptobyte.Builder
builder.AddValue(sct)
return builder.Bytes()
}
func (sct *SignedCertificateTimestamp) Marshal(b *cryptobyte.Builder) error {
b.AddValue(sct.SCTVersion)
b.AddValue(sct.ID)
b.AddUint64(sct.Timestamp)
b.AddValue(sct.Extensions)
b.AddValue(sct.Signature)
return nil
}
func (sct *SignedCertificateTimestamp) Unmarshal(s *cryptobyte.String) error {
if !sct.SCTVersion.Unmarshal(s) {
return fmt.Errorf("error reading SCT version")
}
if sct.SCTVersion != V1 {
return fmt.Errorf("unsupported SCT version 0x%02x", sct.SCTVersion)
}
if !sct.ID.Unmarshal(s) {
return fmt.Errorf("error reading SCT id")
}
if !s.ReadUint64(&sct.Timestamp) {
return fmt.Errorf("error reading SCT timestamp")
}
if !sct.Extensions.Unmarshal(s) {
return fmt.Errorf("error reading SCT extensions")
}
if !sct.Signature.Unmarshal(s) {
return fmt.Errorf("error reading SCT signature")
}
return nil
}
func ParseSignedCertificateTimestamp(data []byte) (*SignedCertificateTimestamp, error) {
str := cryptobyte.String(data)
sct := new(SignedCertificateTimestamp)
if err := sct.Unmarshal(&str); err != nil {
return nil, err
}
if !str.Empty() {
return nil, fmt.Errorf("trailing garbage after SignedCertificateTimestamp")
}
return sct, nil
}
third_party/certspotter/cttypes/signatures.go
Vendored
+29
View File
@@ -0,0 +1,29 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"golang.org/x/crypto/cryptobyte"
)
type SignatureType uint8
const (
CertificateTimestampSignatureType SignatureType = 0
TreeHashSignatureType SignatureType = 1
)
func (v *SignatureType) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint8((*uint8)(v))
}
func (v SignatureType) Marshal(b *cryptobyte.Builder) error {
b.AddUint8(uint8(v))
return nil
}
third_party/certspotter/cttypes/sth.go
Vendored
+37
View File
@@ -0,0 +1,37 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"software.sslmate.com/src/certspotter/merkletree"
"software.sslmate.com/src/certspotter/tlstypes"
"time"
)
type SignedTreeHead struct {
TreeSize uint64 `json:"tree_size"`
Timestamp uint64 `json:"timestamp"`
RootHash merkletree.Hash `json:"sha256_root_hash"`
Signature tlstypes.DigitallySigned `json:"tree_head_signature"`
}
type GossipedSignedTreeHead struct {
SignedTreeHead
STHVersion Version `json:"sth_version"`
LogID LogID `json:"log_id"`
}
func (sth *SignedTreeHead) TimestampTime() time.Time {
return time.UnixMilli(int64(sth.Timestamp))
}
func (sth *SignedTreeHead) Same(other *SignedTreeHead) bool {
return sth.TreeSize == other.TreeSize && sth.Timestamp == other.Timestamp && sth.RootHash == other.RootHash
}
third_party/certspotter/cttypes/version.go
Vendored
+29
View File
@@ -0,0 +1,29 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package cttypes
import (
"golang.org/x/crypto/cryptobyte"
)
type Version uint8
const (
V1 Version = 0
)
func (v Version) Marshal(b *cryptobyte.Builder) error {
b.AddUint8(uint8(v))
return nil
}
func (v *Version) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint8((*uint8)(v))
}
third_party/certspotter/go.mod
Vendored
+14
View File
@@ -0,0 +1,14 @@
module software.sslmate.com/src/certspotter
go 1.25.0
require (
golang.org/x/crypto v0.52.0
golang.org/x/net v0.55.0
golang.org/x/sync v0.20.0
golang.org/x/time v0.15.0
)
require golang.org/x/text v0.37.0 // indirect
retract v0.19.0 // Contains serious bugs.
third_party/certspotter/go.sum
Vendored
+22
View File
@@ -0,0 +1,22 @@
golang.org/x/crypto v0.39.0 h1:SHs+kF4LP+f+p14esP5jAoDpHU8Gu/v9lFRK6IT5imM=
golang.org/x/crypto v0.39.0/go.mod h1:L+Xg3Wf6HoL4Bn4238Z6ft6KfEpN0tJGo53AAPC632U=
golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI=
golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8=
golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988=
golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc=
golang.org/x/net v0.41.0 h1:vBTly1HeNPEn3wtREYfy4GZ/NECgw2Cnl+nK6Nz3uvw=
golang.org/x/net v0.41.0/go.mod h1:B/K4NNqkfmg07DQYrbwvSluqCJOOXwUjeb/5lOisjbA=
golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8=
golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
golang.org/x/sync v0.15.0 h1:KWH3jNZsfyT6xfAfKiz6MRNmd46ByHDYaZ7KSkCtdW8=
golang.org/x/sync v0.15.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc=
golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
third_party/certspotter/helpers.go
Vendored
+80
View File
@@ -0,0 +1,80 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"math/big"
)
type CertInfo struct {
TBS *TBSCertificate
Subject RDNSequence
SubjectParseError error
Issuer RDNSequence
IssuerParseError error
SANs []SubjectAltName
SANsParseError error
SerialNumber *big.Int
SerialNumberParseError error
Validity *CertValidity
ValidityParseError error
IsCA *bool
IsCAParseError error
IsPreCert bool
}
func MakeCertInfoFromTBS(tbs *TBSCertificate) *CertInfo {
info := &CertInfo{TBS: tbs}
info.Subject, info.SubjectParseError = tbs.ParseSubject()
info.Issuer, info.IssuerParseError = tbs.ParseIssuer()
info.SANs, info.SANsParseError = tbs.ParseSubjectAltNames()
info.SerialNumber, info.SerialNumberParseError = tbs.ParseSerialNumber()
info.Validity, info.ValidityParseError = tbs.ParseValidity()
info.IsCA, info.IsCAParseError = tbs.ParseBasicConstraints()
info.IsPreCert = len(tbs.GetExtension(oidExtensionCTPoison)) > 0
return info
}
func MakeCertInfoFromRawTBS(tbsBytes []byte) (*CertInfo, error) {
tbs, err := ParseTBSCertificate(tbsBytes)
if err != nil {
return nil, err
}
return MakeCertInfoFromTBS(tbs), nil
}
func MakeCertInfoFromRawCert(certBytes []byte) (*CertInfo, error) {
cert, err := ParseCertificate(certBytes)
if err != nil {
return nil, err
}
return MakeCertInfoFromRawTBS(cert.GetRawTBSCertificate())
}
func MatchesWildcard(dnsName string, pattern string) bool {
for len(pattern) > 0 {
if pattern[0] == '*' {
if len(dnsName) > 0 && dnsName[0] != '.' && MatchesWildcard(dnsName[1:], pattern) {
return true
}
pattern = pattern[1:]
} else {
if len(dnsName) == 0 || pattern[0] != dnsName[0] {
return false
}
pattern = pattern[1:]
dnsName = dnsName[1:]
}
}
return len(dnsName) == 0
}
third_party/certspotter/helpers_test.go
Vendored
+49
View File
@@ -0,0 +1,49 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"testing"
)
func doWildcardTest(t *testing.T, dnsName string, wildcard string, expected bool) {
if MatchesWildcard(dnsName, wildcard) != expected {
t.Errorf("MatchesWildcard(%q, %q) != %v", dnsName, wildcard, expected)
}
}
func TestMatchesWildcard(t *testing.T) {
doWildcardTest(t, "", "", true)
doWildcardTest(t, "example.com", "example.com", true)
doWildcardTest(t, "example.org", "example.com", false)
doWildcardTest(t, "example.com", "", false)
doWildcardTest(t, "", "example.com", false)
doWildcardTest(t, "", "*.example.com", false)
doWildcardTest(t, "", "exam*ple.com", false)
doWildcardTest(t, "", "exam*ple.co*m", false)
doWildcardTest(t, "example.org", "example.com", false)
doWildcardTest(t, "example.org", "*.example.com", false)
doWildcardTest(t, "example.org", "exam*ple.com", false)
doWildcardTest(t, "example.org", "exam*ple.co*m", false)
doWildcardTest(t, "example.com", "*.example.com", false)
doWildcardTest(t, "www.example.com", "*.example.com", true)
doWildcardTest(t, "", "*", true)
doWildcardTest(t, "", "****", true)
doWildcardTest(t, "a", "****", true)
doWildcardTest(t, "a", "*", true)
doWildcardTest(t, "a", "****", true)
doWildcardTest(t, "abcd", "****", true)
doWildcardTest(t, "abcdef", "****", true)
doWildcardTest(t, "www-example.com", "*-example.com", true)
doWildcardTest(t, "www-example-www.com", "*-example-*.com", true)
doWildcardTest(t, "examplecom", "example*", true)
doWildcardTest(t, "example.com", "example*", false)
doWildcardTest(t, "examplea.com", "example*", false)
}
third_party/certspotter/identifiers.go
Vendored
+301
View File
@@ -0,0 +1,301 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"bytes"
"golang.org/x/net/idna"
"net"
"strings"
"unicode/utf8"
)
const UnparsableDNSLabelPlaceholder = "<unparsable>"
/*
const (
IdentifierSourceSubjectCN = iota
IdentifierSourceDNSName
IdentifierSourceIPAddr
)
type IdentifierSource int
type UnknownIdentifier struct {
Source IdentifierSource
Value []byte
}
*/
type Identifiers struct {
DNSNames []string // stored as ASCII, with IDNs in Punycode
IPAddrs []net.IP
//Unknowns []UnknownIdentifier
}
func NewIdentifiers() *Identifiers {
return &Identifiers{
DNSNames: []string{},
IPAddrs: []net.IP{},
//Unknowns: []UnknownIdentifier{},
}
}
func parseIPAddrString(str string) net.IP {
return net.ParseIP(str)
}
func isASCIIString(value []byte) bool {
for _, b := range value {
if b > 127 {
return false
}
}
return true
}
func isUTF8String(value []byte) bool {
return utf8.Valid(value)
}
func latin1ToUTF8(value []byte) string {
runes := make([]rune, len(value))
for i, b := range value {
runes[i] = rune(b)
}
return string(runes)
}
// Make sure the DNS label doesn't have any weird characters that
// could cause trouble during later processing.
func isSaneDNSLabelChar(ch rune) bool {
return ch == '\t' || (ch >= 32 && ch <= 126)
}
func isSaneDNSLabel(label string) bool {
for _, ch := range label {
if !isSaneDNSLabelChar(ch) {
return false
}
}
return true
}
func trimHttpPrefixString(value string) string {
if strings.HasPrefix(value, "http://") {
return value[7:]
} else if strings.HasPrefix(value, "https://") {
return value[8:]
} else {
return value
}
}
func trimHttpPrefixBytes(value []byte) []byte {
if bytes.HasPrefix(value, []byte("http://")) {
return value[7:]
} else if bytes.HasPrefix(value, []byte("https://")) {
return value[8:]
} else {
return value
}
}
func trimTrailingDots(value string) string {
length := len(value)
for length > 0 && value[length-1] == '.' {
length--
}
return value[0:length]
}
// Try to canonicalize/sanitize the DNS name:
// 1. Trim leading and trailing whitespace
// 2. Trim trailing dots
// 3. Convert to lower case
// 4. Replace totally nonsensical labels (e.g. having non-printable characters) with a placeholder
func sanitizeDNSName(value string) string {
value = strings.ToLower(trimTrailingDots(strings.TrimSpace(value)))
labels := strings.Split(value, ".")
for i, label := range labels {
if !isSaneDNSLabel(label) {
labels[i] = UnparsableDNSLabelPlaceholder
}
}
return strings.Join(labels, ".")
}
// Like sanitizeDNSName, but labels that are Unicode are converted to Punycode.
func sanitizeUnicodeDNSName(value string) string {
value = strings.ToLower(trimTrailingDots(strings.TrimSpace(value)))
labels := strings.Split(value, ".")
for i, label := range labels {
if asciiLabel, err := idna.ToASCII(label); err == nil && isSaneDNSLabel(asciiLabel) {
labels[i] = asciiLabel
} else {
labels[i] = UnparsableDNSLabelPlaceholder
}
}
return strings.Join(labels, ".")
}
func (ids *Identifiers) appendDNSName(dnsName string) {
if dnsName != "" && !ids.hasDNSName(dnsName) {
ids.DNSNames = append(ids.DNSNames, dnsName)
}
}
func (ids *Identifiers) appendIPAddress(ipaddr net.IP) {
if !ids.hasIPAddress(ipaddr) {
ids.IPAddrs = append(ids.IPAddrs, ipaddr)
}
}
func (ids *Identifiers) hasDNSName(target string) bool {
for _, value := range ids.DNSNames {
if value == target {
return true
}
}
return false
}
func (ids *Identifiers) hasIPAddress(target net.IP) bool {
for _, value := range ids.IPAddrs {
if value.Equal(target) {
return true
}
}
return false
}
func (ids *Identifiers) addDnsSANfinal(value []byte) {
if ipaddr := parseIPAddrString(string(value)); ipaddr != nil {
// Stupid CAs put IP addresses in DNS SANs because stupid Microsoft
// used to not support IP address SANs. Since there's no way for an IP
// address to also be a valid DNS name, just treat it like an IP address
// and not try to process it as a DNS name.
ids.appendIPAddress(ipaddr)
} else if isASCIIString(value) {
ids.appendDNSName(sanitizeDNSName(string(value)))
} else {
// DNS SANs are supposed to be IA5Strings (i.e. ASCII) but CAs can't follow
// simple rules. Unfortunately, we have no idea what the encoding really is
// in this case, so interpret it as both UTF-8 (if it's valid UTF-8)
// and Latin-1.
if isUTF8String(value) {
ids.appendDNSName(sanitizeUnicodeDNSName(string(value)))
}
ids.appendDNSName(sanitizeUnicodeDNSName(latin1ToUTF8(value)))
}
}
func (ids *Identifiers) addDnsSANnonull(value []byte) {
if slashIndex := bytes.IndexByte(value, '/'); slashIndex != -1 {
// If the value contains a slash, then this might be a URL,
// so process the part of the value up to the first slash,
// which should be the domain. Even though no client should
// ever successfully validate such a DNS name, the domain owner
// might still want to know about it.
ids.addDnsSANfinal(value[0:slashIndex])
}
ids.addDnsSANfinal(value)
}
func (ids *Identifiers) AddDnsSAN(value []byte) {
// Trim http:// and https:// prefixes, which are all too common in the wild,
// so http://example.com becomes just example.com. Even though clients
// should never successfully validate a DNS name like http://example.com,
// the owner of example.com might still want to know about it.
value = trimHttpPrefixBytes(value)
if nullIndex := bytes.IndexByte(value, 0); nullIndex != -1 {
// If the value contains a null byte, process the part of
// the value up to the first null byte in addition to the
// complete value, in case this certificate is an attempt to
// fake out validators that only compare up to the first null.
ids.addDnsSANnonull(value[0:nullIndex])
}
ids.addDnsSANnonull(value)
}
func (ids *Identifiers) addCNfinal(value string) {
if ipaddr := parseIPAddrString(value); ipaddr != nil {
ids.appendIPAddress(ipaddr)
} else if !strings.ContainsRune(value, ' ') {
// If the CN contains a space it's clearly not a DNS name, so ignore it.
ids.appendDNSName(sanitizeUnicodeDNSName(value))
}
}
func (ids *Identifiers) addCNnonull(value string) {
if slashIndex := strings.IndexRune(value, '/'); slashIndex != -1 {
// If the value contains a slash, then this might be a URL,
// so process the part of the value up to the first slash,
// which should be the domain. Even though no client should
// ever successfully validate such a DNS name, the domain owner
// might still want to know about it.
ids.addCNfinal(value[0:slashIndex])
}
ids.addCNfinal(value)
}
func (ids *Identifiers) AddCN(value string) {
// Trim http:// and https:// prefixes, which are all too common in the wild,
// so http://example.com becomes just example.com. Even though clients
// should never successfully validate a DNS name like http://example.com,
// the owner of example.com might still want to know about it.
value = trimHttpPrefixString(value)
if nullIndex := strings.IndexRune(value, 0); nullIndex != -1 {
// If the value contains a null byte, process the part of
// the value up to the first null byte in addition to the
// complete value, in case this certificate is an attempt to
// fake out validators that only compare up to the first null.
ids.addCNnonull(value[0:nullIndex])
}
ids.addCNnonull(value)
}
func (ids *Identifiers) AddIPAddress(value net.IP) {
ids.appendIPAddress(value)
}
func (cert *CertInfo) ParseIdentifiers() (*Identifiers, error) {
ids := NewIdentifiers()
if cert.SubjectParseError != nil {
return nil, cert.SubjectParseError
}
cns, err := cert.Subject.ParseCNs()
if err != nil {
return nil, err
}
for _, cn := range cns {
ids.AddCN(cn)
}
if cert.SANsParseError != nil {
return nil, cert.SANsParseError
}
for _, san := range cert.SANs {
switch san.Type {
case sanDNSName:
ids.AddDnsSAN(san.Value)
case sanIPAddress:
if len(san.Value) == 4 || len(san.Value) == 16 {
ids.AddIPAddress(net.IP(san.Value))
}
// TODO: decide what to do with IP addresses with an invalid length.
// The two encoding errors I've observed in CT logs are:
// 1. encoding the IP address as a string
// 2. a value of 0x00000000FFFFFF00 (WTF?)
// IP addresses aren't a high priority so just ignore invalid ones for now.
// Hopefully no clients out there are dumb enough to process IP address
// SANs encoded as strings...
}
}
return ids, nil
}
third_party/certspotter/loglist/helpers.go
Vendored
+40
View File
@@ -0,0 +1,40 @@
// Copyright (C) 2020 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package loglist
import (
"time"
)
// Return all tiled and non-tiled logs from all operators
func (list *List) AllLogs() []*Log {
logs := []*Log{}
for operator := range list.Operators {
for log := range list.Operators[operator].Logs {
logs = append(logs, &list.Operators[operator].Logs[log])
}
for log := range list.Operators[operator].TiledLogs {
logs = append(logs, &list.Operators[operator].TiledLogs[log])
}
}
return logs
}
func (log *Log) LogIDString() string {
return log.LogID.Base64String()
}
func (log *Log) AcceptsExpiration(expiration time.Time) bool {
return log.TemporalInterval == nil || withinInterval(expiration, log.TemporalInterval.StartInclusive, log.TemporalInterval.EndExclusive)
}
func withinInterval(expiration, startInclusive, endExclusive time.Time) bool {
return !expiration.Before(startInclusive) && expiration.Before(endExclusive)
}
third_party/certspotter/loglist/load.go
Vendored
+129
View File
@@ -0,0 +1,129 @@
// Copyright (C) 2020, 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package loglist
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
var UserAgent = ""
// maxResponseBytes bounds the size of a log list we are willing to read into memory.
const maxResponseBytes = 16 << 20 // 16 MiB
// httpClient is used for fetching log lists.
var httpClient = &http.Client{
Timeout: 60 * time.Second,
}
type ModificationToken struct {
etag string
modified time.Time
}
var ErrNotModified = errors.New("loglist has not been modified")
func newModificationToken(response *http.Response) *ModificationToken {
token := &ModificationToken{
etag: response.Header.Get("ETag"),
}
if t, err := time.Parse(http.TimeFormat, response.Header.Get("Last-Modified")); err == nil {
token.modified = t
}
return token
}
func (token *ModificationToken) setRequestHeaders(request *http.Request) {
if token.etag != "" {
request.Header.Set("If-None-Match", token.etag)
} else if !token.modified.IsZero() {
request.Header.Set("If-Modified-Since", token.modified.Format(http.TimeFormat))
}
}
func Load(ctx context.Context, urlOrFile string) (*List, error) {
list, _, err := LoadIfModified(ctx, urlOrFile, nil)
return list, err
}
func LoadIfModified(ctx context.Context, urlOrFile string, token *ModificationToken) (*List, *ModificationToken, error) {
if strings.HasPrefix(urlOrFile, "https://") {
return FetchIfModified(ctx, urlOrFile, token)
} else {
list, err := ReadFile(urlOrFile)
return list, nil, err
}
}
func Fetch(ctx context.Context, url string) (*List, error) {
list, _, err := FetchIfModified(ctx, url, nil)
return list, err
}
func FetchIfModified(ctx context.Context, url string, token *ModificationToken) (*List, *ModificationToken, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, nil, err
}
request.Header.Set("User-Agent", UserAgent)
if token != nil {
token.setRequestHeaders(request)
}
response, err := httpClient.Do(request)
if err != nil {
return nil, nil, err
}
content, err := io.ReadAll(io.LimitReader(response.Body, maxResponseBytes+1))
response.Body.Close()
if err != nil {
return nil, nil, err
}
if len(content) > maxResponseBytes {
return nil, nil, fmt.Errorf("%s: response body exceeds maximum allowed size of %d bytes", url, maxResponseBytes)
}
if token != nil && response.StatusCode == http.StatusNotModified {
return nil, nil, ErrNotModified
}
if response.StatusCode != 200 {
return nil, nil, fmt.Errorf("%s: %s", url, response.Status)
}
list, err := Unmarshal(content)
if err != nil {
return nil, nil, fmt.Errorf("error parsing %s: %w", url, err)
}
return list, newModificationToken(response), err
}
func ReadFile(filename string) (*List, error) {
content, err := os.ReadFile(filename)
if err != nil {
return nil, err
}
return Unmarshal(content)
}
func Unmarshal(jsonBytes []byte) (*List, error) {
list := new(List)
if err := json.Unmarshal(jsonBytes, list); err != nil {
return nil, err
}
if err := list.Validate(); err != nil {
return nil, fmt.Errorf("invalid log list: %s", err)
}
return list, nil
}
third_party/certspotter/loglist/mozilla/ctlogs.go
Vendored
+275
View File
@@ -0,0 +1,275 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
// Package mozilla contains a parser for Mozilla's CTKnownLogs.h file
package mozilla
import (
"bufio"
"errors"
"io"
"strconv"
"strings"
"time"
)
// CTLogInfo describes a certificate transparency log from Mozilla's CTKnownLogs.h
// file.
type CTLogInfo struct {
Name string
State string // Admissible or Retired
Protocol string // RFC6962 or Tiled
Timestamp time.Time
OperatorIndex int
Key []byte
}
// CTLogOperatorInfo describes a CT log operator from Mozilla's CTKnownLogs.h
// file.
type CTLogOperatorInfo struct {
Name string
ID int
}
// Parse reads the CTKnownLogs.h content from r and returns the parsed logs and
// operators. Blocks enclosed by `#ifdef DEBUG` and `#endif` are ignored.
func Parse(r io.Reader) ([]CTLogInfo, []CTLogOperatorInfo, error) {
scanner := bufio.NewScanner(r)
skip := 0
inLogs := false
inOps := false
var logs []CTLogInfo
var ops []CTLogOperatorInfo
for scanner.Scan() {
line := scanner.Text()
trimmed := strings.TrimSpace(line)
switch {
case strings.HasPrefix(trimmed, "#ifdef DEBUG"):
skip++
continue
case strings.HasPrefix(trimmed, "#endif"):
if skip > 0 {
skip--
continue
}
}
if skip > 0 {
continue
}
if strings.HasPrefix(trimmed, "const CTLogInfo kCTLogList[]") {
inLogs = true
continue
}
if strings.HasPrefix(trimmed, "const CTLogOperatorInfo kCTLogOperatorList[]") {
inOps = true
continue
}
if inLogs {
if trimmed == "};" {
inLogs = false
continue
}
if strings.HasPrefix(trimmed, "{") {
log, err := readLogEntry(trimmed, scanner)
if err != nil {
return nil, nil, err
}
logs = append(logs, log)
}
continue
}
if inOps {
if trimmed == "};" {
inOps = false
continue
}
if strings.HasPrefix(trimmed, "{") {
op, err := readOperatorEntry(trimmed)
if err != nil {
return nil, nil, err
}
ops = append(ops, op)
}
}
}
if err := scanner.Err(); err != nil {
return nil, nil, err
}
return logs, ops, nil
}
func readLogEntry(firstLine string, s *bufio.Scanner) (CTLogInfo, error) {
var log CTLogInfo
// Example first line:
// {"Name", CTLogState::Admissible, CTLogFormat::RFC6962,
firstLine = strings.TrimSpace(firstLine)
if !strings.HasPrefix(firstLine, "{") {
return log, errors.New("invalid log entry start")
}
firstLine = strings.TrimPrefix(firstLine, "{")
firstLine = strings.TrimSuffix(firstLine, ",")
parts := splitCSV(firstLine)
for len(parts) < 3 {
if !s.Scan() {
return log, io.ErrUnexpectedEOF
}
next := strings.TrimSpace(strings.TrimSuffix(s.Text(), ","))
parts = append(parts, splitCSV(next)...)
}
log.Name = trimQuotes(parts[0])
log.State = strings.TrimPrefix(strings.TrimSpace(parts[1]), "CTLogState::")
log.Protocol = strings.TrimPrefix(strings.TrimSpace(parts[2]), "CTLogFormat::")
// Next line: timestamp
if !s.Scan() {
return log, io.ErrUnexpectedEOF
}
tsLine := strings.TrimSpace(s.Text())
tsValue := strings.Split(tsLine, ",")[0]
ts, err := strconv.ParseInt(strings.TrimSpace(tsValue), 10, 64)
if err != nil {
return log, err
}
log.Timestamp = time.Unix(0, ts*int64(time.Millisecond))
// Next line: operator index
if !s.Scan() {
return log, io.ErrUnexpectedEOF
}
opLine := strings.TrimSpace(s.Text())
opValue := strings.Split(opLine, ",")[0]
opIndex, err := strconv.Atoi(strings.TrimSpace(opValue))
if err != nil {
return log, err
}
log.OperatorIndex = opIndex
// Key lines
var keyHex strings.Builder
for {
if !s.Scan() {
return log, io.ErrUnexpectedEOF
}
l := strings.TrimSpace(s.Text())
if strings.HasPrefix(l, "\"") {
// remove trailing comma if any
trimmed := strings.TrimSuffix(l, ",")
keyHex.WriteString(trimQuotes(trimmed))
if strings.HasSuffix(l, ",") {
// last key line
break
}
continue
}
return log, errors.New("unexpected line while reading key")
}
key, err := decodeHexEscapes(keyHex.String())
if err != nil {
return log, err
}
// key length line
if !s.Scan() {
return log, io.ErrUnexpectedEOF
}
lenLine := strings.TrimSpace(s.Text())
lenValue := strings.TrimSuffix(lenLine, "},")
keyLen, err := strconv.Atoi(strings.TrimSpace(lenValue))
if err != nil {
return log, err
}
if len(key) != keyLen {
// ignore mismatch but continue
}
log.Key = key
return log, nil
}
func readOperatorEntry(line string) (CTLogOperatorInfo, error) {
var op CTLogOperatorInfo
line = strings.TrimSuffix(strings.TrimSpace(line), ",")
if !strings.HasPrefix(line, "{") || !strings.HasSuffix(line, "}") {
return op, errors.New("invalid operator entry")
}
line = strings.TrimPrefix(line, "{")
line = strings.TrimSuffix(line, "}")
parts := splitCSV(line)
if len(parts) != 2 {
return op, errors.New("invalid operator fields")
}
op.Name = trimQuotes(parts[0])
id, err := strconv.Atoi(strings.TrimSpace(parts[1]))
if err != nil {
return op, err
}
op.ID = id
return op, nil
}
func splitCSV(s string) []string {
var parts []string
var cur strings.Builder
inQuote := false
for i := 0; i < len(s); i++ {
c := s[i]
if c == '"' {
inQuote = !inQuote
cur.WriteByte(c)
continue
}
if c == ',' && !inQuote {
parts = append(parts, strings.TrimSpace(cur.String()))
cur.Reset()
continue
}
cur.WriteByte(c)
}
if cur.Len() > 0 {
parts = append(parts, strings.TrimSpace(cur.String()))
}
return parts
}
func trimQuotes(s string) string {
s = strings.TrimSpace(s)
if len(s) >= 2 && s[0] == '"' && s[len(s)-1] == '"' {
return s[1 : len(s)-1]
}
return s
}
func decodeHexEscapes(s string) ([]byte, error) {
var out []byte
for i := 0; i < len(s); {
if i+3 >= len(s) || s[i] != '\\' || s[i+1] != 'x' {
return nil, errors.New("invalid escape")
}
b, err := strconv.ParseUint(s[i+2:i+4], 16, 8)
if err != nil {
return nil, err
}
out = append(out, byte(b))
i += 4
}
return out, nil
}
third_party/certspotter/loglist/mozilla/ctlogs_test.go
Vendored
+87
View File
@@ -0,0 +1,87 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package mozilla
import (
"crypto/sha256"
"encoding/base64"
"errors"
"net/http"
"testing"
)
// parseFromURL downloads the CTKnownLogs.h file from the given URL and parses it.
func parseFromURL(url string) ([]CTLogInfo, []CTLogOperatorInfo, error) {
resp, err := http.Get(url)
if err != nil {
return nil, nil, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, nil, errors.New(resp.Status)
}
return Parse(resp.Body)
}
func TestParseFromURL(t *testing.T) {
logs, ops, err := parseFromURL("https://hg-edge.mozilla.org/mozilla-central/raw-file/tip/security/ct/CTKnownLogs.h")
if err != nil {
t.Fatal(err)
}
if len(ops) == 0 {
t.Fatal("no operators parsed")
}
foundGoogle := false
foundSectigo := false
foundLets := false
for _, op := range ops {
if op.Name == "" {
t.Error("operator with empty name")
}
switch op.Name {
case "Google":
foundGoogle = true
case "Sectigo":
foundSectigo = true
case "Let's Encrypt":
foundLets = true
}
}
if !foundGoogle || !foundSectigo || !foundLets {
t.Errorf("missing expected operators: Google=%v Sectigo=%v Let's=%v", foundGoogle, foundSectigo, foundLets)
}
if len(logs) == 0 {
t.Fatal("no logs parsed")
}
foundHash := false
targetHash := "1219ENGn9XfCx+lf1wC/+YLJM1pl4dCzAXMXwMjFaXc="
for _, l := range logs {
if l.Name == "" {
t.Error("log with empty name")
}
if len(l.Key) == 0 {
t.Error("log with empty key")
}
if l.State != "Admissible" && l.State != "Retired" {
t.Errorf("unexpected state %q", l.State)
}
if l.Protocol != "RFC6962" && l.Protocol != "Tiled" {
t.Errorf("unexpected protocol %q", l.Protocol)
}
hash := sha256.Sum256(l.Key)
if base64.StdEncoding.EncodeToString(hash[:]) == targetHash {
foundHash = true
}
}
if !foundHash {
t.Errorf("log with key hash %s not found", targetHash)
}
}
third_party/certspotter/loglist/mozilla/example_test.go
Vendored
+55
View File
@@ -0,0 +1,55 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package mozilla_test
import (
"crypto/sha256"
"encoding/base64"
"fmt"
"log"
"net/http"
"os"
"text/tabwriter"
"time"
"software.sslmate.com/src/certspotter/loglist/mozilla"
)
func ExampleParse() {
resp, err := http.Get("https://hg-edge.mozilla.org/mozilla-central/raw-file/tip/security/ct/CTKnownLogs.h")
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
log.Fatal(resp.Status)
}
logs, operators, err := mozilla.Parse(resp.Body)
if err != nil {
log.Fatal(err)
}
tw := tabwriter.NewWriter(os.Stdout, 0, 8, 1, ' ', 0)
fmt.Fprintln(tw, "Operator\tName")
for _, o := range operators {
fmt.Fprintf(tw, "%d\t%s\n", o.ID, o.Name)
}
tw.Flush()
tw = tabwriter.NewWriter(os.Stdout, 0, 8, 1, ' ', 0)
fmt.Fprintln(tw, "LogID\tProtocol\tState\tTimestamp\tOperator\tName")
for _, l := range logs {
hash := sha256.Sum256(l.Key)
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%d\t%s\n",
base64.StdEncoding.EncodeToString(hash[:]),
l.Protocol, l.State, l.Timestamp.UTC().Format(time.RFC3339), l.OperatorIndex, l.Name)
}
tw.Flush()
}
third_party/certspotter/loglist/schema.go
Vendored
+158
View File
@@ -0,0 +1,158 @@
// Copyright (C) 2020 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package loglist
import (
"time"
"software.sslmate.com/src/certspotter/cttypes"
)
type List struct {
Version string `json:"version,omitzero"`
LogListTimestamp time.Time `json:"log_list_timestamp,omitzero"` // Only present in v3 of schema
Operators []Operator `json:"operators,omitzero"`
}
type Operator struct {
Name string `json:"name"`
Email []string `json:"email,omitzero"`
Logs []Log `json:"logs,omitzero"`
TiledLogs []Log `json:"tiled_logs,omitzero"`
}
// Log represents the log object in the "v3" JSON log list schema, as published by Chrome, Apple, and SSLMate.
type Log struct {
Key []byte `json:"key"`
LogID cttypes.LogID `json:"log_id"`
MMD int `json:"mmd"`
URL string `json:"url,omitzero"` // only for rfc6962 logs
SubmissionURL string `json:"submission_url,omitzero"` // only for static-ct-api logs
MonitoringURL string `json:"monitoring_url,omitzero"` // only for static-ct-api logs
Description string `json:"description"`
State State `json:"state,omitzero"`
LogType LogType `json:"log_type,omitzero"`
TemporalInterval *struct {
StartInclusive time.Time `json:"start_inclusive"`
EndExclusive time.Time `json:"end_exclusive"`
} `json:"temporal_interval,omitzero"`
LogSoftware *struct {
Name string `json:"name"`
Version string `json:"version"`
} `json:"log_software,omitzero"`
// certspotter-specific extensions
CertspotterDownloadSize int `json:"certspotter_download_size,omitzero"`
CertspotterDownloadJobs int `json:"certspotter_download_jobs,omitzero"`
CertspotterDownloadQPS float64 `json:"certspotter_download_qps,omitzero"`
// TODO: add previous_operators
}
func (log *Log) IsRFC6962() bool { return log.URL != "" }
func (log *Log) IsStaticCTAPI() bool { return log.SubmissionURL != "" && log.MonitoringURL != "" }
// Return URL prefix for submission using the RFC6962 protocol
func (log *Log) GetSubmissionURL() string {
if log.SubmissionURL != "" {
return log.SubmissionURL
} else {
return log.URL
}
}
// Return URL prefix for monitoring.
// Since the protocol understood by the URL might be either RFC6962 or static-ct-api, this URL is
// only useful for informational purposes.
func (log *Log) GetMonitoringURL() string {
if log.MonitoringURL != "" {
return log.MonitoringURL
} else {
return log.URL
}
}
type State struct {
Pending *struct {
Timestamp time.Time `json:"timestamp"`
} `json:"pending,omitzero"`
Qualified *struct {
Timestamp time.Time `json:"timestamp"`
} `json:"qualified,omitzero"`
Usable *struct {
Timestamp time.Time `json:"timestamp"`
} `json:"usable,omitzero"`
Readonly *struct {
Timestamp time.Time `json:"timestamp"`
FinalTreeHead struct {
TreeSize int64 `json:"tree_size"`
SHA256RootHash []byte `json:"sha256_root_hash"`
} `json:"final_tree_head"`
} `json:"readonly,omitzero"`
Retired *struct {
Timestamp time.Time `json:"timestamp"`
} `json:"retired,omitzero"`
Rejected *struct {
Timestamp time.Time `json:"timestamp"`
} `json:"rejected,omitzero"`
}
func (state *State) IsApproved() bool {
return state.Qualified != nil || state.Usable != nil || state.Readonly != nil
}
func (state *State) WasApprovedAt(t time.Time) bool {
return state.Retired != nil && t.Before(state.Retired.Timestamp)
}
type LogType string
const (
LogTypeProd = "prod"
LogTypeTest = "test"
)
type LogSpec string
const (
StaticCTAPI = "static-ct-api"
RFC6962 = "rfc6962"
)
// InclusionRequest represents the "v2" JSON log object, as published by log operators to request inclusion in browsers.
type InclusionRequest struct {
Key []byte `json:"key"`
LogID cttypes.LogID `json:"log_id"`
MMDSeconds int `json:"mmd_seconds"`
FriendlyName string `json:"friendly_name"`
LogSpec LogSpec `json:"log_spec"`
Endpoint struct {
URL string `json:"url"`
} `json:"endpoint,omitzero"`
SubmissionEndpoint struct {
URL string `json:"url"`
} `json:"submission_endpoint,omitzero"`
MonitoringEndpoint struct {
URL string `json:"url"`
} `json:"monitoring_endpoint,omitzero"`
TemporalInterval struct {
StartInclusive time.Time `json:"start_inclusive"`
EndExclusive time.Time `json:"end_exclusive"`
} `json:"temporal_interval"`
LogSoftware struct {
Name string `json:"name"`
Version string `json:"version"`
} `json:"log_software"`
}
third_party/certspotter/loglist/validate.go
Vendored
+57
View File
@@ -0,0 +1,57 @@
// Copyright (C) 2020 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package loglist
import (
"crypto/sha256"
"fmt"
)
func (list *List) Validate() error {
for i := range list.Operators {
if err := list.Operators[i].Validate(); err != nil {
return fmt.Errorf("problem with %dth operator (%s): %w", i, list.Operators[i].Name, err)
}
}
return nil
}
func (operator *Operator) Validate() error {
for i := range operator.Logs {
if err := operator.Logs[i].Validate(); err != nil {
return fmt.Errorf("problem with %dth non-tiled log (%s): %w", i, operator.Logs[i].LogIDString(), err)
}
}
for i := range operator.TiledLogs {
if err := operator.TiledLogs[i].Validate(); err != nil {
return fmt.Errorf("problem with %dth tiled log (%s): %w", i, operator.TiledLogs[i].LogIDString(), err)
}
}
return nil
}
func (log *Log) Validate() error {
realLogID := sha256.Sum256(log.Key)
if log.LogID != realLogID {
return fmt.Errorf("log ID does not match log key")
}
if !log.IsRFC6962() && !log.IsStaticCTAPI() {
return fmt.Errorf("URL(s) not provided")
} else if log.IsRFC6962() && log.IsStaticCTAPI() {
return fmt.Errorf("inconsistent URLs provided")
}
if log.MMD < 0 {
return fmt.Errorf("log has a negative MMD")
}
return nil
}
third_party/certspotter/man/.gitignore
+1
View File
@@ -0,0 +1 @@
*.8
third_party/certspotter/man/Makefile
Vendored
+9
View File
@@ -0,0 +1,9 @@
all: certspotter-script.8 certspotter.8 certspotter-authorize.8 certspotter-metrics.8
%.8: %.md
lowdown -s -Tman \
-M title:$(basename $(notdir $@)) \
-M section:$(subst .,,$(suffix $@)) \
-M date:$(if $(SOURCE_DATE_EPOCH),$(shell date -I -u -d "@$(SOURCE_DATE_EPOCH)"),$(shell date -I -u)) \
-o $@ $<
third_party/certspotter/man/certspotter-authorize.md
Vendored
+105
View File
@@ -0,0 +1,105 @@
# NAME
**certspotter-authorize** - Authorize certificates to suppress certspotter notifications
# SYNOPSIS
**certspotter-authorize** [`-printhash`] *CERTFILE*...
**certspotter-authorize** [`-printhash`] `-`
# DESCRIPTION
**certspotter-authorize** is a utility for preemptively authorizing
certificates so that **certspotter(8)** will not send notifications when
those certificates are discovered in Certificate Transparency logs.
This is useful for preventing false alarms when you know in advance that a
certificate will be issued. For example, you might run **certspotter-authorize**
immediately after receiving a certificate from your certificate authority, as
part of your certificate issuance pipeline.
**certspotter-authorize** uses the TBSCertificate hash as defined by RFC
6962 Section 3.2 to identify certificates. This hash is the same for a
certificate and its corresponding precertificate. This means authorizing
the certificate will suppress notifications for the precertificate
as well. Certificates with different serial numbers, validity periods,
or other changes to the TBSCertificate will not be covered by the authorization
and will trigger notifications.
# ARGUMENTS
**certspotter-authorize** takes paths to one or more PEM-encoded certificates
on the command line. If `-` is specified as the sole path, it reads the certificate
from stdin.
# OPTIONS
-state\_dir *PATH*
: Directory where certspotter stores state. Defaults to
`$CERTSPOTTER_STATE_DIR` if set, or `~/.certspotter` otherwise.
This should be the same directory used by **certspotter(8)**.
-printhash
: Instead of authorizing the certificate, print its TBS hash (a hex-encoded SHA-256 digest) to stdout.
-version
: Print version information and exit.
# EXAMPLES
Authorize a certificate from a file:
$ certspotter-authorize /path/to/cert.pem
Authorize multiple certificates from files:
$ certspotter-authorize cert-a.pem cert-b.pem
Authorize a certificate from stdin:
$ cat cert.pem | certspotter-authorize -
Authorize a certificate in a custom state directory:
$ certspotter-authorize -state_dir /var/lib/certspotter cert.pem
Print the TBS hash without authorizing:
$ certspotter-authorize -printhash /path/to/cert.pem
# OPERATION
**certspotter-authorize** computes the SHA-256 hash of each
certificate's TBSCertificate as defined by RFC 6962 Section 3.2 and
creates a `.notified` marker file in the certspotter
state directory. When certspotter later discovers a certificate with
the same TBSCertificate in a CT log, it will skip sending notifications
because the marker file is present.
# ENVIRONMENT
`CERTSPOTTER_STATE_DIR`
: Directory for storing state. Overridden by `-state_dir`. Defaults to
`~/.certspotter`. This should be the same directory used by **certspotter(8)**.
# FILES
`$CERTSPOTTER_STATE_DIR/certs/XX/.HASH.notified`
: Marker files indicating that a certificate with TBS hash `HASH`
has been authorized. `XX` is the first two hex digits of the hash.
The file is empty; only its existence is checked.
# EXIT STATUS
**certspotter-authorize** exits with status 0 on success, 1 on error, or 2 on
invalid usage.
# SEE ALSO
certspotter(8), certspotter-script(8)
third_party/certspotter/man/certspotter-metrics.md
Vendored
+128
View File
@@ -0,0 +1,128 @@
# NAME
**certspotter-metrics** - Output Prometheus metrics about certspotter's operation
# SYNOPSIS
**certspotter-metrics** [`-state_dir` *PATH*]
# DESCRIPTION
**certspotter-metrics** reads a certspotter state directory and writes
Prometheus metrics, in the Prometheus text exposition format, about
certspotter's progress monitoring each Certificate Transparency log.
Metrics are written to standard output.
The intended use is to run **certspotter-metrics** periodically (e.g., from
cron) and expose its output to Prometheus, for example with the
prometheus-node-exporter(1) textfile collector.
**certspotter-metrics** only reads the state directory; it never modifies it.
It is safe to run while certspotter is running, and it does not require
certspotter to be running.
Metrics are output for every log for which certspotter has a state directory
(`$STATE_DIR/logs/LOG_ID`, where `LOG_ID` is the base64url-encoded log ID).
A log has a state directory once certspotter has started monitoring it.
The `log_id` label on each metric is the standard base64 encoding (with
padding) of the 32-byte log ID defined by RFC 6962. See <https://sslmate.com/app/ctlogs>
for a list of human-friendly names for each log ID.
**certspotter-metrics** is experimental, and the metrics may change in a future release.
# OPTIONS
-state\_dir *PATH*
: Directory where certspotter stores state. Defaults to
`$CERTSPOTTER_STATE_DIR` if set, or `~/.certspotter` otherwise.
This should be the same directory used by **certspotter(8)**.
-version
: Print version information and exit.
# METRICS
All metrics are gauges with a `log_id` label identifying the log the
metric pertains to. If a metric does not pertain to a log (because
certspotter has not stored the relevant state for the log yet), the log
does not have a sample for that metric.
certspotter\_log\_max\_sth\_size
: Tree size of the largest signed tree head (STH) certspotter has observed
from the log; in other words, the number of entries in the log. Absent
if certspotter has not yet downloaded an STH from the log.
certspotter\_log\_max\_sth\_timestamp
: Timestamp, in milliseconds since the Epoch, of the STH reported by
certspotter\_log\_max\_sth\_size.
certspotter\_log\_download\_position
: Number of entries certspotter has downloaded from the log, including
entries that have not yet been verified.
certspotter\_log\_verified\_position
: Number of entries certspotter has downloaded from the log and verified
against an STH.
certspotter\_log\_verified\_sth\_timestamp
: Timestamp, in milliseconds since the Epoch, of the most recent STH
certspotter has verified for the log. Absent if certspotter has not
verified an STH for the log. The size of this STH is equal to
certspotter\_log\_verified\_position.
certspotter\_log\_unverified\_sths
: Number of unverified STHs certspotter is currently storing for the log.
certspotter\_log\_malformed\_entries
: Number of malformed entries certspotter has recorded for the log.
certspotter records an entry as malformed when it cannot parse the
entry. See **certspotter(8)** for more information.
certspotter\_log\_health\_check\_failures
: Number of failed health checks certspotter has recorded for the log.
See **certspotter(8)** for information about health checks.
# EXAMPLES
crontab(5) entry that exposes metrics to the node\_exporter textfile collector
every five minutes (writing to a temporary file and renaming it
ensures that node\_exporter never reads a partially-written file):
*/5 * * * * root certspotter-metrics > /var/lib/node_exporter/certspotter.prom.tmp && mv /var/lib/node_exporter/certspotter.prom.tmp /var/lib/node_exporter/certspotter.prom
Prometheus query to determine download backlog (number entries in the
log that haven't been downloaded yet):
certspotter_log_max_sth_size-certspotter_log_download_position
# ENVIRONMENT
`CERTSPOTTER_STATE_DIR`
: Directory for storing state. Overridden by `-state_dir`. Defaults to
`~/.certspotter`. This should be the same directory used by
**certspotter(8)**.
`STATE_DIRECTORY`
: Directory for storing state, if set and certspotter-metrics is running
under systemd.
# EXIT STATUS
**certspotter-metrics** exits with status 0 on success, 1 on error, or 2 on
invalid usage.
# SEE ALSO
certspotter(8)
third_party/certspotter/man/certspotter-script.md
Vendored
+296
View File
@@ -0,0 +1,296 @@
# NAME
**certspotter-script** - Certificate Transparency Log Monitor (hook script)
# DESCRIPTION
**certspotter-script** is *any* program that is executed by **certspotter(8)**
when it needs to notify you about an event, such as detecting a certificate for
a domain on your watch list.
Scripts are placed in the `$CERTSPOTTER_CONFIG_DIR/hooks.d` directory
(`~/.certspotter/hooks.d` by default), or specified on the command line
using the `-script` argument.
# ENVIRONMENT
## Event information
The following environment variables are set for all types of events:
`EVENT`
: One of the following values, indicating the type of event:
* `discovered_cert` - certspotter has discovered a certificate for a
domain on your watch list.
* `malformed_cert` - certspotter can't determine if a certificate
matches your watch list because the certificate or the log entry
is malformed.
* `error` - a problem is preventing certspotter from monitoring all
logs.
Additional event types may be defined in the future, so your script should
be able to handle unknown values.
`SUMMARY`
: A short human-readable string describing the event. This is the same string
used in the subject line of emails sent by certspotter.
## Discovered certificate information
The following environment variables are set for `discovered_cert` events:
`WATCH_ITEM`
: The item from your watch list which matches this certificate.
(If more than one item matches, the first one is used.)
`LOG_URI`
: The URI of the log containing the certificate.
`ENTRY_INDEX`
: The index of the log entry containing the certificate.
`TBS_SHA256`
: The hex-encoded SHA-256 digest of the TBSCertificate, as defined in RFC 6962 Section 3.2.
Certificates and their corresponding precertificates have the same `TBS_SHA256` value.
You can compute the TBS digest of a certificate using the `-printhash` option of the certspotter-authorize(8) command.
`CERT_SHA256`
: The hex-encoded SHA-256 digest (sometimes called fingerprint) of the certificate or precertificate.
The digest is computed over the ASN.1 DER encoding.
`PUBKEY_SHA256`
: The hex-encoded SHA-256 digest of the certificate's Subject Public Key Info.
`CERT_FILENAME`
: Path to a file containing the PEM-encoded certificate chain. When `-no_save` is used, this points to a temporary file that is automatically deleted after your script completes.
`JSON_FILENAME`
: Path to a JSON file containing additional information about the certificate. See below for the format of the JSON file.
When `-no_save` is used, this points to a temporary file that is automatically deleted after your script completes.
`TEXT_FILENAME`
: Path to a text file containing information about the certificate. This file contains the same text that
certspotter uses in emails. You should not attempt to parse this file because its format may change in the future.
When `-no_save` is used, this points to a temporary file that is automatically deleted after your script completes.
`NOT_BEFORE`, `NOT_BEFORE_UNIXTIME`, `NOT_BEFORE_RFC3339`
: The not before time of the certificate, in a human-readable format, seconds since the UNIX epoch, and RFC3339, respectively. These variables may be unset if there was a parse error, in which case `VALIDITY_PARSE_ERROR` is set.
`NOT_AFTER`, `NOT_AFTER_UNIXTIME`, `NOT_AFTER_RFC3339`
: The not after (expiration) time of the certificate, in a human-readable format, seconds since the UNIX epoch, and RFC3339, respectively. These variables may be unset if there was a parse error, in which case `VALIDITY_PARSE_ERROR` is set.
`VALIDITY_PARSE_ERROR`
: Error parsing not before and not after, if any. If this variable is set, then the `NOT_BEFORE` and `NOT_AFTER` family of variables are unset.
`SUBJECT_DN`
: The distinguished name of the certificate's subject. This variable may be unset if there was a parse error, in which case `SUBJECT_PARSE_ERROR` is set.
`SUBJECT_PARSE_ERROR`
: Error parsing the subject, if any. If this variable is set, then `SUBJECT_DN` is unset.
`ISSUER_DN`
: The distinguished name of the certificate's issuer. This variable may be unset if there was a parse error, in which case `ISSUER_PARSE_ERROR` is set. Be aware that this value is controlled by the certificate authority and often contains [misleading, incorrect, or unhelpful information](https://www.agwa.name/blog/post/the_certificate_issuer_field_is_a_lie).
`ISSUER_PARSE_ERROR`
: Error parsing the issuer, if any. If this variable is set, then `ISSUER_DN` is unset.
`SERIAL`
: The hex-encoded serial number of the certificate. Prefixed with a minus (-) sign if negative. This variable may be unset if there was a parse error, in which case `SERIAL_PARSE_ERROR` is set. Be aware that malicious certificate authorities can issue certificates with duplicate serial numbers, so it is not safe to use the serial number to make security decisions (such as to determine if a certificate is legitimate or not).
`SERIAL_PARSE_ERROR`
: Error parsing the serial number, if any. If this variable is set, then `SERIAL` is unset.
`CHAIN_ERROR`
: Error building or verifying the certificate chain, if any. If this variable is set, then the certificate chain in `CERT_FILENAME` may be incomplete or invalid.
## Malformed certificate information
The following environment variables are set for `malformed_cert` events:
`LOG_URI`
: The URI of the log containing the malformed certificate.
`ENTRY_INDEX`
: The index of the log entry containing the malformed certificate.
`LEAF_HASH`
: The base64-encoded Merkle hash of the leaf containing the malformed certificate.
`PARSE_ERROR`
: A human-readable string describing why the certificate is malformed.
`ENTRY_FILENAME`
: Path to a file containing the JSON log entry. The file contains a JSON object with two fields, `leaf_input` and `extra_data`, as described in RFC 6962 Section 4.6.
`TEXT_FILENAME`
: Path to a text file containing a description of the malformed certificate. This file contains the same text that certspotter uses in emails.
## Error information
The following environment variables are set for `error` events:
`TEXT_FILENAME`
: Path to a text file containing a description of the error. This file contains the same text that certspotter uses in emails.
# JSON FILE FORMAT
certspotter saves a JSON file for every discovered certificate and puts the path to the file in `$JSON_FILENAME`.
When `-no_save` is used, the JSON file is created in a temporary location and automatically deleted after your script completes.
Otherwise, it is saved in the state directory for permanent storage. Your script can read the JSON file, such as with the jq(1) command,
to get additional information about the certificate which isn't appropriate for environment variables.
The JSON file contains an object with the following fields:
`tbs_sha256`
: A string containing the hex-encoded SHA-256 digest of the TBSCertificate, as defined in RFC 6962 Section 3.2.
Certificates and their corresponding precertificates have the same `tbs_sha256` value. You can compute the
TBS digest of a certificate using the `-printhash` option of the certspotter-authorize(8) command.
`pubkey_sha256`
: A string containing the hex-encoded SHA-256 digest of the certificate's Subject Public Key Info.
`dns_names`
: An array of strings containing the DNS names for which the
certificate is valid, taken from both the DNS subject alternative names
(SANs) and the subject common name (CN). Internationalized domain names
are encoded in Punycode.
`ip_addresses`
: An array of strings containing the IP addresses for which the certificate is valid,
taken from both the IP subject alternative names (SANs) and the subject common name (CN).
`not_before`
: A string containing the not before time of the certificate in RFC3339 format.
Null if there was an error parsing the certificate's validity.
`not_after`
: A string containing the not after (expiration) time of the certificate in RFC3339 format.
Null if there was an error parsing the certificate's validity.
Additional fields will be added in the future based on user feedback. Please open
an issue at <https://github.com/SSLMate/certspotter> if you have a use case for another field.
# DETERMINING IF A CERTIFICATE IS AUTHORIZED
You may want to compare a discovered certificate against a database of legitimate certificates
to decide whether to raise an alarm about it. Beware the following caveats:
* Since certificates may be logged to CT in the form of a precertificate, it doesn't work
to compare the discovered certificate's fingerprint against a list of legitimate fingerprints.
* Since malicious certificate authorities can issue certificates with duplicate serial numbers,
it is not secure to compare the the discovered certificate's serial number against a list of
legitimate serial numbers.
* Since malicious certificate authorities can put any value in a certificate's Subject Key Identifier (SKI)
extension, it is not secure to compare a discovered certificate's Subject Key Identifier against
a list of legitimate Subject Key Identifiers.
* Since malicious certificate authorities can put any value in the certificate's Issuer field,
it is not secure to compare a discovered certificate's Issuer against a list of authorized Issuers.
To determine if a certificate is legitimate, you should do one of the following:
* Compare the discovered certificate's TBS hash (as provided in the `$TBS_SHA256` variable) against
a list of authorized TBS hashes. The TBS hash is computed over the DER encoding of the certificate's
TBSCertificate, *with any SCT extension removed*. You can use the `-printhash` option
of the certspotter-authorize(8) command to compute a certificate's TBS hash.
* Compare the discovered certificate's public key hash (as provided in the `$PUBKEY_SHA256` variable) against
a list of authorized public key hashes. The public key hash is computed over the DER encoding of the
certificate's Subject Public Key Info. You can use the following openssl command to compute a certificate's
public key hash: `openssl x509 -pubkey -noout | openssl pkey -pubin -outform DER | openssl sha256`
# EXAMPLES
Example environment variables for a `discovered_cert` event:
```
CERT_FILENAME=/home/andrew/.certspotter/certs/3c/3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a.pem
CERT_SHA256=3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a
ENTRY_INDEX=6464843
EVENT=discovered_cert
ISSUER_DN=C=GB, ST=Greater Manchester, L=Salford, O=Sectigo Limited, CN=Sectigo RSA Domain Validation Secure Server CA
JSON_FILENAME=/usr2/andrew/.certspotter/certs/3c/3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a.v1.json
LOG_URI=https://ct.cloudflare.com/logs/nimbus2024/
NOT_AFTER='2024-01-26 03:47:26 +0000 UTC'
NOT_AFTER_RFC3339=2024-01-26T03:47:26Z
NOT_AFTER_UNIXTIME=1706240846
NOT_BEFORE='2023-01-31 03:47:26 +0000 UTC'
NOT_BEFORE_RFC3339=2023-01-31T03:47:26Z
NOT_BEFORE_UNIXTIME=1675136846
PUBKEY_SHA256=33ac1d9b9e56005ccac045eac2398b3e9dd6b3f5b66ae6260f2d478c7c0d82c8
SERIAL=c170fbf3bf27481e5c351a4db6f2dc5f
SUBJECT_DN=CN=sslmate.com
SUMMARY='certificate discovered for .sslmate.com'
TBS_SHA256=2388ee81c6f45cffc73e68a35fa8921e839e20acc9a98e8e6dcaea07cbfbdef8
TEXT_FILENAME=/usr2/andrew/.certspotter/certs/3c/3cdc83b3932c194fcdf17aa2bf1abc34e8438b293c3d5c70693e175b38ff128a.txt
WATCH_ITEM=.sslmate.com
```
Example JSON file for a discovered certificate:
```
{
"dns_names": [
"sslmate.com",
"www.sslmate.com"
],
"ip_addresses": [],
"not_after": "2024-01-26T03:47:26Z",
"not_before": "2023-01-31T03:47:26Z",
"pubkey_sha256": "33ac1d9b9e56005ccac045eac2398b3e9dd6b3f5b66ae6260f2d478c7c0d82c8",
"tbs_sha256": "2388ee81c6f45cffc73e68a35fa8921e839e20acc9a98e8e6dcaea07cbfbdef8"
}
```
# SEE ALSO
certspotter(8)
# COPYRIGHT
Copyright (c) 2016-2025 Opsmate, Inc.
# BUGS
Report bugs to <https://github.com/SSLMate/certspotter>.
third_party/certspotter/man/certspotter.md
Vendored
+261
View File
@@ -0,0 +1,261 @@
# NAME
**certspotter** - Certificate Transparency Log Monitor
# SYNOPSIS
**certspotter** [`-start_at_end`] [`-watchlist` *FILENAME*] [`-email` *ADDRESS*] `...`
# DESCRIPTION
**Cert Spotter** is a Certificate Transparency log monitor from SSLMate that
alerts you when a SSL/TLS certificate is issued for one of your domains.
Cert Spotter is easier to use than other open source CT monitors, since it does not require
a database. It's also more robust, since it uses a special certificate parser
that ensures it won't miss certificates.
Cert Spotter is also available as a hosted service by SSLMate,
<https://sslmate.com/certspotter>.
You can use Cert Spotter to detect:
* Certificates issued to attackers who have compromised your DNS and
are redirecting your visitors to their malicious site.
* Certificates issued to attackers who have taken over an abandoned
sub-domain in order to serve malware under your name.
* Certificates issued to attackers who have compromised a certificate
authority and want to impersonate your site.
* Certificates issued in violation of your corporate policy
or outside of your centralized certificate procurement process.
# OPTIONS
-email *ADDRESS*
: Email address to contact when a matching certificate is discovered, or
an error occurs. You can specify this option more than once to email
multiple addresses. Your system must have a working sendmail(1) command.
Regardless of the `-email` option, certspotter also emails any address listed
in `$CERTSPOTTER_CONFIG_DIR/email_recipients` file
(`~/.certspotter/email_recipients` by default). (One address per line,
blank lines are ignored.) This file is read only at startup, so you
must restart certspotter if you change it.
-healthcheck *INTERVAL*
: Perform a health check at the given interval (default: "24h") as described
below. *INTERVAL* must be a decimal number followed by "h" for hours or
"m" for minutes.
-logs *ADDRESS*
: Filename or HTTPS URL of a v2 or v3 JSON log list containing logs to monitor.
The schema for this file can be found at <https://www.gstatic.com/ct/log_list/v3/log_list_schema.json>.
Defaults to <https://loglist.certspotter.org/monitor.json>, which includes
the union of active logs recognized by Chrome and Apple. certspotter loads the
log list when starting up, and periodically reloads it in case it has changed.
-no\_save
: Do not save a copy of matching certificates. Note that enabling this option
will cause you to receive duplicate notifications, since certspotter will
have no way of knowing if you've been previously notified about a certificate.
-script *COMMAND*
: Command to execute when a matching certificate is found or an error occurs. See
certspotter-script(8) for information about the interface to scripts.
Regardless of the `-script` option, certspotter also executes any executable
file in the `$CERTSPOTTER_CONFIG_DIR/hooks.d` directory
(`~/.certspotter/hooks.d` by default).
-start\_at\_end
: Start monitoring logs from the end rather than the beginning.
**WARNING**: monitoring from the beginning guarantees detection of all
certificates, but requires downloading hundreds of millions of
certificates, which takes days.
-state\_dir *PATH*
: Directory for storing state. Defaults to `$CERTSPOTTER_STATE_DIR`, which is
"~/.certspotter" by default.
-stdout
: Write matching certificates and errors to stdout.
-verbose
: Print detailed information about certspotter's operation to stderr.
-version
: Print version and exit.
-watchlist *PATH*
: File containing DNS names to monitor, one per line. To monitor an entire
domain namespace (including the domain itself and all sub-domains) prefix
the domain name with a dot (e.g. ".example.com"). To monitor a single DNS
name only, do not prefix the name with a dot.
Defaults to `$CERTSPOTTER_CONFIG_DIR/watchlist`, which is
"~/.certspotter/watchlist" by default.
Specify `-` to read the watch list from stdin.
certspotter reads the watch list only when starting up, so you must restart
certspotter if you change it.
# NOTIFICATIONS
When certspotter detects a certificate matching your watchlist, or encounters
an error that is preventing it from discovering certificates, it notifies you
as follows:
* Emails any address specified by the `-email` command line flag.
* Emails any address listed in the `$CERTSPOTTER_CONFIG_DIR/email_recipients`
file (`~/.certspotter/email_recipients` by default). (One address per line,
blank lines are ignored.) This file is read only at startup, so you
must restart certspotter if you change it.
* Executes the script specified by the `-script` command line flag.
* Executes every executable file in the `$CERTSPOTTER_CONFIG_DIR/hooks.d`
directory (`~/.certspotter/hooks.d` by default).
* Writes the notification to standard out if the `-stdout` flag was specified.
Sending email requires a working sendmail(1) command. For details about
the script interface, see certspotter-script(8).
# OPERATION
certspotter continuously monitors all browser-recognized Certificate
Transparency logs (both RFC6962 and static-ct-api) looking for certificates (including precertificates)
which are valid for any domain on your watch list. When certspotter
detects a matching certificate, it emails you, executes a script, and/or
writes a report to standard out, as described above.
certspotter also saves a copy of matching certificates in
`$CERTSPOTTER_STATE_DIR/certs` ("~/.certspotter/certs" by default)
unless you specify the `-no_save` option.
When certspotter has not previously monitored a log, it can either start
monitoring the log from the beginning, or seek to the end of the log and
start monitoring from there. Monitoring from the beginning guarantees
detection of all certificates, but requires downloading hundreds of
millions of certificates, which takes days. The default behavior is to
monitor from the beginning. To start monitoring new logs from the end,
specify the `-start_at_end` option.
If certspotter has previously monitored a log, it resumes monitoring
the log from the previous position. This means that if you add
a domain to your watch list, certspotter will not detect any certificates
that were logged prior to the addition. To detect such certificates,
you must delete `$CERTSPOTTER_STATE_DIR/logs`, which will cause certspotter
to restart monitoring from the very beginning of each log (provided
`-start_at_end` is not specified). This will cause certspotter to download
hundreds of millions of certificates, which takes days. To find preexisting
certificates, it's faster to use the Cert Spotter service
<https://sslmate.com/certspotter>, SSLMate's Certificate Transparency Search
API <https://sslmate.com/ct_search_api>, or a CT search engine such as
<https://crt.sh>.
# ERROR HANDLING
When certspotter encounters a problem with the local system (e.g. failure
to write a file, send an email, or execute a script), it prints a message to stderr and
exits with a non-zero status.
When certspotter encounters a problem loading the log list during startup, it
prints a message to stderr and exits with a non-zero status. When certspotter encounters a problem
reloading the log list, it prints a message to stderr and continues running with the previously-loaded
log list. It will try reloading the log list again later.
When certspotter encounters a problem contacting a log, it writes the error to a file in
the state directory and continues running. It will try contacting the log again later;
most log errors are transient. The last 7 days of errors are kept.
Every 24 hours (unless overridden by `-healthcheck`), certspotter performs the
following health checks:
* Ensure that the log list has been successfully retrieved at least once
since the previous health check.
* Ensure that every log has been successfully contacted at least once
since the previous health check.
* Ensure that certspotter is not falling behind monitoring any logs.
If any health check fails, certspotter notifies you by email, script, and/or
standard out, as described above. The notification includes the last several errors
encountered when contacting the log.
Health check failures should be rare, and you should take them seriously because it means
certspotter might not detect all certificates. It might also be an indication
of CT log misbehavior. Check the error files for details, and if
you need help, file an issue at <https://github.com/SSLMate/certspotter>.
# EXIT STATUS
certspotter exits 0 when it receives `SIGTERM` or `SIGINT`,
and non-zero when a serious error occurs.
# ENVIRONMENT
`CERTSPOTTER_STATE_DIR`
: Directory for storing state. Overridden by `-state_dir`. Defaults to
`~/.certspotter`.
`CERTSPOTTER_CONFIG_DIR`
: Directory from which any configuration, such as the watch list, is read.
Defaults to `~/.certspotter`.
`EMAIL`
: Email address from which to send emails. If not set, certspotter lets sendmail pick
the address.
`HTTPS_PROXY`
: URL of proxy server for making HTTPS requests. `http://`, `https://`, and
`socks5://` URLs are supported. By default, no proxy server is used.
`SENDMAIL_PATH`
: Path to the sendmail binary used for sending emails. Defaults to `/usr/sbin/sendmail`.
If the `SYSTEMD_EXEC_PID` environment variable equals the PID of certspotter, then certspotter assumes it is running under systemd and respects the `CONFIGURATION_DIRECTORY`, `STATE_DIRECTORY`, and `CACHE_DIRECTORY` environment variables, unless overridden by a certspotter-specific environment variable or flag.
certspotter respects the `XDG_CACHE_HOME` environment variable for locating its cache directory. For historical reasons, certspotter does not respect `XDG_CONFIG_HOME` or `XDG_STATE_HOME`.
# DIRECTORIES
Config directory
: Stores configuration, such as the watch list. The location is: (1) the `CERTSPOTTER_CONFIG_DIR` environment variable, if set, (2) the `CONFIGURATION_DIRECTORY` environment variable, if set and certspotter is running under systemd, or (3) the default location `~/.certspotter`. certspotter does not write to this directory.
State directory
: Stores state, such as the position of each log and a store of discovered certificates. The location is: (1) the `-state_dir` command line flag, if provided, (2) the `CERTSPOTTER_STATE_DIR` environment variable, if set, (3) the `STATE_DIRECTORY` environment variable, if set and certspotter is running under systemd, or (4) the default location `~/.certspotter`. certspotter creates this directory if necessary.
Cache directory
: Stores cached data. The location is (1) the `CERTSPOTTER_CACHE_DIR` environment variable if set, (2) the `CACHE_DIRECTORY` environment, if set and certspotter is running under systemd, or (3) `$XDG_CACHE_HOME/certspotter` (which on Linux is `~/.cache/certspotter` by default). You can delete this directory without impacting functionality, but certspotter may need to perform additional computation or network requests.
# SEE ALSO
certspotter-script(8)
# COPYRIGHT
Copyright (c) 2016-2025 Opsmate, Inc.
# BUGS
Report bugs to <https://github.com/SSLMate/certspotter>.
third_party/certspotter/merkletree/collapsed_tree.go
Vendored
+182
View File
@@ -0,0 +1,182 @@
// Copyright (C) 2022 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package merkletree
import (
"encoding/json"
"fmt"
"math/bits"
"slices"
)
// CollapsedTree is an efficient representation of a Merkle (sub)tree that permits appending
// nodes and calculating the root hash.
type CollapsedTree struct {
offset uint64
nodes []Hash
size uint64
}
func calculateNumNodes(size uint64) int {
return bits.OnesCount64(size)
}
// TODO: phase out this function
func EmptyCollapsedTree() *CollapsedTree {
return &CollapsedTree{nodes: []Hash{}, size: 0}
}
// TODO: phase out this function
func NewCollapsedTree(nodes []Hash, size uint64) (*CollapsedTree, error) {
tree := new(CollapsedTree)
if err := tree.Init(nodes, size); err != nil {
return nil, err
}
return tree, nil
}
func (tree CollapsedTree) Equal(other CollapsedTree) bool {
return tree.offset == other.offset && tree.size == other.size && slices.Equal(tree.nodes, other.nodes)
}
func (tree CollapsedTree) Clone() CollapsedTree {
return CollapsedTree{
offset: tree.offset,
nodes: slices.Clone(tree.nodes),
size: tree.size,
}
}
// Add a new leaf hash to the end of the tree.
// Returns an error if and only if the new tree would be too large for the subtree offset.
// Always returns a nil error if tree.Offset() == 0.
func (tree *CollapsedTree) Add(hash Hash) error {
if tree.offset > 0 {
maxSize := uint64(1) << bits.TrailingZeros64(tree.offset)
if tree.size+1 > maxSize {
return fmt.Errorf("subtree at offset %d is already at maximum size %d", tree.offset, maxSize)
}
}
tree.nodes = append(tree.nodes, hash)
tree.size++
tree.collapse()
return nil
}
func (tree *CollapsedTree) Append(other CollapsedTree) error {
if tree.offset+tree.size != other.offset {
return fmt.Errorf("subtree at offset %d cannot be appended to subtree ending at offset %d", other.offset, tree.offset+tree.size)
}
if tree.offset > 0 {
newSize := tree.size + other.size
maxSize := uint64(1) << bits.TrailingZeros64(tree.offset)
if newSize > maxSize {
return fmt.Errorf("size of new subtree (%d) would exceed maximum size %d for a subtree at offset %d", newSize, maxSize, tree.offset)
}
}
if tree.size > 0 {
maxSize := uint64(1) << bits.TrailingZeros64(tree.size)
if other.size > maxSize {
return fmt.Errorf("tree of size %d is too large to append to a tree of size %d (maximum size is %d)", other.size, tree.size, maxSize)
}
}
tree.nodes = append(tree.nodes, other.nodes...)
tree.size += other.size
tree.collapse()
return nil
}
func (tree *CollapsedTree) collapse() {
numNodes := calculateNumNodes(tree.size)
for len(tree.nodes) > numNodes {
left, right := tree.nodes[len(tree.nodes)-2], tree.nodes[len(tree.nodes)-1]
tree.nodes = tree.nodes[:len(tree.nodes)-2]
tree.nodes = append(tree.nodes, HashChildren(left, right))
}
}
func (tree CollapsedTree) CalculateRoot() Hash {
if len(tree.nodes) == 0 {
return HashNothing()
}
i := len(tree.nodes) - 1
hash := tree.nodes[i]
for i > 0 {
i -= 1
hash = HashChildren(tree.nodes[i], hash)
}
return hash
}
// Return the subtree offset (0 if this represents an entire tree)
func (tree CollapsedTree) Offset() uint64 {
return tree.offset
}
// Return a non-nil slice containing the nodes. The slice
// must not be modified.
func (tree CollapsedTree) Nodes() []Hash {
if tree.nodes == nil {
return []Hash{}
} else {
return tree.nodes
}
}
// Return the number of leaf nodes in the tree.
func (tree CollapsedTree) Size() uint64 {
return tree.size
}
type collapsedTreeMessage struct {
Offset uint64 `json:"offset,omitzero"`
Nodes []Hash `json:"nodes"` // never nil
Size uint64 `json:"size"`
}
func (tree CollapsedTree) MarshalJSON() ([]byte, error) {
return json.Marshal(collapsedTreeMessage{
Offset: tree.offset,
Nodes: tree.Nodes(),
Size: tree.size,
})
}
func (tree *CollapsedTree) UnmarshalJSON(b []byte) error {
var rawTree collapsedTreeMessage
if err := json.Unmarshal(b, &rawTree); err != nil {
return fmt.Errorf("error unmarshalling Collapsed Merkle Tree: %w", err)
}
if err := tree.InitSubtree(rawTree.Offset, rawTree.Nodes, rawTree.Size); err != nil {
return fmt.Errorf("error unmarshalling Collapsed Merkle Tree: %w", err)
}
return nil
}
func (tree *CollapsedTree) Init(nodes []Hash, size uint64) error {
if len(nodes) != calculateNumNodes(size) {
return fmt.Errorf("nodes has wrong length (should be %d, not %d)", calculateNumNodes(size), len(nodes))
}
tree.size = size
tree.nodes = nodes
return nil
}
func (tree *CollapsedTree) InitSubtree(offset uint64, nodes []Hash, size uint64) error {
if offset > 0 {
maxSize := uint64(1) << bits.TrailingZeros64(offset)
if size > maxSize {
return fmt.Errorf("subtree size (%d) is too large for offset %d (maximum size is %d)", size, offset, maxSize)
}
}
tree.offset = offset
return tree.Init(nodes, size)
}
third_party/certspotter/merkletree/fragment.go
Vendored
+122
View File
@@ -0,0 +1,122 @@
// Copyright (C) 2024 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package merkletree
import (
"encoding/json"
"fmt"
"slices"
)
// FragmentedCollapsedTree represents a sequence of non-overlapping subtrees
type FragmentedCollapsedTree struct {
subtrees []CollapsedTree // sorted by offset
}
func (tree *FragmentedCollapsedTree) AddHash(position uint64, hash Hash) error {
return tree.Add(CollapsedTree{
offset: position,
nodes: []Hash{hash},
size: 1,
})
}
func (tree *FragmentedCollapsedTree) Add(subtree CollapsedTree) error {
if subtree.size == 0 {
return nil
}
i := len(tree.subtrees)
for i > 0 && tree.subtrees[i-1].offset > subtree.offset {
i--
}
if i > 0 && tree.subtrees[i-1].offset+tree.subtrees[i-1].size > subtree.offset {
return fmt.Errorf("overlaps with subtree ending at %d", tree.subtrees[i-1].offset+tree.subtrees[i-1].size)
}
if i < len(tree.subtrees) && subtree.offset+subtree.size > tree.subtrees[i].offset {
return fmt.Errorf("overlaps with subtree starting at %d", tree.subtrees[i].offset)
}
if i == 0 || tree.subtrees[i-1].Append(subtree) != nil {
tree.subtrees = slices.Insert(tree.subtrees, i, subtree)
i++
}
for i < len(tree.subtrees) && tree.subtrees[i-1].Append(tree.subtrees[i]) == nil {
tree.subtrees = slices.Delete(tree.subtrees, i, i+1)
}
return nil
}
func (tree *FragmentedCollapsedTree) Merge(other FragmentedCollapsedTree) error {
// TODO: try to make this linear time instead of quadratic; it should be possible since the subtrees are sorted by offset
for _, subtree := range other.subtrees {
if err := tree.Add(subtree); err != nil {
return err
}
}
return nil
}
func (tree FragmentedCollapsedTree) Gaps(yield func(uint64, uint64) bool) {
var prevEnd uint64
for i := range tree.subtrees {
if prevEnd != tree.subtrees[i].offset {
if !yield(prevEnd, tree.subtrees[i].offset) {
return
}
}
prevEnd = tree.subtrees[i].offset + tree.subtrees[i].size
}
yield(prevEnd, 0)
}
func (tree FragmentedCollapsedTree) NumSubtrees() int {
return len(tree.subtrees)
}
func (tree FragmentedCollapsedTree) Subtree(i int) CollapsedTree {
return tree.subtrees[i]
}
func (tree FragmentedCollapsedTree) Subtrees() []CollapsedTree {
if tree.subtrees == nil {
return []CollapsedTree{}
} else {
return tree.subtrees
}
}
// Return true iff the tree contains at least the first n nodes (without any gaps)
func (tree FragmentedCollapsedTree) ContainsFirstN(n uint64) bool {
return len(tree.subtrees) >= 1 && tree.subtrees[0].offset == 0 && tree.subtrees[0].size >= n
}
func (tree *FragmentedCollapsedTree) Init(subtrees []CollapsedTree) error {
for i := 1; i < len(subtrees); i++ {
if subtrees[i-1].offset+subtrees[i-1].size > subtrees[i].offset {
return fmt.Errorf("subtrees %d and %d overlap", i-1, i)
}
}
tree.subtrees = subtrees
return nil
}
func (tree FragmentedCollapsedTree) MarshalJSON() ([]byte, error) {
return json.Marshal(tree.Subtrees())
}
func (tree *FragmentedCollapsedTree) UnmarshalJSON(b []byte) error {
var subtrees []CollapsedTree
if err := json.Unmarshal(b, &subtrees); err != nil {
return fmt.Errorf("error unmarshaling Fragmented Collapsed Merkle Tree: %w", err)
}
if err := tree.Init(subtrees); err != nil {
return fmt.Errorf("error unmarshaling Fragmented Collapsed Merkle Tree: %w", err)
}
return nil
}
third_party/certspotter/merkletree/hash.go
Vendored
+77
View File
@@ -0,0 +1,77 @@
// Copyright (C) 2022 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package merkletree
import (
"bytes"
"crypto/sha256"
"encoding/base64"
"encoding/json"
"fmt"
)
const HashLen = 32
type Hash [HashLen]byte
func (h Hash) Compare(other Hash) int {
return bytes.Compare(h[:], other[:])
}
func (h Hash) Base64String() string {
return base64.StdEncoding.EncodeToString(h[:])
}
func (h Hash) MarshalJSON() ([]byte, error) {
return json.Marshal(h[:])
}
func (h Hash) MarshalBinary() ([]byte, error) {
return h[:], nil
}
func (h *Hash) UnmarshalJSON(b []byte) error {
var hashBytes []byte
if err := json.Unmarshal(b, &hashBytes); err != nil {
return err
}
return h.UnmarshalBinary(hashBytes)
}
func (h *Hash) UnmarshalBinary(hashBytes []byte) error {
if len(hashBytes) != HashLen {
return fmt.Errorf("Merkle Tree hash has wrong length (should be %d bytes long, not %d)", HashLen, len(hashBytes))
}
copy(h[:], hashBytes)
return nil
}
func HashNothing() Hash {
return sha256.Sum256(nil)
}
func HashLeaf(leafBytes []byte) Hash {
var hash Hash
hasher := sha256.New()
hasher.Write([]byte{0x00})
hasher.Write(leafBytes)
hasher.Sum(hash[:0])
return hash
}
func HashChildren(left Hash, right Hash) Hash {
var hash Hash
hasher := sha256.New()
hasher.Write([]byte{0x01})
hasher.Write(left[:])
hasher.Write(right[:])
hasher.Sum(hash[:0])
return hash
}
third_party/certspotter/monitor/config.go
Vendored
+39
View File
@@ -0,0 +1,39 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"time"
"software.sslmate.com/src/certspotter"
)
// IdentifierSink receives the DNS identifiers parsed from every monitored
// certificate. This is a CTCLOUD PATCH to upstream certspotter; see
// CTCLOUD_PATCH.md. When non-nil it is invoked on the monitoring hot path
// instead of the watchlist/notification machinery.
type IdentifierSink interface {
HandleIdentifiers(ctx context.Context, identifiers *certspotter.Identifiers)
}
type Config struct {
LogListSource string
State StateProvider
StartAtEnd bool
WatchList WatchList
Verbose bool
HealthCheckInterval time.Duration
// IdentifierSink is a CTCLOUD PATCH (see CTCLOUD_PATCH.md). When set,
// parsed identifiers are handed directly to it in-process and the normal
// watchlist/chain/notification path is skipped.
IdentifierSink IdentifierSink
}
third_party/certspotter/monitor/daemon.go
Vendored
+173
View File
@@ -0,0 +1,173 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"errors"
"fmt"
"golang.org/x/sync/errgroup"
"log"
insecurerand "math/rand"
"software.sslmate.com/src/certspotter/loglist"
"time"
)
const (
reloadLogListIntervalMin = 30 * time.Minute
reloadLogListIntervalMax = 90 * time.Minute
)
func randomDuration(min, max time.Duration) time.Duration {
return min + time.Duration(insecurerand.Int63n(int64(max-min+1)))
}
func reloadLogListInterval() time.Duration {
return randomDuration(reloadLogListIntervalMin, reloadLogListIntervalMax)
}
type task struct {
log *loglist.Log
stop context.CancelFunc
}
type daemon struct {
config *Config
taskgroup *errgroup.Group
tasks map[LogID]task
logsLoadedAt time.Time
logListToken *loglist.ModificationToken
}
func (daemon *daemon) healthCheck(ctx context.Context) error {
if time.Since(daemon.logsLoadedAt) >= daemon.config.HealthCheckInterval {
errors, err := daemon.config.State.GetErrors(ctx, nil, recentErrorCount)
if err != nil {
return fmt.Errorf("error getting recent errors: %w", err)
}
var errorsDir string
if fsstate, ok := daemon.config.State.(*FilesystemState); ok {
errorsDir = fsstate.errorDir(nil)
}
info := &StaleLogListInfo{
Source: daemon.config.LogListSource,
LastSuccess: daemon.logsLoadedAt,
RecentErrors: errors,
ErrorsDir: errorsDir,
}
if err := daemon.config.State.NotifyHealthCheckFailure(ctx, nil, info); err != nil {
return fmt.Errorf("error notifying about stale log list: %w", err)
}
}
for _, task := range daemon.tasks {
if err := healthCheckLog(ctx, daemon.config, task.log); err != nil {
return fmt.Errorf("error checking health of log %q: %w", task.log.GetMonitoringURL(), err)
}
}
return nil
}
func (daemon *daemon) startTask(ctx context.Context, ctlog *loglist.Log) task {
ctx, cancel := context.WithCancel(ctx)
daemon.taskgroup.Go(func() error {
defer cancel()
err := monitorLogContinously(ctx, daemon.config, ctlog)
if daemon.config.Verbose {
log.Printf("%s: task stopped with error: %s", ctlog.GetMonitoringURL(), err)
}
if ctx.Err() == context.Canceled && errors.Is(err, context.Canceled) {
return nil
} else {
return fmt.Errorf("error while monitoring %s: %w", ctlog.GetMonitoringURL(), err)
}
})
return task{log: ctlog, stop: cancel}
}
func (daemon *daemon) loadLogList(ctx context.Context) error {
newLogList, newToken, err := getLogList(ctx, daemon.config.LogListSource, daemon.logListToken)
if errors.Is(err, loglist.ErrNotModified) {
daemon.logsLoadedAt = time.Now() // so the log list is not reported as stale
return nil
} else if err != nil {
return err
}
if daemon.config.Verbose {
log.Printf("fetched %d logs from %q", len(newLogList), daemon.config.LogListSource)
}
for logID, task := range daemon.tasks {
if _, exists := newLogList[logID]; exists {
continue
}
if daemon.config.Verbose {
log.Printf("stopping task for log %s", logID.Base64String())
}
task.stop()
delete(daemon.tasks, logID)
}
for logID, ctlog := range newLogList {
if _, isRunning := daemon.tasks[logID]; isRunning {
continue
}
if daemon.config.Verbose {
log.Printf("starting task for log %s (%s)", logID.Base64String(), ctlog.GetMonitoringURL())
}
daemon.tasks[logID] = daemon.startTask(ctx, ctlog)
}
daemon.logsLoadedAt = time.Now()
daemon.logListToken = newToken
return nil
}
func (daemon *daemon) run(ctx context.Context) error {
if err := daemon.config.State.Prepare(ctx); err != nil {
return fmt.Errorf("error preparing state: %w", err)
}
if err := daemon.loadLogList(ctx); err != nil {
return fmt.Errorf("error loading log list: %w", err)
}
reloadLogListTicker := time.NewTicker(reloadLogListInterval())
defer reloadLogListTicker.Stop()
healthCheckTicker := time.NewTicker(daemon.config.HealthCheckInterval)
defer healthCheckTicker.Stop()
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-reloadLogListTicker.C:
if err := daemon.loadLogList(ctx); err != nil {
recordError(ctx, daemon.config, nil, fmt.Errorf("error reloading log list (will try again later): %w", err))
}
reloadLogListTicker.Reset(reloadLogListInterval())
case <-healthCheckTicker.C:
if err := daemon.healthCheck(ctx); err != nil {
return err
}
}
}
}
func Run(ctx context.Context, config *Config) error {
group, ctx := errgroup.WithContext(ctx)
daemon := &daemon{
config: config,
taskgroup: group,
tasks: make(map[LogID]task),
}
group.Go(func() error { return daemon.run(ctx) })
return group.Wait()
}
third_party/certspotter/monitor/discoveredcert.go
Vendored
+189
View File
@@ -0,0 +1,189 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"bytes"
"encoding/hex"
"encoding/pem"
"fmt"
"strings"
"time"
"software.sslmate.com/src/certspotter"
"software.sslmate.com/src/certspotter/cttypes"
)
type DiscoveredCert struct {
WatchItem WatchItem
LogEntry *LogEntry
Info *certspotter.CertInfo
Chain []cttypes.ASN1Cert // first entry is the leaf certificate or precertificate
ChainError error // any error generating or validating Chain; if non-nil, Chain may be partial or incorrect
TBSSHA256 [32]byte // computed over Info.TBS.Raw
SHA256 [32]byte // computed over Chain[0]
PubkeySHA256 [32]byte // computed over Info.TBS.PublicKey.FullBytes
Identifiers *certspotter.Identifiers
}
type certPaths struct {
temporary bool
certPath string
jsonPath string
textPath string
}
func (cert *DiscoveredCert) pemChain() []byte {
var buffer bytes.Buffer
if cert.ChainError != nil {
fmt.Fprintln(&buffer, "Warning: this chain may be incomplete or invalid: ", cert.ChainError)
}
for _, certBytes := range cert.Chain {
if err := pem.Encode(&buffer, &pem.Block{
Type: "CERTIFICATE",
Bytes: certBytes,
}); err != nil {
panic(fmt.Errorf("encoding certificate as PEM failed unexpectedly: %w", err))
}
}
return buffer.Bytes()
}
func (cert *DiscoveredCert) json() any {
object := map[string]any{
"tbs_sha256": hex.EncodeToString(cert.TBSSHA256[:]),
"pubkey_sha256": hex.EncodeToString(cert.PubkeySHA256[:]),
"dns_names": cert.Identifiers.DNSNames,
"ip_addresses": cert.Identifiers.IPAddrs,
}
if cert.Info.ValidityParseError == nil {
object["not_before"] = cert.Info.Validity.NotBefore
object["not_after"] = cert.Info.Validity.NotAfter
} else {
object["not_before"] = nil
object["not_after"] = nil
}
return object
}
func writeCertFiles(cert *DiscoveredCert, paths *certPaths) error {
if err := writeFile(paths.certPath, cert.pemChain(), 0666); err != nil {
return err
}
if err := writeJSONFile(paths.jsonPath, cert.json(), 0666); err != nil {
return err
}
if err := writeTextFile(paths.textPath, certNotificationText(cert, paths), 0666); err != nil {
return err
}
return nil
}
func certNotificationEnviron(cert *DiscoveredCert, paths *certPaths) []string {
env := []string{
"EVENT=discovered_cert",
"SUMMARY=" + certNotificationSummary(cert),
"CERT_PARSEABLE=yes", // backwards compat with pre-0.15.0; not documented
"LOG_URI=" + cert.LogEntry.Log.GetMonitoringURL(),
"ENTRY_INDEX=" + fmt.Sprint(cert.LogEntry.Index),
"WATCH_ITEM=" + cert.WatchItem.String(),
"TBS_SHA256=" + hex.EncodeToString(cert.TBSSHA256[:]),
"CERT_SHA256=" + hex.EncodeToString(cert.SHA256[:]),
"FINGERPRINT=" + hex.EncodeToString(cert.SHA256[:]), // backwards compat with pre-0.15.0; not documented
"PUBKEY_SHA256=" + hex.EncodeToString(cert.PubkeySHA256[:]),
"PUBKEY_HASH=" + hex.EncodeToString(cert.PubkeySHA256[:]), // backwards compat with pre-0.15.0; not documented
}
if paths != nil {
env = append(env, "CERT_FILENAME="+paths.certPath)
env = append(env, "JSON_FILENAME="+paths.jsonPath)
env = append(env, "TEXT_FILENAME="+paths.textPath)
}
if cert.Info.ValidityParseError == nil {
env = append(env, "NOT_BEFORE="+cert.Info.Validity.NotBefore.String())
env = append(env, "NOT_BEFORE_UNIXTIME="+fmt.Sprint(cert.Info.Validity.NotBefore.Unix()))
env = append(env, "NOT_BEFORE_RFC3339="+cert.Info.Validity.NotBefore.Format(time.RFC3339))
env = append(env, "NOT_AFTER="+cert.Info.Validity.NotAfter.String())
env = append(env, "NOT_AFTER_UNIXTIME="+fmt.Sprint(cert.Info.Validity.NotAfter.Unix()))
env = append(env, "NOT_AFTER_RFC3339="+cert.Info.Validity.NotAfter.Format(time.RFC3339))
} else {
env = append(env, "VALIDITY_PARSE_ERROR="+cert.Info.ValidityParseError.Error())
}
if cert.Info.SubjectParseError == nil {
env = append(env, "SUBJECT_DN="+cert.Info.Subject.String())
} else {
env = append(env, "SUBJECT_PARSE_ERROR="+cert.Info.SubjectParseError.Error())
}
if cert.Info.IssuerParseError == nil {
env = append(env, "ISSUER_DN="+cert.Info.Issuer.String())
} else {
env = append(env, "ISSUER_PARSE_ERROR="+cert.Info.IssuerParseError.Error())
}
if cert.Info.SerialNumberParseError == nil {
env = append(env, "SERIAL="+fmt.Sprintf("%x", cert.Info.SerialNumber))
} else {
env = append(env, "SERIAL_PARSE_ERROR="+cert.Info.SerialNumberParseError.Error())
}
if cert.ChainError != nil {
env = append(env, "CHAIN_ERROR="+cert.ChainError.Error())
}
return env
}
func certNotificationText(cert *DiscoveredCert, paths *certPaths) string {
// TODO-4: improve the output: include WatchItem, indicate hash algorithm used for fingerprints, ... (look at SSLMate email for inspiration)
text := new(strings.Builder)
writeField := func(name string, value any) { fmt.Fprintf(text, "\t%13s = %s\n", name, value) }
fmt.Fprintf(text, "%x:\n", cert.SHA256)
for _, dnsName := range cert.Identifiers.DNSNames {
writeField("DNS Name", dnsName)
}
for _, ipaddr := range cert.Identifiers.IPAddrs {
writeField("IP Address", ipaddr)
}
writeField("Pubkey", hex.EncodeToString(cert.PubkeySHA256[:]))
if cert.Info.IssuerParseError == nil {
writeField("Issuer", cert.Info.Issuer)
} else {
writeField("Issuer", fmt.Sprintf("[unable to parse: %s]", cert.Info.IssuerParseError))
}
if cert.Info.ValidityParseError == nil {
writeField("Not Before", cert.Info.Validity.NotBefore)
writeField("Not After", cert.Info.Validity.NotAfter)
} else {
writeField("Not Before", fmt.Sprintf("[unable to parse: %s]", cert.Info.ValidityParseError))
writeField("Not After", fmt.Sprintf("[unable to parse: %s]", cert.Info.ValidityParseError))
}
writeField("Log Entry", fmt.Sprintf("%d @ %s", cert.LogEntry.Index, cert.LogEntry.Log.GetMonitoringURL()))
writeField("crt.sh", "https://crt.sh/?sha256="+hex.EncodeToString(cert.SHA256[:]))
if cert.ChainError != nil {
writeField("Error Building Chain", cert.ChainError.Error())
}
if paths != nil && !paths.temporary {
writeField("Filename", paths.certPath)
}
return text.String()
}
func certNotificationSummary(cert *DiscoveredCert) string {
return fmt.Sprintf("Certificate Discovered for %s", cert.WatchItem)
}
third_party/certspotter/monitor/errors.go
Vendored
+28
View File
@@ -0,0 +1,28 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"log"
"software.sslmate.com/src/certspotter/loglist"
)
func recordError(ctx context.Context, config *Config, ctlog *loglist.Log, errToRecord error) {
if err := config.State.NotifyError(ctx, ctlog, errToRecord); err != nil {
log.Print("unable to notify about error: ", err)
if ctlog == nil {
log.Print(errToRecord)
} else {
log.Print(ctlog.GetMonitoringURL(), ": ", errToRecord)
}
}
}
third_party/certspotter/monitor/fileutils.go
Vendored
+138
View File
@@ -0,0 +1,138 @@
// Copyright (C) 2017, 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"slices"
"strings"
)
func randomFileSuffix() string {
var randomBytes [12]byte
if _, err := rand.Read(randomBytes[:]); err != nil {
panic(err)
}
return hex.EncodeToString(randomBytes[:])
}
func writeSyncFile(filename string, data []byte, perm os.FileMode) error {
f, err := os.OpenFile(filename, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)
if err != nil {
return err
}
_, err = f.Write(data)
if err2 := f.Sync(); err2 != nil && err == nil {
err = err2
}
if err2 := f.Close(); err2 != nil && err == nil {
err = err2
}
return err
}
func writeFile(filename string, data []byte, perm os.FileMode) error {
tempname := filename + ".tmp." + randomFileSuffix()
if err := writeSyncFile(tempname, data, perm); err != nil {
return fmt.Errorf("error writing %s: %w", filename, err)
}
if err := os.Rename(tempname, filename); err != nil {
os.Remove(tempname)
return fmt.Errorf("error writing %s: %w", filename, err)
}
return nil
}
func writeTextFile(filename string, fileText string, perm os.FileMode) error {
return writeFile(filename, []byte(fileText), perm)
}
func writeJSONFile(filename string, data any, perm os.FileMode) error {
fileBytes, err := json.Marshal(data)
if err != nil {
return err
}
fileBytes = append(fileBytes, '\n')
return writeFile(filename, fileBytes, perm)
}
func fileExists(filename string) bool {
_, err := os.Lstat(filename)
return err == nil
}
// countFiles returns the number of non-hidden files in dirPath whose
// name ends in suffix. Returns 0 if dirPath does not exist.
func countFiles(dirPath string, suffix string) (int, error) {
entries, err := os.ReadDir(dirPath)
if errors.Is(err, fs.ErrNotExist) {
return 0, nil
} else if err != nil {
return 0, err
}
count := 0
for _, entry := range entries {
if !entry.IsDir() && !strings.HasPrefix(entry.Name(), ".") && strings.HasSuffix(entry.Name(), suffix) {
count++
}
}
return count, nil
}
func tailFile(filename string, linesWanted int) ([]byte, int, error) {
file, err := os.Open(filename)
if err != nil {
return nil, 0, err
}
defer file.Close()
return tail(file, linesWanted, 4096)
}
func tail(r io.ReadSeeker, linesWanted int, chunkSize int) ([]byte, int, error) {
var buf []byte
linesGot := 0
offset, err := r.Seek(0, io.SeekEnd)
if err != nil {
return nil, 0, err
}
for offset > 0 {
readSize := chunkSize
if offset < int64(readSize) {
readSize = int(offset)
}
offset -= int64(readSize)
if _, err := r.Seek(offset, io.SeekStart); err != nil {
return nil, 0, err
}
buf = slices.Grow(buf, readSize)
copy(buf[readSize:len(buf)+readSize], buf)
buf = buf[:len(buf)+readSize]
if _, err := io.ReadFull(r, buf[:readSize]); err != nil {
return nil, 0, err
}
for i := readSize; i > 0; i-- {
if buf[i-1] == '\n' {
if linesGot == linesWanted {
return buf[i:], linesGot, nil
}
linesGot++
}
}
}
return buf, linesGot, nil
}
third_party/certspotter/monitor/fsstate.go
Vendored
+416
View File
@@ -0,0 +1,416 @@
// Copyright (C) 2024 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"crypto/rand"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io/fs"
"log"
"os"
"path/filepath"
"strings"
"sync"
"time"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/loglist"
"software.sslmate.com/src/certspotter/merkletree"
)
const keepErrorDays = 7
const errorDateFormat = "2006-01-02"
type FilesystemState struct {
StateDir string
CacheDir string
SaveCerts bool
Script string
ScriptDir string
Email []string
Stdout bool
errorMu sync.Mutex
}
func (s *FilesystemState) logStateDir(logID LogID) string {
return filepath.Join(s.StateDir, "logs", logID.Base64URLString())
}
func (s *FilesystemState) Prepare(ctx context.Context) error {
if err := prepareStateDir(s.StateDir); err != nil {
return err
}
if err := prepareCacheDir(s.CacheDir); err != nil {
return err
}
return nil
}
func (s *FilesystemState) PrepareLog(ctx context.Context, logID LogID) error {
var (
stateDirPath = s.logStateDir(logID)
sthsDirPath = filepath.Join(stateDirPath, "unverified_sths")
malformedDirPath = filepath.Join(stateDirPath, "malformed_entries")
healthchecksDirPath = filepath.Join(stateDirPath, "healthchecks")
errorsDirPath = filepath.Join(stateDirPath, "errors")
)
for _, dirPath := range []string{stateDirPath, sthsDirPath, malformedDirPath, healthchecksDirPath, errorsDirPath} {
if err := os.Mkdir(dirPath, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return err
}
}
return nil
}
// ListLogs returns the IDs of the logs that have a state directory (i.e., for
// which PrepareLog has been called at some point). Note that LoadLogState can
// still return nil for a returned log ID, if no state has been stored yet.
func (s *FilesystemState) ListLogs(ctx context.Context) ([]LogID, error) {
entries, err := os.ReadDir(filepath.Join(s.StateDir, "logs"))
if errors.Is(err, fs.ErrNotExist) {
return []LogID{}, nil
} else if err != nil {
return nil, err
}
logIDs := make([]LogID, 0, len(entries))
for _, entry := range entries {
name := entry.Name()
if strings.HasPrefix(name, ".") || !entry.IsDir() {
continue
}
nameBytes, err := base64.RawURLEncoding.DecodeString(name)
if err != nil || len(nameBytes) != 32 {
continue
}
var logID LogID
copy(logID[:], nameBytes)
logIDs = append(logIDs, logID)
}
return logIDs, nil
}
func (s *FilesystemState) LoadLogState(ctx context.Context, logID LogID) (*LogState, error) {
filePath := filepath.Join(s.logStateDir(logID), "state.json")
fileBytes, err := os.ReadFile(filePath)
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
} else if err != nil {
return nil, err
}
state := new(LogState)
if err := json.Unmarshal(fileBytes, state); err != nil {
return nil, fmt.Errorf("error parsing %s: %w", filePath, err)
}
return state, nil
}
func (s *FilesystemState) StoreLogState(ctx context.Context, logID LogID, state *LogState) error {
filePath := filepath.Join(s.logStateDir(logID), "state.json")
return writeJSONFile(filePath, state, 0666)
}
func (s *FilesystemState) StoreSTH(ctx context.Context, logID LogID, sth *cttypes.SignedTreeHead) (*StoredSTH, error) {
sthsDirPath := filepath.Join(s.logStateDir(logID), "unverified_sths")
return storeSTHInDir(sthsDirPath, sth)
}
func (s *FilesystemState) LoadSTHs(ctx context.Context, logID LogID) ([]*StoredSTH, error) {
sthsDirPath := filepath.Join(s.logStateDir(logID), "unverified_sths")
return loadSTHsFromDir(sthsDirPath)
}
func (s *FilesystemState) RemoveSTH(ctx context.Context, logID LogID, sth *cttypes.SignedTreeHead) error {
sthsDirPath := filepath.Join(s.logStateDir(logID), "unverified_sths")
return removeSTHFromDir(sthsDirPath, sth)
}
// LoadLargestSTH returns the stored STH with the largest tree size, breaking
// ties by later timestamp. Returns nil if no STHs are stored for the log.
func (s *FilesystemState) LoadLargestSTH(ctx context.Context, logID LogID) (*StoredSTH, error) {
sthsDirPath := filepath.Join(s.logStateDir(logID), "unverified_sths")
return loadLargestSTHFromDir(sthsDirPath)
}
// CountSTHs returns the number of STHs stored for the log.
func (s *FilesystemState) CountSTHs(ctx context.Context, logID LogID) (int, error) {
return countFiles(filepath.Join(s.logStateDir(logID), "unverified_sths"), ".json")
}
// CountMalformedEntries returns the number of malformed entries recorded for the log.
func (s *FilesystemState) CountMalformedEntries(ctx context.Context, logID LogID) (int, error) {
return countFiles(filepath.Join(s.logStateDir(logID), "malformed_entries"), ".json")
}
// CountHealthCheckFailures returns the number of health check failures recorded for the log.
func (s *FilesystemState) CountHealthCheckFailures(ctx context.Context, logID LogID) (int, error) {
return countFiles(filepath.Join(s.logStateDir(logID), "healthchecks"), ".txt")
}
func (s *FilesystemState) StoreIssuer(ctx context.Context, fingerprint *[32]byte, issuer []byte) error {
filePath := filepath.Join(s.CacheDir, "issuers", hex.EncodeToString(fingerprint[:]))
return writeFile(filePath, issuer, 0666)
}
func (s *FilesystemState) LoadIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error) {
filePath := filepath.Join(s.CacheDir, "issuers", hex.EncodeToString(fingerprint[:]))
issuer, err := os.ReadFile(filePath)
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
} else if err != nil {
return nil, err
} else {
return issuer, err
}
}
func (s *FilesystemState) NotifyCert(ctx context.Context, cert *DiscoveredCert) error {
var (
tbsFingerprint = hex.EncodeToString(cert.TBSSHA256[:])
certFingerprint = hex.EncodeToString(cert.SHA256[:])
)
var (
tbsDir = filepath.Join(s.StateDir, "certs", tbsFingerprint[0:2])
certDir = filepath.Join(s.StateDir, "certs", certFingerprint[0:2])
)
notifiedPath := filepath.Join(tbsDir, "."+tbsFingerprint+".notified")
if fileExists(notifiedPath) {
return nil
}
// previous versions of certspotter used these files to record that a notification was sent
for _, filename := range []string{
"." + certFingerprint + ".notified",
certFingerprint + ".cert.pem",
certFingerprint + ".precert.pem",
} {
if fileExists(filepath.Join(certDir, filename)) {
return nil
}
}
var paths *certPaths
if s.SaveCerts {
if err := os.Mkdir(tbsDir, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return fmt.Errorf("error creating directory in which to save certificate %x: %w", cert.SHA256, err)
}
if err := os.Mkdir(certDir, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return fmt.Errorf("error creating directory in which to save certificate %x: %w", cert.SHA256, err)
}
paths = &certPaths{
certPath: filepath.Join(certDir, certFingerprint+".pem"),
jsonPath: filepath.Join(certDir, certFingerprint+".v1.json"),
textPath: filepath.Join(certDir, certFingerprint+".txt"),
}
if err := writeCertFiles(cert, paths); err != nil {
return fmt.Errorf("error saving certificate %x: %w", cert.SHA256, err)
}
} else {
paths = &certPaths{
temporary: true,
certPath: filepath.Join(os.TempDir(), "certspotter-"+rand.Text()+".pem"),
jsonPath: filepath.Join(os.TempDir(), "certspotter-"+rand.Text()+".v1.json"),
textPath: filepath.Join(os.TempDir(), "certspotter-"+rand.Text()+".txt"),
}
defer func() {
os.Remove(paths.certPath)
os.Remove(paths.jsonPath)
os.Remove(paths.textPath)
}()
if err := writeCertFiles(cert, paths); err != nil {
return fmt.Errorf("error saving certificate %x to temporary files: %w", cert.SHA256, err)
}
}
if err := s.notify(ctx, &notification{
summary: certNotificationSummary(cert),
environ: certNotificationEnviron(cert, paths),
text: certNotificationText(cert, paths),
}); err != nil {
return fmt.Errorf("error notifying about discovered certificate for %s (%x): %w", cert.WatchItem, cert.SHA256, err)
}
if s.SaveCerts {
if err := os.WriteFile(notifiedPath, nil, 0666); err != nil {
return fmt.Errorf("error saving certificate %x: %w", cert.SHA256, err)
}
}
return nil
}
func (s *FilesystemState) NotifyMalformedEntry(ctx context.Context, entry *LogEntry, parseError error) error {
var (
dirPath = filepath.Join(s.logStateDir(entry.Log.LogID), "malformed_entries")
entryPath = filepath.Join(dirPath, fmt.Sprintf("%d.json", entry.Index))
textPath = filepath.Join(dirPath, fmt.Sprintf("%d.txt", entry.Index))
)
summary := fmt.Sprintf("Unable to Parse Entry %d in %s", entry.Index, entry.Log.GetMonitoringURL())
leafHash := merkletree.HashLeaf(entry.LeafInput())
text := new(strings.Builder)
writeField := func(name string, value any) { fmt.Fprintf(text, "\t%13s = %s\n", name, value) }
fmt.Fprintf(text, "Unable to determine if log entry matches your watchlist. Please file a bug report at https://github.com/SSLMate/certspotter/issues/new with the following details:\n")
writeField("Log Entry", fmt.Sprintf("%d @ %s", entry.Index, entry.Log.GetMonitoringURL()))
writeField("Leaf Hash", leafHash.Base64String())
writeField("Error", parseError.Error())
if err := writeJSONFile(entryPath, entry.Entry, 0666); err != nil {
return fmt.Errorf("error saving JSON file: %w", err)
}
if err := writeTextFile(textPath, text.String(), 0666); err != nil {
return fmt.Errorf("error saving texT file: %w", err)
}
environ := []string{
"EVENT=malformed_cert",
"SUMMARY=" + summary,
"LOG_URI=" + entry.Log.GetMonitoringURL(),
"ENTRY_INDEX=" + fmt.Sprint(entry.Index),
"LEAF_HASH=" + leafHash.Base64String(),
"PARSE_ERROR=" + parseError.Error(),
"ENTRY_FILENAME=" + entryPath,
"TEXT_FILENAME=" + textPath,
"CERT_PARSEABLE=no", // backwards compat with pre-0.15.0; not documented
}
if err := s.notify(ctx, &notification{
environ: environ,
summary: summary,
text: text.String(),
}); err != nil {
return err
}
return nil
}
func (s *FilesystemState) healthCheckDir(ctlog *loglist.Log) string {
if ctlog == nil {
return filepath.Join(s.StateDir, "healthchecks")
} else {
return filepath.Join(s.logStateDir(ctlog.LogID), "healthchecks")
}
}
func (s *FilesystemState) errorDir(ctlog *loglist.Log) string {
if ctlog == nil {
return filepath.Join(s.StateDir, "errors")
}
return filepath.Join(s.logStateDir(ctlog.LogID), "errors")
}
func (s *FilesystemState) NotifyHealthCheckFailure(ctx context.Context, ctlog *loglist.Log, info HealthCheckFailure) error {
textPath := filepath.Join(s.healthCheckDir(ctlog), healthCheckFilename())
environ := []string{
"EVENT=error",
"SUMMARY=" + info.Summary(),
"TEXT_FILENAME=" + textPath,
}
text := info.Text()
if err := writeTextFile(textPath, text, 0666); err != nil {
return fmt.Errorf("error saving text file: %w", err)
}
if err := s.notify(ctx, &notification{
environ: environ,
summary: info.Summary(),
text: text,
}); err != nil {
return err
}
return nil
}
func (s *FilesystemState) NotifyError(ctx context.Context, ctlog *loglist.Log, notifyErr error) error {
if ctlog == nil {
log.Print(notifyErr)
}
var (
now = time.Now()
filePath = filepath.Join(s.errorDir(ctlog), now.Format(errorDateFormat))
line = now.Format(time.RFC3339) + " " + notifyErr.Error() + "\n"
)
s.errorMu.Lock()
defer s.errorMu.Unlock()
file, err := os.OpenFile(filePath, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
return err
}
defer file.Close()
if _, err := file.WriteString(line); err != nil {
return err
}
return file.Close()
}
func (s *FilesystemState) GetErrors(ctx context.Context, ctlog *loglist.Log, count int) (string, error) {
dir := s.errorDir(ctlog)
now := time.Now()
var buf []byte
for daysBack := 0; count > 0 && daysBack < keepErrorDays; daysBack++ {
datePath := filepath.Join(dir, now.AddDate(0, 0, -daysBack).Format(errorDateFormat))
dateBuf, dateLines, err := tailFile(datePath, count)
if errors.Is(err, fs.ErrNotExist) {
continue
} else if err != nil {
return "", err
}
buf = append(dateBuf, buf...)
count -= dateLines
}
return string(buf), nil
}
func (s *FilesystemState) PruneOldErrors() {
cutoff := time.Now().AddDate(0, 0, -keepErrorDays)
pruneDir := func(dir string) {
entries, err := os.ReadDir(dir)
if errors.Is(err, fs.ErrNotExist) {
return
} else if err != nil {
log.Printf("unable to read error directory: %s", err)
return
}
for _, entry := range entries {
if entry.IsDir() {
continue
}
date, err := time.Parse(errorDateFormat, entry.Name())
if err != nil {
continue
}
if date.Before(cutoff) {
if err := os.Remove(filepath.Join(dir, entry.Name())); err != nil && !errors.Is(err, fs.ErrNotExist) {
log.Printf("unable to remove old error file: %s", err)
}
}
}
}
pruneDir(filepath.Join(s.StateDir, "errors"))
logsDir := filepath.Join(s.StateDir, "logs")
logDirs, err := os.ReadDir(logsDir)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
log.Printf("unable to read logs directory: %s", err)
return
}
for _, d := range logDirs {
if !d.IsDir() {
continue
}
pruneDir(filepath.Join(logsDir, d.Name(), "errors"))
}
}
third_party/certspotter/monitor/healthcheck.go
Vendored
+188
View File
@@ -0,0 +1,188 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"fmt"
"strings"
"time"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/loglist"
)
const recentErrorCount = 10
func healthCheckFilename() string {
return time.Now().UTC().Format(time.RFC3339) + ".txt"
}
func healthCheckLog(ctx context.Context, config *Config, ctlog *loglist.Log) error {
var (
position uint64
lastSuccess time.Time
verifiedSTH *cttypes.SignedTreeHead
)
if state, err := config.State.LoadLogState(ctx, ctlog.LogID); err != nil {
return fmt.Errorf("error loading log state: %w", err)
} else if state != nil {
if time.Since(state.LastSuccess) < config.HealthCheckInterval {
// log is healthy
return nil
}
position = state.DownloadPosition.Size()
lastSuccess = state.LastSuccess
verifiedSTH = state.VerifiedSTH
}
sths, err := config.State.LoadSTHs(ctx, ctlog.LogID)
if err != nil {
return fmt.Errorf("error loading STHs: %w", err)
}
var errorsDir string
if fsstate, ok := config.State.(*FilesystemState); ok {
errorsDir = fsstate.errorDir(ctlog)
}
if len(sths) == 0 {
errors, err := config.State.GetErrors(ctx, ctlog, recentErrorCount)
if err != nil {
return fmt.Errorf("error getting recent errors: %w", err)
}
info := &StaleSTHInfo{
Log: ctlog,
LastSuccess: lastSuccess,
LatestSTH: verifiedSTH,
RecentErrors: errors,
ErrorsDir: errorsDir,
}
if err := config.State.NotifyHealthCheckFailure(ctx, ctlog, info); err != nil {
return fmt.Errorf("error notifying about stale STH: %w", err)
}
} else {
errors, err := config.State.GetErrors(ctx, ctlog, recentErrorCount)
if err != nil {
return fmt.Errorf("error getting recent errors: %w", err)
}
info := &BacklogInfo{
Log: ctlog,
LatestSTH: sths[len(sths)-1],
Position: position,
RecentErrors: errors,
ErrorsDir: errorsDir,
}
if err := config.State.NotifyHealthCheckFailure(ctx, ctlog, info); err != nil {
return fmt.Errorf("error notifying about backlog: %w", err)
}
}
return nil
}
type HealthCheckFailure interface {
Summary() string
Text() string
}
type StaleSTHInfo struct {
Log *loglist.Log
LastSuccess time.Time // may be zero
LatestSTH *cttypes.SignedTreeHead // may be nil
RecentErrors string
ErrorsDir string
}
type BacklogInfo struct {
Log *loglist.Log
LatestSTH *StoredSTH
Position uint64
RecentErrors string
ErrorsDir string
}
type StaleLogListInfo struct {
Source string
LastSuccess time.Time
RecentErrors string
ErrorsDir string
}
func (e *StaleSTHInfo) LastSuccessString() string {
if e.LastSuccess.IsZero() {
return "never"
} else {
return e.LastSuccess.String()
}
}
func (e *BacklogInfo) Backlog() uint64 {
return e.LatestSTH.TreeSize - e.Position
}
func (e *StaleSTHInfo) Summary() string {
return fmt.Sprintf("Unable to contact %s since %s", e.Log.GetMonitoringURL(), e.LastSuccessString())
}
func (e *BacklogInfo) Summary() string {
return fmt.Sprintf("Backlog of size %d from %s", e.Backlog(), e.Log.GetMonitoringURL())
}
func (e *StaleLogListInfo) Summary() string {
return fmt.Sprintf("Unable to retrieve log list since %s", e.LastSuccess)
}
func (e *StaleSTHInfo) Text() string {
text := new(strings.Builder)
fmt.Fprintf(text, "certspotter has been unable to contact %s since %s. Consequentially, certspotter may fail to notify you about certificates in this log.\n", e.Log.GetMonitoringURL(), e.LastSuccessString())
fmt.Fprintf(text, "\n")
if e.LatestSTH != nil {
fmt.Fprintf(text, "Latest known log size = %d\n", e.LatestSTH.TreeSize)
} else {
fmt.Fprintf(text, "Latest known log size = none\n")
}
if e.RecentErrors != "" {
fmt.Fprintf(text, "\n")
fmt.Fprintf(text, "Recent errors (see %s for complete records):\n", e.ErrorsDir)
fmt.Fprintf(text, "\n")
fmt.Fprint(text, e.RecentErrors)
}
return text.String()
}
func (e *BacklogInfo) Text() string {
text := new(strings.Builder)
fmt.Fprintf(text, "certspotter has been unable to download entries from %s in a timely manner. Consequentially, certspotter may be slow to notify you about certificates in this log.\n", e.Log.GetMonitoringURL())
fmt.Fprintf(text, "\n")
fmt.Fprintf(text, "Current log size = %d (as of %s)\n", e.LatestSTH.TreeSize, e.LatestSTH.StoredAt)
fmt.Fprintf(text, "Current position = %d\n", e.Position)
fmt.Fprintf(text, " Backlog = %d\n", e.Backlog())
if e.RecentErrors != "" {
fmt.Fprintf(text, "\n")
fmt.Fprintf(text, "Recent errors (see %s for complete records):\n", e.ErrorsDir)
fmt.Fprintf(text, "\n")
fmt.Fprint(text, e.RecentErrors)
}
return text.String()
}
func (e *StaleLogListInfo) Text() string {
text := new(strings.Builder)
fmt.Fprintf(text, "certspotter has been unable to retrieve the log list from %s since %s.\n", e.Source, e.LastSuccess)
fmt.Fprintf(text, "\n")
fmt.Fprintf(text, "Consequentially, certspotter may not be monitoring all logs, and might fail to detect certificates.\n")
if e.RecentErrors != "" {
fmt.Fprintf(text, "\n")
fmt.Fprintf(text, "Recent errors (see %s for complete records):\n", e.ErrorsDir)
fmt.Fprintf(text, "\n")
fmt.Fprint(text, e.RecentErrors)
}
return text.String()
}
// TODO-3: make the errors more actionable
third_party/certspotter/monitor/loglist.go
Vendored
+45
View File
@@ -0,0 +1,45 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"fmt"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/loglist"
)
type LogID = cttypes.LogID
func getLogList(ctx context.Context, source string, token *loglist.ModificationToken) (map[LogID]*loglist.Log, *loglist.ModificationToken, error) {
list, newToken, err := loglist.LoadIfModified(ctx, source, token)
if err != nil {
return nil, nil, err
}
logs := make(map[LogID]*loglist.Log)
for operatorIndex := range list.Operators {
for logIndex := range list.Operators[operatorIndex].Logs {
log := &list.Operators[operatorIndex].Logs[logIndex]
if _, exists := logs[log.LogID]; exists {
return nil, nil, fmt.Errorf("log list contains more than one entry with ID %s", log.LogID.Base64String())
}
logs[log.LogID] = log
}
for logIndex := range list.Operators[operatorIndex].TiledLogs {
log := &list.Operators[operatorIndex].TiledLogs[logIndex]
if _, exists := logs[log.LogID]; exists {
return nil, nil, fmt.Errorf("log list contains more than one entry with ID %s", log.LogID.Base64String())
}
logs[log.LogID] = log
}
}
return logs, newToken, nil
}
third_party/certspotter/monitor/mailutils.go
Vendored
+34
View File
@@ -0,0 +1,34 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"crypto/rand"
"encoding/hex"
"os"
)
const mailDateFormat = "Mon, 2 Jan 2006 15:04:05 -0700"
func generateMessageID() string {
var randomBytes [16]byte
if _, err := rand.Read(randomBytes[:]); err != nil {
panic(err)
}
return hex.EncodeToString(randomBytes[:]) + "@selfhosted.certspotter.org"
}
func sendmailPath() string {
if envVar := os.Getenv("SENDMAIL_PATH"); envVar != "" {
return envVar
} else {
return "/usr/sbin/sendmail"
}
}
third_party/certspotter/monitor/monitor.go
Vendored
+608
View File
@@ -0,0 +1,608 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"errors"
"fmt"
"golang.org/x/sync/errgroup"
"golang.org/x/time/rate"
"log"
mathrand "math/rand/v2"
"net/url"
"slices"
"time"
"software.sslmate.com/src/certspotter/ctclient"
"software.sslmate.com/src/certspotter/ctcrypto"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/loglist"
"software.sslmate.com/src/certspotter/merkletree"
"software.sslmate.com/src/certspotter/sequencer"
)
const (
getSTHInterval = 5 * time.Minute
maxPartialTileAge = 5 * time.Minute
)
func downloadJobSize(ctlog *loglist.Log) uint64 {
if ctlog.IsStaticCTAPI() {
return ctclient.StaticTileWidth
} else if ctlog.CertspotterDownloadSize != 0 {
return uint64(ctlog.CertspotterDownloadSize)
} else {
return 1000
}
}
func downloadWorkers(ctlog *loglist.Log) int {
if ctlog.CertspotterDownloadQPS != 0 {
// parallelism is effectively governed by the rate limit so for now just hard code a number here
return 10
} else if ctlog.CertspotterDownloadJobs != 0 {
return ctlog.CertspotterDownloadJobs
} else {
return 1
}
}
func downloadRateLimit(ctlog *loglist.Log) rate.Limit {
if ctlog.CertspotterDownloadQPS != 0 {
return rate.Limit(ctlog.CertspotterDownloadQPS)
} else {
return rate.Inf
}
}
type verifyEntriesError struct {
sth *cttypes.SignedTreeHead
entriesRootHash merkletree.Hash
}
func (e *verifyEntriesError) Error() string {
return fmt.Sprintf("error verifying at tree size %d: the STH root hash (%x) does not match the entries returned by the log (%x)", e.sth.TreeSize, e.sth.RootHash, e.entriesRootHash)
}
type partialError struct {
error
}
func (e partialError) Unwrap() error {
return e.error
}
func isPartialError(err error) bool {
var pe partialError
return errors.As(err, &pe)
}
func withRetry(ctx context.Context, config *Config, ctlog *loglist.Log, maxRetries int, f func() error) error {
minSleep := 100 * time.Millisecond
numRetries := 0
for ctx.Err() == nil {
err := f()
if err == nil || errors.Is(err, context.Canceled) {
return err
}
if numRetries > 0 {
err = fmt.Errorf("%w (retried %d times)", err, numRetries)
}
if isPartialError(err) {
recordError(ctx, config, ctlog, err)
return nil
}
if maxRetries != -1 && numRetries >= maxRetries {
return err
}
recordError(ctx, config, ctlog, err)
sleepTime := minSleep + mathrand.N(minSleep)
if err := sleep(ctx, sleepTime); err != nil {
return err
}
minSleep = min(minSleep*2, 30*time.Second)
numRetries++
}
return ctx.Err()
}
func getEntriesFull(ctx context.Context, client ctclient.Log, startInclusive, endExclusive uint64) ([]ctclient.Entry, error) {
allEntries := make([]ctclient.Entry, 0, endExclusive-startInclusive)
for startInclusive < endExclusive {
entries, err := client.GetEntries(ctx, startInclusive, endExclusive-1)
if err != nil {
return nil, err
}
allEntries = append(allEntries, entries...)
startInclusive += uint64(len(entries))
}
return allEntries, nil
}
func getAuthenticSTH(ctx context.Context, ctlog *loglist.Log, client ctclient.Log) (*cttypes.SignedTreeHead, string, error) {
sth, url, err := client.GetSTH(ctx)
if err != nil {
return nil, "", err
}
if err := ctcrypto.PublicKey(ctlog.Key).Verify(ctcrypto.SignatureInputForSTH(sth), sth.Signature); err != nil {
return nil, "", fmt.Errorf("STH has invalid signature: %w", err)
}
return sth, url, nil
}
type logClient struct {
config *Config
log *loglist.Log
client ctclient.Log
lim *rate.Limiter
}
func (client *logClient) GetSTH(ctx context.Context) (sth *cttypes.SignedTreeHead, url string, err error) {
err = withRetry(ctx, client.config, client.log, -1, func() error {
if err := client.lim.Wait(ctx); err != nil {
return err
}
sth, url, err = getAuthenticSTH(ctx, client.log, client.client)
return err
})
return
}
func (client *logClient) GetRoots(ctx context.Context) (roots [][]byte, err error) {
err = withRetry(ctx, client.config, client.log, -1, func() error {
if err := client.lim.Wait(ctx); err != nil {
return err
}
roots, err = client.client.GetRoots(ctx)
return err
})
return
}
func (client *logClient) GetEntries(ctx context.Context, startInclusive, endInclusive uint64) (entries []ctclient.Entry, err error) {
err = withRetry(ctx, client.config, client.log, -1, func() error {
if err := client.lim.Wait(ctx); err != nil {
return err
}
entries, err = client.client.GetEntries(ctx, startInclusive, endInclusive)
if err != nil && len(entries) != 0 {
err = partialError{err}
}
return err
})
return
}
func (client *logClient) ReconstructTree(ctx context.Context, sth *cttypes.SignedTreeHead) (tree *merkletree.CollapsedTree, err error) {
err = withRetry(ctx, client.config, client.log, -1, func() error {
if err := client.lim.Wait(ctx); err != nil {
return err
}
tree, err = client.client.ReconstructTree(ctx, sth)
return err
})
return
}
type issuerGetter struct {
config *Config
log *loglist.Log
logGetter ctclient.IssuerGetter
}
func (ig *issuerGetter) GetIssuer(ctx context.Context, fingerprint *[32]byte) ([]byte, error) {
if issuer, err := ig.config.State.LoadIssuer(ctx, fingerprint); err != nil {
log.Printf("error loading cached issuer %x (issuer will be retrieved from log instead): %s", *fingerprint, err)
} else if issuer != nil {
return issuer, nil
}
var issuer []byte
if err := withRetry(ctx, ig.config, ig.log, 7, func() error {
var err error
issuer, err = ig.logGetter.GetIssuer(ctx, fingerprint)
return err
}); err != nil {
return nil, err
}
if err := ig.config.State.StoreIssuer(ctx, fingerprint, issuer); err != nil {
log.Printf("error caching issuer %x (issuer will be re-retrieved from log in the future): %s", *fingerprint, err)
}
return issuer, nil
}
func newLogClient(config *Config, ctlog *loglist.Log) (ctclient.Log, ctclient.IssuerGetter, error) {
switch {
case ctlog.IsRFC6962():
logURL, err := url.Parse(ctlog.URL)
if err != nil {
return nil, nil, fmt.Errorf("log has invalid URL: %w", err)
}
return &logClient{
config: config,
log: ctlog,
client: &ctclient.RFC6962Log{URL: logURL},
lim: rate.NewLimiter(downloadRateLimit(ctlog), 1),
}, nil, nil
case ctlog.IsStaticCTAPI():
submissionURL, err := url.Parse(ctlog.SubmissionURL)
if err != nil {
return nil, nil, fmt.Errorf("log has invalid submission URL: %w", err)
}
monitoringURL, err := url.Parse(ctlog.MonitoringURL)
if err != nil {
return nil, nil, fmt.Errorf("log has invalid monitoring URL: %w", err)
}
client := &ctclient.StaticLog{
SubmissionURL: submissionURL,
MonitoringURL: monitoringURL,
ID: ctlog.LogID,
}
return &logClient{
config: config,
log: ctlog,
client: client,
lim: rate.NewLimiter(downloadRateLimit(ctlog), 1),
}, &issuerGetter{
config: config,
log: ctlog,
logGetter: client,
}, nil
default:
return nil, nil, errors.New("log uses unknown protocol")
}
}
func monitorLogContinously(ctx context.Context, config *Config, ctlog *loglist.Log) (returnedErr error) {
client, issuerGetter, err := newLogClient(config, ctlog)
if err != nil {
return err
}
if err := config.State.PrepareLog(ctx, ctlog.LogID); err != nil {
return fmt.Errorf("error preparing state: %w", err)
}
state, err := config.State.LoadLogState(ctx, ctlog.LogID)
if err != nil {
return fmt.Errorf("error loading log state: %w", err)
}
if state == nil {
if config.StartAtEnd {
sth, _, err := client.GetSTH(ctx)
if err != nil {
return err
}
tree, err := client.ReconstructTree(ctx, sth)
if err != nil {
return err
}
state = &LogState{
DownloadPosition: tree,
VerifiedPosition: tree,
VerifiedSTH: sth,
LastSuccess: time.Now(),
}
} else {
state = &LogState{
DownloadPosition: merkletree.EmptyCollapsedTree(),
VerifiedPosition: merkletree.EmptyCollapsedTree(),
VerifiedSTH: nil,
LastSuccess: time.Now(),
}
}
if config.Verbose {
log.Printf("%s: monitoring brand new log starting from position %d", ctlog.GetMonitoringURL(), state.DownloadPosition.Size())
}
if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
return fmt.Errorf("error storing log state: %w", err)
}
} else {
if config.Verbose {
log.Printf("%s: resuming monitoring from position %d", ctlog.GetMonitoringURL(), state.DownloadPosition.Size())
}
}
defer func() {
storeCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := config.State.StoreLogState(storeCtx, ctlog.LogID, state); err != nil && returnedErr == nil {
returnedErr = fmt.Errorf("error storing log state: %w", err)
}
}()
retry:
position := state.DownloadPosition.Size()
// logs are monitored using the following pipeline of workers, with each worker sending results to the next worker:
// 1 getSTHWorker ==> 1 generateBatchesWorker ==> multiple downloadWorkers ==> multiple processWorkers ==> 1 saveStateWorker
// getSTHWorker - periodically download STHs from the log
// generateBatchesWorker - generate batches of work
// downloadWorkers - download the entries in each batch
// processWorkers - process the certificates (store/notify if matches watch list) in each batch
// saveStateWorker - builds the Merkle Tree and compares against STHs
sths := make(chan *cttypes.SignedTreeHead, 1)
batches := make(chan *batch, downloadWorkers(ctlog))
processedBatches := sequencer.New[batch](0, uint64(downloadWorkers(ctlog))*10)
group, gctx := errgroup.WithContext(ctx)
group.Go(func() error { return getSTHWorker(gctx, config, ctlog, client, sths) })
group.Go(func() error { return generateBatchesWorker(gctx, config, ctlog, position, sths, batches) })
for range downloadWorkers(ctlog) {
downloadedBatches := make(chan *batch, 1)
group.Go(func() error { return downloadWorker(gctx, config, ctlog, client, batches, downloadedBatches) })
group.Go(func() error {
return processWorker(gctx, config, ctlog, issuerGetter, downloadedBatches, processedBatches)
})
}
group.Go(func() error { return saveStateWorker(gctx, config, ctlog, state, processedBatches) })
err = group.Wait()
if verifyErr := (*verifyEntriesError)(nil); errors.As(err, &verifyErr) {
recordError(ctx, config, ctlog, verifyErr)
state.rewindDownloadPosition()
if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
return fmt.Errorf("error storing log state: %w", err)
}
if err := sleep(ctx, 5*time.Minute); err != nil {
return err
}
goto retry
}
return err
}
func getSTHWorker(ctx context.Context, config *Config, ctlog *loglist.Log, client ctclient.Log, sthsOut chan<- *cttypes.SignedTreeHead) error {
ticker := time.NewTicker(getSTHInterval)
defer ticker.Stop()
for {
sth, _, err := client.GetSTH(ctx)
if err != nil {
return err
}
select {
case <-ctx.Done():
return ctx.Err()
case sthsOut <- sth:
}
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
}
}
}
type batch struct {
number uint64
begin, end uint64
discoveredAt time.Time // time at which we became aware of the log having entries in range [begin,end)
sths []*StoredSTH // STHs with sizes in range [begin,end], sorted by TreeSize
entries []ctclient.Entry // in range [begin,end)
}
// Create a batch starting from begin, based on sths (which must be non-empty, sorted by TreeSize, and contain only STHs with TreeSize >= begin). Returns the batch, plus the remaining STHs.
func newBatch(number uint64, begin uint64, sths []*StoredSTH, downloadJobSize uint64) (*batch, []*StoredSTH) {
batch := &batch{
number: number,
begin: begin,
discoveredAt: sths[0].StoredAt,
}
maxEnd := (begin/downloadJobSize + 1) * downloadJobSize
for _, sth := range sths {
if sth.StoredAt.Before(batch.discoveredAt) {
batch.discoveredAt = sth.StoredAt
}
if sth.TreeSize <= maxEnd {
batch.end = sth.TreeSize
batch.sths = append(batch.sths, sth)
} else {
batch.end = maxEnd
break
}
}
return batch, sths[len(batch.sths):]
}
// insert sth into sths, which is sorted by TreeSize, and return a new, still-sorted slice.
// if an equivalent STH is already in sths, it is returned unchanged.
func insertSTH(sths []*StoredSTH, sth *StoredSTH) []*StoredSTH {
i := len(sths)
for i > 0 {
if sths[i-1].Same(&sth.SignedTreeHead) {
return sths
}
if sths[i-1].TreeSize < sth.TreeSize {
break
}
i--
}
return slices.Insert(sths, i, sth)
}
func generateBatchesWorker(ctx context.Context, config *Config, ctlog *loglist.Log, position uint64, sthsIn <-chan *cttypes.SignedTreeHead, batchesOut chan<- *batch) error {
downloadJobSize := downloadJobSize(ctlog)
sths, err := config.State.LoadSTHs(ctx, ctlog.LogID)
if err != nil {
return fmt.Errorf("error loading STHs: %w", err)
}
// sths is sorted by TreeSize but may contain STHs with TreeSize < position; get rid of them
for len(sths) > 0 && sths[0].TreeSize < position {
// TODO-4: audit sths[0] against log's verified STH
if err := config.State.RemoveSTH(ctx, ctlog.LogID, &sths[0].SignedTreeHead); err != nil {
return fmt.Errorf("error removing STH: %w", err)
}
sths = sths[1:]
}
// from this point, sths is sorted by TreeSize and contains only STHs with TreeSize >= position
handleSTH := func(sth *cttypes.SignedTreeHead) error {
if sth.TreeSize < position {
// TODO-4: audit against log's verified STH
} else {
storedSTH, err := config.State.StoreSTH(ctx, ctlog.LogID, sth)
if err != nil {
return fmt.Errorf("error storing STH: %w", err)
}
sths = insertSTH(sths, storedSTH)
}
return nil
}
var number uint64
for {
for len(sths) == 0 {
select {
case <-ctx.Done():
return ctx.Err()
case sth := <-sthsIn:
if err := handleSTH(sth); err != nil {
return err
}
}
}
batch, remainingSTHs := newBatch(number, position, sths, downloadJobSize)
if ctlog.IsStaticCTAPI() && batch.end%downloadJobSize != 0 {
// Wait to download this partial tile until it's old enough
if age := time.Since(batch.discoveredAt); age < maxPartialTileAge {
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(maxPartialTileAge - age):
case sth := <-sthsIn:
if err := handleSTH(sth); err != nil {
return err
}
continue
}
}
}
select {
case <-ctx.Done():
return ctx.Err()
case sth := <-sthsIn:
if err := handleSTH(sth); err != nil {
return err
}
case batchesOut <- batch:
number = batch.number + 1
position = batch.end
sths = remainingSTHs
}
}
}
func downloadWorker(ctx context.Context, config *Config, ctlog *loglist.Log, client ctclient.Log, batchesIn <-chan *batch, batchesOut chan<- *batch) error {
for {
var batch *batch
select {
case <-ctx.Done():
return ctx.Err()
case batch = <-batchesIn:
}
entries, err := getEntriesFull(ctx, client, batch.begin, batch.end)
if err != nil {
return err
}
batch.entries = entries
select {
case <-ctx.Done():
return ctx.Err()
case batchesOut <- batch:
}
}
}
func processWorker(ctx context.Context, config *Config, ctlog *loglist.Log, issuerGetter ctclient.IssuerGetter, batchesIn <-chan *batch, batchesOut *sequencer.Channel[batch]) error {
for {
var batch *batch
select {
case <-ctx.Done():
return ctx.Err()
case batch = <-batchesIn:
}
for offset, entry := range batch.entries {
index := batch.begin + uint64(offset)
if err := processLogEntry(ctx, config, issuerGetter, &LogEntry{
Entry: entry,
Index: index,
Log: ctlog,
}); err != nil {
return fmt.Errorf("error processing entry %d: %w", index, err)
}
}
if err := batchesOut.Add(ctx, batch.number, batch); err != nil {
return err
}
}
}
func saveStateWorker(ctx context.Context, config *Config, ctlog *loglist.Log, state *LogState, batchesIn *sequencer.Channel[batch]) error {
for {
batch, err := batchesIn.Next(ctx)
if err != nil {
return err
}
if batch.begin != state.DownloadPosition.Size() {
panic(fmt.Errorf("saveStateWorker: expected batch to start at %d but got %d instead", state.DownloadPosition.Size(), batch.begin))
}
for {
for len(batch.sths) > 0 && batch.sths[0].TreeSize == state.DownloadPosition.Size() {
sth := batch.sths[0]
batch.sths = batch.sths[1:]
if rootHash := state.DownloadPosition.CalculateRoot(); sth.RootHash != rootHash {
return &verifyEntriesError{
sth: &sth.SignedTreeHead,
entriesRootHash: rootHash,
}
}
state.advanceVerifiedPosition()
state.LastSuccess = sth.StoredAt
state.VerifiedSTH = &sth.SignedTreeHead
if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
return fmt.Errorf("error storing log state: %w", err)
}
// don't remove the STH until state has been durably stored
if err := config.State.RemoveSTH(ctx, ctlog.LogID, &sth.SignedTreeHead); err != nil {
return fmt.Errorf("error removing verified STH: %w", err)
}
if config.Verbose {
log.Printf("%s: verified position is now %d", ctlog.GetMonitoringURL(), sth.SignedTreeHead.TreeSize)
}
}
if len(batch.entries) == 0 {
break
}
entry := batch.entries[0]
batch.entries = batch.entries[1:]
leafHash := merkletree.HashLeaf(entry.LeafInput())
state.DownloadPosition.Add(leafHash)
}
if err := config.State.StoreLogState(ctx, ctlog.LogID, state); err != nil {
return fmt.Errorf("error storing log state: %w", err)
}
}
}
func sleep(ctx context.Context, duration time.Duration) error {
timer := time.NewTimer(duration)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
third_party/certspotter/monitor/notify.go
Vendored
+166
View File
@@ -0,0 +1,166 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"bytes"
"context"
"errors"
"fmt"
"io/fs"
"os"
"os/exec"
"path/filepath"
"strings"
"sync"
"time"
)
var stdoutMu sync.Mutex
type notification struct {
environ []string
summary string
text string
}
func (s *FilesystemState) notify(ctx context.Context, notif *notification) error {
if s.Stdout {
writeToStdout(notif)
}
if len(s.Email) > 0 {
if err := sendEmail(ctx, s.Email, notif); err != nil {
return err
}
}
if s.Script != "" {
if err := execScript(ctx, s.Script, notif); err != nil {
return err
}
}
if s.ScriptDir != "" {
if err := execScriptDir(ctx, s.ScriptDir, notif); err != nil {
return err
}
}
return nil
}
func writeToStdout(notif *notification) {
stdoutMu.Lock()
defer stdoutMu.Unlock()
os.Stdout.WriteString(notif.text + "\n")
}
func sendEmail(ctx context.Context, to []string, notif *notification) error {
stdin := new(bytes.Buffer)
stderr := new(bytes.Buffer)
from := os.Getenv("EMAIL")
if from != "" {
fmt.Fprintf(stdin, "From: %s\n", from)
}
fmt.Fprintf(stdin, "To: %s\n", strings.Join(to, ", "))
fmt.Fprintf(stdin, "Subject: [certspotter] %s\n", notif.summary)
fmt.Fprintf(stdin, "Date: %s\n", time.Now().Format(mailDateFormat))
fmt.Fprintf(stdin, "Message-ID: <%s>\n", generateMessageID())
fmt.Fprintf(stdin, "Mime-Version: 1.0\n")
fmt.Fprintf(stdin, "Content-Type: text/plain; charset=US-ASCII\n")
fmt.Fprintf(stdin, "X-Mailer: certspotter\n")
fmt.Fprintf(stdin, "\n")
fmt.Fprint(stdin, notif.text)
args := []string{"-i"}
if from != "" {
args = append(args, "-f", from)
}
args = append(args, "--")
args = append(args, to...)
sendmailCtx, cancel := context.WithDeadline(ctx, time.Now().Add(2*time.Minute))
defer cancel()
sendmail := exec.CommandContext(sendmailCtx, sendmailPath(), args...)
sendmail.Stdin = stdin
sendmail.Stderr = stderr
sendmail.WaitDelay = 5 * time.Second
if err := sendmail.Run(); err == nil || err == exec.ErrWaitDelay {
return nil
} else if sendmailCtx.Err() != nil && ctx.Err() == nil {
return fmt.Errorf("error sending email to %v: sendmail command timed out", to)
} else if ctx.Err() != nil {
// if the context was canceled, we can't be sure that the error is the fault of sendmail, so ignore it
return ctx.Err()
} else if exitErr, isExitError := err.(*exec.ExitError); isExitError && exitErr.Exited() {
return fmt.Errorf("error sending email to %v: sendmail failed with exit code %d and error %q", to, exitErr.ExitCode(), strings.TrimSpace(stderr.String()))
} else if isExitError {
return fmt.Errorf("error sending email to %v: sendmail terminated by signal with error %q", to, strings.TrimSpace(stderr.String()))
} else {
return fmt.Errorf("error sending email to %v: error running sendmail command: %w", to, err)
}
}
func execScript(ctx context.Context, scriptName string, notif *notification) error {
stderr := new(bytes.Buffer)
cmd := exec.CommandContext(ctx, scriptName)
cmd.Env = os.Environ()
cmd.Env = append(cmd.Env, notif.environ...)
cmd.Stderr = stderr
cmd.WaitDelay = 5 * time.Second
if err := cmd.Run(); err == nil || err == exec.ErrWaitDelay {
return nil
} else if ctx.Err() != nil {
// if the context was canceled, we can't be sure that the error is the fault of the script, so ignore it
return ctx.Err()
} else if exitErr, isExitError := err.(*exec.ExitError); isExitError && exitErr.Exited() {
return fmt.Errorf("script %q exited with code %d and error %q", scriptName, exitErr.ExitCode(), strings.TrimSpace(stderr.String()))
} else if isExitError {
return fmt.Errorf("script %q terminated by signal with error %q", scriptName, strings.TrimSpace(stderr.String()))
} else {
return fmt.Errorf("error executing script: %w", err)
}
}
func execScriptDir(ctx context.Context, dirPath string, notif *notification) error {
dirents, err := os.ReadDir(dirPath)
if errors.Is(err, fs.ErrNotExist) {
return nil
} else if err != nil {
return fmt.Errorf("error executing scripts in directory %q: %w", dirPath, err)
}
for _, dirent := range dirents {
if strings.HasPrefix(dirent.Name(), ".") {
continue
}
scriptPath := filepath.Join(dirPath, dirent.Name())
info, err := os.Stat(scriptPath)
if errors.Is(err, fs.ErrNotExist) {
continue
} else if err != nil {
return fmt.Errorf("error executing %q in directory %q: %w", dirent.Name(), dirPath, err)
} else if info.Mode().IsRegular() && isExecutable(info.Mode()) {
if err := execScript(ctx, scriptPath, notif); err != nil {
return err
}
}
}
return nil
}
func isExecutable(mode os.FileMode) bool {
return mode&0111 != 0
}
third_party/certspotter/monitor/process.go
Vendored
+151
View File
@@ -0,0 +1,151 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"crypto/sha256"
"errors"
"fmt"
"software.sslmate.com/src/certspotter"
"software.sslmate.com/src/certspotter/ctclient"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/loglist"
)
type LogEntry struct {
ctclient.Entry
Index uint64
Log *loglist.Log
}
func processLogEntry(ctx context.Context, config *Config, issuerGetter ctclient.IssuerGetter, entry *LogEntry) error {
leaf, err := cttypes.ParseLeafInput(entry.LeafInput())
if err != nil {
return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error parsing Merkle Tree Leaf: %w", err))
}
switch leaf.TimestampedEntry.EntryType {
case cttypes.X509EntryType:
return processX509LogEntry(ctx, config, issuerGetter, entry, leaf.TimestampedEntry.SignedEntryASN1Cert)
case cttypes.PrecertEntryType:
return processPrecertLogEntry(ctx, config, issuerGetter, entry, leaf.TimestampedEntry.SignedEntryPreCert)
default:
return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("unknown log entry type %d", leaf.TimestampedEntry.EntryType))
}
}
func processX509LogEntry(ctx context.Context, config *Config, issuerGetter ctclient.IssuerGetter, entry *LogEntry, cert *cttypes.ASN1Cert) error {
certInfo, err := certspotter.MakeCertInfoFromRawCert(*cert)
if err != nil {
return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error parsing X.509 certificate: %w", err))
}
if precertTBS, err := certspotter.ReconstructPrecertTBS(certInfo.TBS); err == nil {
certInfo.TBS = precertTBS
} else {
return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error reconstructing precertificate TBSCertificate: %w", err))
}
getChain := func(ctx context.Context) ([]cttypes.ASN1Cert, error) {
var (
chain = []cttypes.ASN1Cert{*cert}
errs = []error{}
)
if issuers, err := entry.GetChain(ctx, issuerGetter); err == nil {
chain = append(chain, issuers...)
} else {
errs = append(errs, err)
}
return chain, errors.Join(errs...)
}
return processCertificate(ctx, config, entry, certInfo, getChain)
}
func processPrecertLogEntry(ctx context.Context, config *Config, issuerGetter ctclient.IssuerGetter, entry *LogEntry, precert *cttypes.PreCert) error {
certInfo, err := certspotter.MakeCertInfoFromRawTBS(precert.TBSCertificate)
if err != nil {
return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error parsing precert TBSCertificate: %w", err))
}
precertBytes, err := entry.Precertificate()
if err != nil {
return processMalformedLogEntry(ctx, config, entry, fmt.Errorf("error getting precert entry's precertificate: %w", err))
}
getChain := func(ctx context.Context) ([]cttypes.ASN1Cert, error) {
var (
chain = []cttypes.ASN1Cert{precertBytes}
errs = []error{}
)
if issuers, err := entry.GetChain(ctx, issuerGetter); err == nil {
chain = append(chain, issuers...)
} else {
errs = append(errs, err)
}
if _, err := certspotter.ValidatePrecert(precertBytes, precert.TBSCertificate); err != nil {
errs = append(errs, fmt.Errorf("precertificate in extra_data does not match TBSCertificate in leaf_input: %w", err))
}
return chain, errors.Join(errs...)
}
return processCertificate(ctx, config, entry, certInfo, getChain)
}
func processCertificate(ctx context.Context, config *Config, entry *LogEntry, certInfo *certspotter.CertInfo, getChain func(context.Context) ([]cttypes.ASN1Cert, error)) error {
identifiers, err := certInfo.ParseIdentifiers()
if err != nil {
return processMalformedLogEntry(ctx, config, entry, err)
}
// CTCLOUD PATCH: hand identifiers directly to the in-process sink and skip
// the watchlist/chain/notification path. See CTCLOUD_PATCH.md.
if config.IdentifierSink != nil {
config.IdentifierSink.HandleIdentifiers(ctx, identifiers)
return nil
}
matched, watchItem := config.WatchList.Matches(identifiers)
if !matched {
return nil
}
chain, chainErr := getChain(ctx)
if chainErr != nil {
if ctx.Err() != nil {
// Getting chain failed, but it was probably because our context
// has been canceled, so just act like we never called getChain.
return ctx.Err()
}
// Although getting the chain failed, we still want to notify
// the user about the matching certificate. We'll include chainErr in the
// notification so the user knows why the chain is missing or incorrect.
}
cert := &DiscoveredCert{
WatchItem: watchItem,
LogEntry: entry,
Info: certInfo,
Chain: chain,
ChainError: chainErr,
TBSSHA256: sha256.Sum256(certInfo.TBS.Raw),
SHA256: sha256.Sum256(chain[0]),
PubkeySHA256: sha256.Sum256(certInfo.TBS.PublicKey.FullBytes),
Identifiers: identifiers,
}
if err := config.State.NotifyCert(ctx, cert); err != nil {
return fmt.Errorf("error notifying about certificate %x: %w", cert.SHA256, err)
}
return nil
}
func processMalformedLogEntry(ctx context.Context, config *Config, entry *LogEntry, parseError error) error {
if err := config.State.NotifyMalformedEntry(ctx, entry, parseError); err != nil {
return fmt.Errorf("error notifying about malformed log entry %d in %s (%q): %w", entry.Index, entry.Log.GetMonitoringURL(), parseError, err)
}
return nil
}
third_party/certspotter/monitor/state.go
Vendored
+91
View File
@@ -0,0 +1,91 @@
// Copyright (C) 2024 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"context"
"time"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/loglist"
"software.sslmate.com/src/certspotter/merkletree"
)
type LogState struct {
DownloadPosition *merkletree.CollapsedTree `json:"download_position"`
VerifiedPosition *merkletree.CollapsedTree `json:"verified_position"`
VerifiedSTH *cttypes.SignedTreeHead `json:"verified_sth"`
LastSuccess time.Time `json:"last_success"`
}
func (state *LogState) rewindDownloadPosition() {
position := state.VerifiedPosition.Clone()
state.DownloadPosition = &position
}
func (state *LogState) advanceVerifiedPosition() {
position := state.DownloadPosition.Clone()
state.VerifiedPosition = &position
}
// Methods are safe to call concurrently.
type StateProvider interface {
// Initialize the state. Called before any other method in this interface.
// Idempotent: returns nil if the state is already initialized.
Prepare(context.Context) error
// Initialize the state for the given log. Called before any other method
// with the log ID. Idempotent: returns nil if log state already initialized.
PrepareLog(context.Context, LogID) error
// Store log state for retrieval by LoadLogState.
StoreLogState(context.Context, LogID, *LogState) error
// Load log state that was previously stored with StoreLogState.
// Returns nil, nil if StoreLogState has not been called yet for this log.
LoadLogState(context.Context, LogID) (*LogState, error)
// Store STH for retrieval by LoadSTHs. If an STH with the same
// timestamp and root hash is already stored, this STH can be ignored.
StoreSTH(context.Context, LogID, *cttypes.SignedTreeHead) (*StoredSTH, error)
// Load all STHs for this log previously stored with StoreSTH.
// The returned slice must be sorted by tree size.
LoadSTHs(context.Context, LogID) ([]*StoredSTH, error)
// Remove an STH so it is no longer returned by LoadSTHs.
RemoveSTH(context.Context, LogID, *cttypes.SignedTreeHead) error
// Store a DER-encoded issuer certificate with the given fingerprint for
// retrieval by LoadIssuer. Returns nil if the issuer has already been stored.
StoreIssuer(context.Context, *[32]byte, []byte) error
// Retrieve a DER-encoded issuer certificate previously stored with StoreIssuer.
// Returns nil, nil if this issuer certificate has not been stored.
LoadIssuer(context.Context, *[32]byte) ([]byte, error)
// Called when a certificate matching the watch list is discovered.
NotifyCert(context.Context, *DiscoveredCert) error
// Called when certspotter fails to parse a log entry.
NotifyMalformedEntry(context.Context, *LogEntry, error) error
// Called when a health check fails. The log is nil if the
// feailure is not associated with a log.
NotifyHealthCheckFailure(context.Context, *loglist.Log, HealthCheckFailure) error
// Called when a non-fatal error occurs. The log is nil if the error is
// not associated with a log. Note that most errors are transient, and
// certspotter will retry the failed operation later.
NotifyError(context.Context, *loglist.Log, error) error
// Retrieve the specified number of most recent errors.
GetErrors(context.Context, *loglist.Log, int) (string, error)
}
third_party/certspotter/monitor/statedir.go
Vendored
+167
View File
@@ -0,0 +1,167 @@
// Copyright (C) 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"encoding/json"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"software.sslmate.com/src/certspotter/cttypes"
"software.sslmate.com/src/certspotter/merkletree"
"strconv"
"strings"
"time"
)
func readVersion(stateDir string) (int, error) {
path := filepath.Join(stateDir, "version")
fileBytes, err := os.ReadFile(path)
if errors.Is(err, fs.ErrNotExist) {
if fileExists(filepath.Join(stateDir, "evidence")) {
return 0, nil
} else {
return -1, nil
}
} else if err != nil {
return -1, err
}
version, err := strconv.Atoi(strings.TrimSpace(string(fileBytes)))
if err != nil {
return -1, fmt.Errorf("version file %q is malformed: %w", path, err)
}
return version, nil
}
func writeVersion(stateDir string) error {
return writeFile(filepath.Join(stateDir, "version"), []byte{'2', '\n'}, 0666)
}
func migrateLogStateDirV1(dir string) error {
var sth cttypes.SignedTreeHead
var tree merkletree.CollapsedTree
sthPath := filepath.Join(dir, "sth.json")
sthData, err := os.ReadFile(sthPath)
if errors.Is(err, fs.ErrNotExist) {
return nil
} else if err != nil {
return err
}
treePath := filepath.Join(dir, "tree.json")
treeData, err := os.ReadFile(treePath)
if errors.Is(err, fs.ErrNotExist) {
return nil
} else if err != nil {
return err
}
if err := json.Unmarshal(sthData, &sth); err != nil {
return fmt.Errorf("error unmarshaling %s: %w", sthPath, err)
}
if err := json.Unmarshal(treeData, &tree); err != nil {
return fmt.Errorf("error unmarshaling %s: %w", treePath, err)
}
stateFile := LogState{
DownloadPosition: &tree,
VerifiedPosition: &tree,
VerifiedSTH: &sth,
LastSuccess: time.Now(),
}
if err := writeJSONFile(filepath.Join(dir, "state.json"), stateFile, 0666); err != nil {
return err
}
if err := os.Remove(sthPath); err != nil {
return err
}
if err := os.Remove(treePath); err != nil {
return err
}
return nil
}
func migrateStateDirV1(stateDir string) error {
if lockfile := filepath.Join(stateDir, "lock"); fileExists(lockfile) {
return fmt.Errorf("directory is locked by another instance of certspotter; remove %s if this is not the case", lockfile)
}
if logDirs, err := os.ReadDir(filepath.Join(stateDir, "logs")); err == nil {
for _, logDir := range logDirs {
if strings.HasPrefix(logDir.Name(), ".") || !logDir.IsDir() {
continue
}
if err := migrateLogStateDirV1(filepath.Join(stateDir, "logs", logDir.Name())); err != nil {
return fmt.Errorf("error migrating log state: %w", err)
}
}
} else if !errors.Is(err, fs.ErrNotExist) {
return err
}
if err := writeVersion(stateDir); err != nil {
return err
}
if err := os.Remove(filepath.Join(stateDir, "once")); err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
func prepareStateDir(stateDir string) error {
if err := os.Mkdir(stateDir, 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return err
}
if version, err := readVersion(stateDir); err != nil {
return err
} else if version == -1 {
if err := writeVersion(stateDir); err != nil {
return err
}
} else if version == 0 {
return fmt.Errorf("%s was created by a very old version of certspotter; run any version of certspotter after 0.2 and before 0.15.0 to upgrade this directory, or remove it to start from scratch", stateDir)
} else if version == 1 {
if err := migrateStateDirV1(stateDir); err != nil {
return err
}
} else if version > 2 {
return fmt.Errorf("%s was created by a newer version of certspotter; upgrade to the latest version of certspotter or remove this directory to start from scratch", stateDir)
}
for _, subdir := range []string{"certs", "logs", "healthchecks", "errors"} {
if err := os.Mkdir(filepath.Join(stateDir, subdir), 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return err
}
}
return nil
}
func prepareCacheDir(cacheDir string) error {
if err := os.MkdirAll(cacheDir, 0777); err != nil {
return err
}
for _, subdir := range []string{"issuers"} {
if err := os.Mkdir(filepath.Join(cacheDir, subdir), 0777); err != nil && !errors.Is(err, fs.ErrExist) {
return err
}
}
return nil
}
third_party/certspotter/monitor/sthdir.go
Vendored
+178
View File
@@ -0,0 +1,178 @@
// Copyright (C) 2017, 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"cmp"
"crypto/sha256"
"encoding/base64"
"encoding/binary"
"encoding/json"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"slices"
"software.sslmate.com/src/certspotter/cttypes"
"strconv"
"strings"
"time"
)
type StoredSTH struct {
cttypes.SignedTreeHead
StoredAt time.Time `json:"stored_at"` // time at which the STH was first stored
}
func loadSTHsFromDir(dirPath string) ([]*StoredSTH, error) {
entries, err := os.ReadDir(dirPath)
if errors.Is(err, fs.ErrNotExist) {
return []*StoredSTH{}, nil
} else if err != nil {
return nil, err
}
sths := make([]*StoredSTH, 0, len(entries))
for _, entry := range entries {
filename := entry.Name()
if strings.HasPrefix(filename, ".") || !strings.HasSuffix(filename, ".json") {
continue
}
sth, err := readSTHFile(filepath.Join(dirPath, filename))
if errors.Is(err, fs.ErrNotExist) {
continue
} else if err != nil {
return nil, err
}
sths = append(sths, sth)
}
slices.SortFunc(sths, func(a, b *StoredSTH) int { return cmp.Compare(a.TreeSize, b.TreeSize) })
return sths, nil
}
// loadLargestSTHFromDir returns the STH in dirPath with the largest tree size, breaking
// ties by later timestamp.
func loadLargestSTHFromDir(dirPath string) (*StoredSTH, error) {
compareSTH := func(a, b *StoredSTH) int {
return cmp.Or(
cmp.Compare(a.TreeSize, b.TreeSize),
cmp.Compare(a.Timestamp, b.Timestamp),
)
}
entries, err := os.ReadDir(dirPath)
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
} else if err != nil {
return nil, err
}
var largestSTH *StoredSTH
for _, entry := range entries {
filename := entry.Name()
if strings.HasPrefix(filename, ".") || !strings.HasSuffix(filename, ".json") {
continue
}
if largestSTH != nil {
if treeSize, ok := sthFilenameTreeSize(filename); ok && treeSize < largestSTH.TreeSize {
continue
}
}
sth, err := readSTHFile(filepath.Join(dirPath, filename))
if errors.Is(err, fs.ErrNotExist) {
continue
} else if err != nil {
return nil, err
}
if largestSTH == nil || compareSTH(sth, largestSTH) > 0 {
largestSTH = sth
}
}
return largestSTH, nil
}
func readSTHFile(filePath string) (*StoredSTH, error) {
file, err := os.Open(filePath)
if err != nil {
return nil, err
}
defer file.Close()
info, err := file.Stat()
if err != nil {
return nil, err
}
sth := &StoredSTH{
StoredAt: info.ModTime(),
}
fileBytes, err := io.ReadAll(file)
if err != nil {
return nil, fmt.Errorf("error reading %s: %w", filePath, err)
}
if err := json.Unmarshal(fileBytes, &sth.SignedTreeHead); err != nil {
return nil, fmt.Errorf("error parsing %s: %w", filePath, err)
}
return sth, nil
}
func storeSTHInDir(dirPath string, sth *cttypes.SignedTreeHead) (*StoredSTH, error) {
filePath := filepath.Join(dirPath, sthFilename(sth))
if info, err := os.Lstat(filePath); err == nil {
return &StoredSTH{
SignedTreeHead: *sth,
StoredAt: info.ModTime(),
}, nil
} else if !errors.Is(err, fs.ErrNotExist) {
return nil, err
}
if err := writeJSONFile(filePath, sth, 0666); err != nil {
return nil, err
}
return &StoredSTH{
SignedTreeHead: *sth,
StoredAt: time.Now(), // not the exact modtime of the file, but close enough for our purposes
}, nil
}
func removeSTHFromDir(dirPath string, sth *cttypes.SignedTreeHead) error {
filePath := filepath.Join(dirPath, sthFilename(sth))
err := os.Remove(filePath)
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
// generate a filename that uniquely identifies the STH (within the context of a particular log)
func sthFilename(sth *cttypes.SignedTreeHead) string {
hasher := sha256.New()
binary.Write(hasher, binary.LittleEndian, sth.Timestamp)
hasher.Write(sth.RootHash[:])
return strconv.FormatUint(sth.TreeSize, 10) + "-" + base64.RawURLEncoding.EncodeToString(hasher.Sum(nil)) + ".json"
}
// sthFilenameTreeSize returns the tree size encoded in the filename, or (0, false)
// if the filename doesn't parse.
func sthFilenameTreeSize(filename string) (uint64, bool) {
separator := strings.IndexByte(filename, '-')
if separator <= 0 {
return 0, false
}
treeSize, err := strconv.ParseUint(filename[:separator], 10, 64)
if err != nil {
return 0, false
}
return treeSize, true
}
third_party/certspotter/monitor/watchlist.go
Vendored
+135
View File
@@ -0,0 +1,135 @@
// Copyright (C) 2016, 2023 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package monitor
import (
"bufio"
"fmt"
"golang.org/x/net/idna"
"io"
"software.sslmate.com/src/certspotter"
"strings"
)
type WatchItem struct {
domain []string
acceptSuffix bool
}
type WatchList []WatchItem
func ParseWatchItem(str string) (WatchItem, error) {
fields := strings.Fields(str)
if len(fields) == 0 {
return WatchItem{}, fmt.Errorf("empty domain")
}
domain := fields[0]
for _, field := range fields[1:] {
switch {
case strings.HasPrefix(field, "valid_at:"):
// Ignore for backwards compatibility
default:
return WatchItem{}, fmt.Errorf("unknown parameter %q", field)
}
}
if domain == "." {
// "." as in root zone -> matches everything
return WatchItem{
domain: []string{},
acceptSuffix: true,
}, nil
}
acceptSuffix := false
if strings.HasPrefix(domain, ".") {
acceptSuffix = true
domain = domain[1:]
}
asciiDomain, err := idna.ToASCII(strings.ToLower(strings.TrimRight(domain, ".")))
if err != nil {
return WatchItem{}, fmt.Errorf("invalid domain %q (%w)", domain, err)
}
return WatchItem{
domain: strings.Split(asciiDomain, "."),
acceptSuffix: acceptSuffix,
}, nil
}
func ReadWatchList(reader io.Reader) (WatchList, error) {
items := make(WatchList, 0, 50)
scanner := bufio.NewScanner(reader)
lineNo := 0
for scanner.Scan() {
line := scanner.Text()
lineNo++
if line == "" || strings.HasPrefix(line, "#") {
continue
}
item, err := ParseWatchItem(line)
if err != nil {
return nil, fmt.Errorf("%w on line %d", err, lineNo)
}
items = append(items, item)
}
return items, scanner.Err()
}
func (item WatchItem) String() string {
if item.acceptSuffix {
return "." + strings.Join(item.domain, ".")
} else {
return strings.Join(item.domain, ".")
}
}
func (item WatchItem) matchesDNSName(dnsName []string) bool {
watchDomain := item.domain
for len(dnsName) > 0 && len(watchDomain) > 0 {
certLabel := dnsName[len(dnsName)-1]
watchLabel := watchDomain[len(watchDomain)-1]
if !dnsLabelMatches(certLabel, watchLabel) {
return false
}
dnsName = dnsName[:len(dnsName)-1]
watchDomain = watchDomain[:len(watchDomain)-1]
}
return len(watchDomain) == 0 && (item.acceptSuffix || len(dnsName) == 0)
}
func dnsLabelMatches(certLabel string, watchLabel string) bool {
// For fail-safe behavior, if a label was unparsable, it matches everything.
// Similarly, redacted labels match everything, since the label _might_ be
// for a name we're interested in.
return certLabel == "*" ||
certLabel == "?" ||
certLabel == certspotter.UnparsableDNSLabelPlaceholder ||
certspotter.MatchesWildcard(watchLabel, certLabel)
}
func (list WatchList) Matches(identifiers *certspotter.Identifiers) (bool, WatchItem) {
dnsNames := make([][]string, len(identifiers.DNSNames))
for i, dnsName := range identifiers.DNSNames {
dnsNames[i] = strings.Split(dnsName, ".")
}
for _, item := range list {
for _, dnsName := range dnsNames {
if item.matchesDNSName(dnsName) {
return true, item
}
}
}
return false, WatchItem{}
}
third_party/certspotter/precerts.go
Vendored
+154
View File
@@ -0,0 +1,154 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"bytes"
"encoding/asn1"
"errors"
"fmt"
)
func bitStringEqual(a, b *asn1.BitString) bool {
return a.BitLength == b.BitLength && bytes.Equal(a.Bytes, b.Bytes)
}
var (
oidExtensionAuthorityKeyId = []int{2, 5, 29, 35}
oidExtensionSCT = []int{1, 3, 6, 1, 4, 1, 11129, 2, 4, 2}
oidExtensionCTPoison = []int{1, 3, 6, 1, 4, 1, 11129, 2, 4, 3}
)
type PrecertInfo struct {
SameIssuer bool // The pre-certificate was issued from the same CA as the final certificate
Issuer []byte // The pre-certificate's issuer, if different from the final certificate
AKI []byte // The pre-certificate's AKI, if present and different from the final certificate
}
func ValidatePrecert(precertBytes []byte, tbsBytes []byte) (*PrecertInfo, error) {
precert, err := ParseCertificate(precertBytes)
if err != nil {
return nil, errors.New("failed to parse pre-certificate: " + err.Error())
}
precertTBS, err := precert.ParseTBSCertificate()
if err != nil {
return nil, errors.New("failed to parse pre-certificate TBS: " + err.Error())
}
tbs, err := ParseTBSCertificate(tbsBytes)
if err != nil {
return nil, errors.New("failed to parse TBS: " + err.Error())
}
// Everything must be equal except:
// issuer
// Authority Key Identifier extension (both must have it OR neither can have it)
// CT poison extension (precert must have it, TBS must not have it)
if precertTBS.Version != tbs.Version {
return nil, errors.New("version not equal")
}
if !bytes.Equal(precertTBS.SerialNumber.FullBytes, tbs.SerialNumber.FullBytes) {
return nil, errors.New("serial number not equal")
}
sameIssuer := bytes.Equal(precertTBS.Issuer.FullBytes, tbs.Issuer.FullBytes)
if !bytes.Equal(precertTBS.SignatureAlgorithm.FullBytes, tbs.SignatureAlgorithm.FullBytes) {
return nil, errors.New("SignatureAlgorithm not equal")
}
if !bytes.Equal(precertTBS.Validity.FullBytes, tbs.Validity.FullBytes) {
return nil, errors.New("Validity not equal")
}
if !bytes.Equal(precertTBS.Subject.FullBytes, tbs.Subject.FullBytes) {
return nil, errors.New("Subject not equal")
}
if !bytes.Equal(precertTBS.PublicKey.FullBytes, tbs.PublicKey.FullBytes) {
return nil, errors.New("PublicKey not equal")
}
if !bitStringEqual(&precertTBS.UniqueId, &tbs.UniqueId) {
return nil, errors.New("UniqueId not equal")
}
if !bitStringEqual(&precertTBS.SubjectUniqueId, &tbs.SubjectUniqueId) {
return nil, errors.New("SubjectUniqueId not equal")
}
precertHasPoison := false
tbsIndex := 0
var aki []byte
for precertIndex := range precertTBS.Extensions {
precertExt := &precertTBS.Extensions[precertIndex]
if precertExt.Id.Equal(oidExtensionCTPoison) {
if !precertExt.Critical {
return nil, errors.New("pre-cert poison extension is not critical")
}
/* CAs can't even get this right, and Google's logs don't check. Fortunately,
it's not that important.
if !bytes.Equal(precertExt.Value, []byte{0x05, 0x00}) {
return errors.New("pre-cert poison extension contains incorrect value")
}
*/
precertHasPoison = true
continue
}
if tbsIndex >= len(tbs.Extensions) {
return nil, errors.New("pre-cert contains extension not in TBS")
}
tbsExt := &tbs.Extensions[tbsIndex]
if !precertExt.Id.Equal(tbsExt.Id) {
return nil, fmt.Errorf("pre-cert and TBS contain different extensions (%v vs %v)", precertExt.Id, tbsExt.Id)
}
if precertExt.Critical != tbsExt.Critical {
return nil, fmt.Errorf("pre-cert and TBS %v extension differs in criticality", precertExt.Id)
}
if !sameIssuer && precertExt.Id.Equal(oidExtensionAuthorityKeyId) {
aki = precertExt.Value
} else {
if !bytes.Equal(precertExt.Value, tbsExt.Value) {
return nil, fmt.Errorf("pre-cert and TBS %v extension differs in value", precertExt.Id)
}
}
tbsIndex++
}
if tbsIndex < len(tbs.Extensions) {
return nil, errors.New("TBS contains extension not in pre-cert")
}
if !precertHasPoison {
return nil, errors.New("pre-cert does not have poison extension")
}
return &PrecertInfo{SameIssuer: sameIssuer, Issuer: precertTBS.Issuer.FullBytes, AKI: aki}, nil
}
func ReconstructPrecertTBS(tbs *TBSCertificate) (*TBSCertificate, error) {
precertTBS := TBSCertificate{
Version: tbs.Version,
SerialNumber: tbs.SerialNumber,
SignatureAlgorithm: tbs.SignatureAlgorithm,
Issuer: tbs.Issuer,
Validity: tbs.Validity,
Subject: tbs.Subject,
PublicKey: tbs.PublicKey,
UniqueId: tbs.UniqueId,
SubjectUniqueId: tbs.SubjectUniqueId,
Extensions: make([]Extension, 0, len(tbs.Extensions)),
}
for _, ext := range tbs.Extensions {
switch {
case ext.Id.Equal(oidExtensionSCT):
default:
precertTBS.Extensions = append(precertTBS.Extensions, ext)
}
}
var err error
precertTBS.Raw, err = asn1.Marshal(precertTBS)
return &precertTBS, err
}
third_party/certspotter/sequencer/sequencer.go
Vendored
+151
View File
@@ -0,0 +1,151 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package sequencer
import (
"context"
"slices"
"sync"
)
type seqWriter struct {
seqNbr uint64
ready chan<- struct{}
}
// Channel[T] is a multi-producer, single-consumer channel of items with monotonicaly-increasing sequence numbers.
// Items can be sent in any order, but they are always received in order of their sequence number.
// It is unsafe to call Next concurrently with itself, or to call Add/Reserve concurrently with another Add/Reserve
// call for the same sequence number. Otherwise, methods are safe to call concurrently.
type Channel[T any] struct {
mu sync.Mutex
next uint64
buf []*T
writers []seqWriter
readWaiting bool
readReady chan struct{}
}
func New[T any](initialSequenceNumber uint64, capacity uint64) *Channel[T] {
return &Channel[T]{
buf: make([]*T, capacity),
next: initialSequenceNumber,
readReady: make(chan struct{}, 1),
}
}
func (seq *Channel[T]) parkWriter(seqNbr uint64) <-chan struct{} {
ready := make(chan struct{})
seq.writers = append(seq.writers, seqWriter{seqNbr: seqNbr, ready: ready})
return ready
}
func (seq *Channel[T]) signalWriter(seqNbr uint64) {
for i := range seq.writers {
if seq.writers[i].seqNbr == seqNbr {
close(seq.writers[i].ready)
seq.writers = slices.Delete(seq.writers, i, i+1)
return
}
}
}
func (seq *Channel[T]) forgetWriter(seqNbr uint64) {
for i := range seq.writers {
if seq.writers[i].seqNbr == seqNbr {
seq.writers = slices.Delete(seq.writers, i, i+1)
return
}
}
}
func (seq *Channel[T]) Cap() uint64 {
return uint64(len(seq.buf))
}
func (seq *Channel[T]) index(seqNbr uint64) int {
return int(seqNbr % seq.Cap())
}
// Wait until the channel has capacity for an item with the given sequence number.
// After this function returns nil, calling Add with the same sequence number will not block.
func (seq *Channel[T]) Reserve(ctx context.Context, sequenceNumber uint64) error {
seq.mu.Lock()
if sequenceNumber >= seq.next+seq.Cap() {
ready := seq.parkWriter(sequenceNumber)
seq.mu.Unlock()
select {
case <-ctx.Done():
seq.mu.Lock()
seq.forgetWriter(sequenceNumber)
seq.mu.Unlock()
return ctx.Err()
case <-ready:
}
} else {
seq.mu.Unlock()
}
return nil
}
// Send an item with the given sequence number. Blocks if the channel does not have capacity for the item.
// It is undefined behavior to send a sequence number that has previously been sent.
func (seq *Channel[T]) Add(ctx context.Context, sequenceNumber uint64, item *T) error {
seq.mu.Lock()
if sequenceNumber >= seq.next+seq.Cap() {
ready := seq.parkWriter(sequenceNumber)
seq.mu.Unlock()
select {
case <-ctx.Done():
seq.mu.Lock()
seq.forgetWriter(sequenceNumber)
seq.mu.Unlock()
return ctx.Err()
case <-ready:
}
seq.mu.Lock()
}
seq.buf[seq.index(sequenceNumber)] = item
if sequenceNumber == seq.next && seq.readWaiting {
seq.readReady <- struct{}{}
}
seq.mu.Unlock()
return nil
}
// Return the item with the next sequence number, blocking if necessary.
// Not safe to call concurrently with other Next calls.
func (seq *Channel[T]) Next(ctx context.Context) (*T, error) {
seq.mu.Lock()
if seq.buf[seq.index(seq.next)] == nil {
seq.readWaiting = true
seq.mu.Unlock()
select {
case <-ctx.Done():
seq.mu.Lock()
select {
case <-seq.readReady:
default:
}
seq.readWaiting = false
seq.mu.Unlock()
return nil, ctx.Err()
case <-seq.readReady:
}
seq.mu.Lock()
seq.readWaiting = false
}
item := seq.buf[seq.index(seq.next)]
seq.buf[seq.index(seq.next)] = nil
seq.signalWriter(seq.next + seq.Cap())
seq.next++
seq.mu.Unlock()
return item, nil
}
third_party/certspotter/sequencer/sequencer_test.go
Vendored
+195
View File
@@ -0,0 +1,195 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package sequencer
import (
"context"
"fmt"
mathrand "math/rand/v2"
"testing"
"time"
)
func TestSequencerBasic(t *testing.T) {
ctx := context.Background()
seq := New[uint64](0, 100)
go func() {
for i := range uint64(10_000) {
err := seq.Add(ctx, i, &i)
if err != nil {
panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
}
}
}()
for i := range uint64(10_000) {
next, err := seq.Next(ctx)
if err != nil {
t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
}
if *next != i {
t.Fatalf("%d: got unexpected value %d", i, *next)
}
}
}
func TestSequencerNonZeroStart(t *testing.T) {
ctx := context.Background()
seq := New[uint64](10, 100)
go func() {
for i := range uint64(10_000) {
err := seq.Add(ctx, i+10, &i)
if err != nil {
panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
}
}
}()
for i := range uint64(10_000) {
next, err := seq.Next(ctx)
if err != nil {
t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
}
if *next != i {
t.Fatalf("%d: got unexpected value %d", i, *next)
}
}
}
func TestSequencerCapacity1(t *testing.T) {
ctx := context.Background()
seq := New[uint64](0, 1)
go func() {
for i := range uint64(10_000) {
err := seq.Add(ctx, i, &i)
if err != nil {
panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
}
}
}()
for i := range uint64(10_000) {
next, err := seq.Next(ctx)
if err != nil {
t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
}
if *next != i {
t.Fatalf("%d: got unexpected value %d", i, *next)
}
}
}
func TestSequencerTimeout(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
seq := New[uint64](0, 10_000)
go func() {
var i uint64
for {
newI := i
err := seq.Add(ctx, i, &newI)
if err != nil {
break
}
i++
}
}()
var i uint64
for {
next, err := seq.Next(ctx)
if err != nil {
break
}
if *next != i {
t.Fatalf("%d: got unexpected value %d", i, *next)
}
i++
}
}
func TestSequencerOutOfOrder(t *testing.T) {
ctx := context.Background()
seq := New[uint64](0, 100)
ch := make(chan uint64)
go func() {
for i := range uint64(10_000) {
ch <- i
}
}()
for range 4 {
go func() {
for i := range ch {
time.Sleep(mathrand.N(10 * time.Millisecond))
//t.Logf("seq.Add %d", i)
err := seq.Add(ctx, i, &i)
if err != nil {
panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
}
}
}()
}
for i := range uint64(10_000) {
next, err := seq.Next(ctx)
if err != nil {
t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
}
if *next != i {
t.Fatalf("%d: got unexpected value %d", i, *next)
}
//t.Logf("seq.Next %d", i)
}
}
func TestSequencerOutOfOrderReserve(t *testing.T) {
ctx := context.Background()
seq := New[uint64](0, 10)
ch := make(chan uint64)
go func() {
for i := range uint64(10_000) {
ch <- i
}
}()
ch2 := make(chan uint64)
for job := range 4 {
go func() {
for i := range ch {
time.Sleep(mathrand.N(11 * time.Duration(job+1) * time.Millisecond))
err := seq.Reserve(ctx, i)
if err != nil {
panic(fmt.Sprintf("%d: seq.Reserve returned unexpected error %v", i, err))
}
ch2 <- i
}
}()
}
for range 4 {
go func() {
for i := range ch2 {
time.Sleep(mathrand.N(7 * time.Millisecond))
t.Logf("seq.Add %d", i)
err := seq.Add(ctx, i, &i)
if err != nil {
panic(fmt.Sprintf("%d: seq.Add returned unexpected error %v", i, err))
}
}
}()
}
for i := range uint64(10_000) {
next, err := seq.Next(ctx)
if err != nil {
t.Fatalf("%d: seq.Next returned unexpected error %v", i, err)
}
if *next != i {
t.Fatalf("%d: got unexpected value %d", i, *next)
}
t.Logf("seq.Next %d", i)
}
}
third_party/certspotter/staticcheck.conf
Vendored
+1
View File
@@ -0,0 +1 @@
checks = ["inherit", "-ST1005", "-S1002"]
third_party/certspotter/tlstypes/signatures.go
Vendored
+134
View File
@@ -0,0 +1,134 @@
// Copyright (C) 2025 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package tlstypes
import (
"bytes"
"encoding/json"
"fmt"
"golang.org/x/crypto/cryptobyte"
)
type HashAlgorithm uint8
const (
SHA224 HashAlgorithm = 3
SHA256 HashAlgorithm = 4
SHA384 HashAlgorithm = 5
SHA512 HashAlgorithm = 6
)
type SignatureAlgorithm uint8
const (
RSA SignatureAlgorithm = 1
ECDSA SignatureAlgorithm = 3
)
type SignatureAndHashAlgorithm struct {
Hash HashAlgorithm
Signature SignatureAlgorithm
}
type DigitallySigned struct {
Algorithm SignatureAndHashAlgorithm
Signature []byte
}
func (v HashAlgorithm) Marshal(b *cryptobyte.Builder) error {
b.AddUint8(uint8(v))
return nil
}
func (v *HashAlgorithm) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint8((*uint8)(v))
}
func (v SignatureAlgorithm) Marshal(b *cryptobyte.Builder) error {
b.AddUint8(uint8(v))
return nil
}
func (v *SignatureAlgorithm) Unmarshal(s *cryptobyte.String) bool {
return s.ReadUint8((*uint8)(v))
}
func (v SignatureAndHashAlgorithm) Marshal(b *cryptobyte.Builder) error {
b.AddValue(v.Hash)
b.AddValue(v.Signature)
return nil
}
func (v *SignatureAndHashAlgorithm) Unmarshal(s *cryptobyte.String) bool {
return v.Hash.Unmarshal(s) && v.Signature.Unmarshal(s)
}
func (v DigitallySigned) Marshal(b *cryptobyte.Builder) error {
b.AddValue(v.Algorithm)
b.AddUint16LengthPrefixed(func(b *cryptobyte.Builder) { b.AddBytes(v.Signature) })
return nil
}
func (v *DigitallySigned) Unmarshal(s *cryptobyte.String) bool {
return v.Algorithm.Unmarshal(s) && s.ReadUint16LengthPrefixed((*cryptobyte.String)(&v.Signature))
}
func (v DigitallySigned) Bytes() []byte {
b := cryptobyte.NewBuilder(make([]byte, 0, 4+len(v.Signature)))
b.AddValue(v)
return b.BytesOrPanic()
}
func (v DigitallySigned) MarshalBinary() ([]byte, error) {
b := cryptobyte.NewBuilder(make([]byte, 0, 4+len(v.Signature)))
b.AddValue(v)
return b.Bytes()
}
func (v *DigitallySigned) UnmarshalBinary(data []byte) error {
str := cryptobyte.String(bytes.Clone(data))
if !v.Unmarshal(&str) {
return fmt.Errorf("DigitallySigned bytes are malformed")
}
if !str.Empty() {
return fmt.Errorf("trailing bytes after DigitallySigned")
}
return nil
}
func (v DigitallySigned) MarshalJSON() ([]byte, error) {
b := cryptobyte.NewBuilder(make([]byte, 0, 4+len(v.Signature)))
b.AddValue(v)
if bytes, err := b.Bytes(); err != nil {
return nil, err
} else {
return json.Marshal(bytes)
}
}
func (v *DigitallySigned) UnmarshalJSON(data []byte) error {
str := new(cryptobyte.String)
if err := json.Unmarshal(data, (*[]byte)(str)); err != nil {
return fmt.Errorf("unable to unmarshal DigitallySigned JSON: %w", err)
}
if !v.Unmarshal(str) {
return fmt.Errorf("DigitallySigned bytes are malformed")
}
if !str.Empty() {
return fmt.Errorf("trailing bytes after DigitallySigned")
}
return nil
}
func ParseDigitallySigned(bytes []byte) (*DigitallySigned, error) {
ds := new(DigitallySigned)
str := cryptobyte.String(bytes)
if !ds.Unmarshal(&str) {
return nil, fmt.Errorf("DigitallySigned bytes are malformed")
}
if !str.Empty() {
return nil, fmt.Errorf("trailing bytes after DigitallySigned")
}
return ds, nil
}
third_party/certspotter/x509.go
Vendored
+441
View File
@@ -0,0 +1,441 @@
// Copyright (C) 2016 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"bytes"
"crypto/x509/pkix"
"encoding/asn1"
"errors"
"fmt"
"math/big"
"net"
"time"
"unicode"
)
var (
oidExtensionSubjectAltName = asn1.ObjectIdentifier{2, 5, 29, 17}
oidExtensionBasicConstraints = asn1.ObjectIdentifier{2, 5, 29, 19}
oidCountry = asn1.ObjectIdentifier{2, 5, 4, 6}
oidOrganization = asn1.ObjectIdentifier{2, 5, 4, 10}
oidOrganizationalUnit = asn1.ObjectIdentifier{2, 5, 4, 11}
oidCommonName = asn1.ObjectIdentifier{2, 5, 4, 3}
oidSerialNumber = asn1.ObjectIdentifier{2, 5, 4, 5}
oidLocality = asn1.ObjectIdentifier{2, 5, 4, 7}
oidProvince = asn1.ObjectIdentifier{2, 5, 4, 8}
oidStreetAddress = asn1.ObjectIdentifier{2, 5, 4, 9}
oidPostalCode = asn1.ObjectIdentifier{2, 5, 4, 17}
)
type CertValidity struct {
NotBefore time.Time
NotAfter time.Time
}
type basicConstraints struct {
IsCA bool `asn1:"optional"`
MaxPathLen int `asn1:"optional,default:-1"`
}
type Extension struct {
Id asn1.ObjectIdentifier
Critical bool `asn1:"optional"`
Value []byte
}
const (
sanOtherName = 0
sanRfc822Name = 1
sanDNSName = 2
sanX400Address = 3
sanDirectoryName = 4
sanEdiPartyName = 5
sanURI = 6
sanIPAddress = 7
sanRegisteredID = 8
)
type SubjectAltName struct {
Type int
Value []byte
}
type RDNSequence []RelativeDistinguishedNameSET
type RelativeDistinguishedNameSET []AttributeTypeAndValue
type AttributeTypeAndValue struct {
Type asn1.ObjectIdentifier
Value asn1.RawValue
}
func ParseRDNSequence(rdnsBytes []byte) (RDNSequence, error) {
var rdns RDNSequence
if rest, err := asn1.Unmarshal(rdnsBytes, &rdns); err != nil {
return nil, errors.New("failed to parse RDNSequence: " + err.Error())
} else if len(rest) != 0 {
return nil, fmt.Errorf("trailing data after RDNSequence: %v", rest) // XXX: too strict?
}
return rdns, nil
}
func MarshalRDNSequence(rdns RDNSequence) ([]byte, error) {
return asn1.Marshal(rdns)
}
type TBSCertificate struct {
Raw asn1.RawContent
Version int `asn1:"optional,explicit,default:1,tag:0"`
SerialNumber asn1.RawValue
SignatureAlgorithm asn1.RawValue
Issuer asn1.RawValue
Validity asn1.RawValue
Subject asn1.RawValue
PublicKey asn1.RawValue
UniqueId asn1.BitString `asn1:"optional,tag:1"`
SubjectUniqueId asn1.BitString `asn1:"optional,tag:2"`
Extensions []Extension `asn1:"optional,explicit,tag:3"`
}
type Certificate struct {
Raw asn1.RawContent
TBSCertificate asn1.RawValue
SignatureAlgorithm asn1.RawValue
SignatureValue asn1.RawValue
}
func (rdns RDNSequence) ParseCNs() ([]string, error) {
var cns []string
// An RDN is a SET OF AttributeTypeAndValue, so the commonName attribute
// is not necessarily the first (or only) element. Examine every
// attribute so that a CN doesn't escape detection by being placed in a
// non-first position of a multi-valued RDN.
for _, rdn := range rdns {
for _, atv := range rdn {
if atv.Type.Equal(oidCommonName) {
cnString, err := decodeASN1String(&atv.Value)
if err != nil {
return nil, errors.New("Error decoding CN: " + err.Error())
}
cns = append(cns, cnString)
}
}
}
return cns, nil
}
func rdnLabel(oid asn1.ObjectIdentifier) string {
switch {
case oid.Equal(oidCountry):
return "C"
case oid.Equal(oidOrganization):
return "O"
case oid.Equal(oidOrganizationalUnit):
return "OU"
case oid.Equal(oidCommonName):
return "CN"
case oid.Equal(oidSerialNumber):
return "serialNumber"
case oid.Equal(oidLocality):
return "L"
case oid.Equal(oidProvince):
return "ST"
case oid.Equal(oidStreetAddress):
return "street"
case oid.Equal(oidPostalCode):
return "postalCode"
}
return oid.String()
}
func (rdns RDNSequence) String() string {
var buf bytes.Buffer
for _, rdn := range rdns {
// An RDN is a SET OF AttributeTypeAndValue; render every attribute,
// joining the attributes of a multi-valued RDN with '+' and separate
// RDNs with ", " (per RFC 4514).
for j, atv := range rdn {
if buf.Len() != 0 {
if j == 0 {
buf.WriteString(", ")
} else {
buf.WriteByte('+')
}
}
buf.WriteString(rdnLabel(atv.Type))
buf.WriteString("=")
valueString, err := decodeASN1String(&atv.Value)
if err == nil {
buf.WriteString(escapeDNString(valueString))
} else {
fmt.Fprintf(&buf, "%v", atv.Value.FullBytes)
}
}
}
return buf.String()
}
// escapeDNString escapes a distinguished name attribute value for safe display
// and for safe use as a hook script environment variable value. It escapes the
// backslash and comma characters (so the value is unambiguous within a
// comma-separated RDNSequence) and replaces non-printable characters with a
// backslash hex escape. In particular, this ensures the value never contains a
// NUL byte, which would otherwise be rejected by os/exec when passed as an
// environment variable and abort the notification.
func escapeDNString(value string) string {
var buf bytes.Buffer
for _, r := range value {
switch {
case r == '\\' || r == ',':
buf.WriteByte('\\')
buf.WriteRune(r)
case unicode.IsPrint(r):
buf.WriteRune(r)
case r <= 0xFF:
fmt.Fprintf(&buf, "\\x%02x", r)
case r <= 0xFFFF:
fmt.Fprintf(&buf, "\\u%04x", r)
default:
fmt.Fprintf(&buf, "\\U%08x", r)
}
}
return buf.String()
}
func (san SubjectAltName) String() string {
switch san.Type {
case sanDNSName:
return "DNS:" + escapeDNString(string(san.Value))
case sanIPAddress:
if len(san.Value) == 4 || len(san.Value) == 16 {
return "IP:" + net.IP(san.Value).String()
} else {
return fmt.Sprintf("IP:%v", san.Value)
}
default:
// TODO: support other types of SANs
return fmt.Sprintf("%d:%v", san.Type, san.Value)
}
}
func ParseTBSCertificate(tbsBytes []byte) (*TBSCertificate, error) {
var tbs TBSCertificate
if rest, err := asn1.Unmarshal(tbsBytes, &tbs); err != nil {
return nil, errors.New("failed to parse TBS: " + err.Error())
} else if len(rest) > 0 {
return nil, fmt.Errorf("trailing data after TBS: %v", rest) // XXX: too strict?
}
return &tbs, nil
}
func (tbs *TBSCertificate) ParseValidity() (*CertValidity, error) {
var rawValidity struct {
NotBefore asn1.RawValue
NotAfter asn1.RawValue
}
if rest, err := asn1.Unmarshal(tbs.Validity.FullBytes, &rawValidity); err != nil {
return nil, errors.New("failed to parse validity: " + err.Error())
} else if len(rest) > 0 {
return nil, fmt.Errorf("trailing data after validity: %v", rest)
}
var validity CertValidity
var err error
if validity.NotBefore, err = decodeASN1Time(&rawValidity.NotBefore); err != nil {
return nil, errors.New("failed to decode notBefore time: " + err.Error())
}
if validity.NotAfter, err = decodeASN1Time(&rawValidity.NotAfter); err != nil {
return nil, errors.New("failed to decode notAfter time: " + err.Error())
}
return &validity, nil
}
func (tbs *TBSCertificate) ParseBasicConstraints() (*bool, error) {
isCA := false
isNotCA := false
// Some certs in the wild have multiple BasicConstraints extensions (is there anything
// that CAs haven't screwed up???), so we process all of them and only choke if they
// are contradictory (which has not been observed...yet).
for _, ext := range tbs.GetExtension(oidExtensionBasicConstraints) {
var constraints basicConstraints
if rest, err := asn1.Unmarshal(ext.Value, &constraints); err != nil {
return nil, errors.New("failed to parse Basic Constraints: " + err.Error())
} else if len(rest) > 0 {
return nil, fmt.Errorf("trailing data after Basic Constraints: %v", rest)
}
if constraints.IsCA {
isCA = true
} else {
isNotCA = true
}
}
if !isCA && !isNotCA {
return nil, nil
} else if isCA && !isNotCA {
trueValue := true
return &trueValue, nil
} else if !isCA && isNotCA {
falseValue := false
return &falseValue, nil
} else {
return nil, fmt.Errorf("Certificate has more than one Basic Constraints extension and they are contradictory")
}
}
func (tbs *TBSCertificate) ParseSerialNumber() (*big.Int, error) {
serialNumber := big.NewInt(0)
if rest, err := asn1.Unmarshal(tbs.SerialNumber.FullBytes, &serialNumber); err != nil {
return nil, errors.New("failed to parse serial number: " + err.Error())
} else if len(rest) > 0 {
return nil, fmt.Errorf("trailing data after serial number: %v", rest)
}
return serialNumber, nil
}
func (tbs *TBSCertificate) GetRawPublicKey() []byte {
return tbs.PublicKey.FullBytes
}
func (tbs *TBSCertificate) GetRawSubject() []byte {
return tbs.Subject.FullBytes
}
func (tbs *TBSCertificate) GetRawIssuer() []byte {
return tbs.Issuer.FullBytes
}
func (tbs *TBSCertificate) ParseSubject() (RDNSequence, error) {
subject, err := ParseRDNSequence(tbs.GetRawSubject())
if err != nil {
return nil, errors.New("failed to parse certificate subject: " + err.Error())
}
return subject, nil
}
func (tbs *TBSCertificate) ParseIssuer() (RDNSequence, error) {
issuer, err := ParseRDNSequence(tbs.GetRawIssuer())
if err != nil {
return nil, errors.New("failed to parse certificate issuer: " + err.Error())
}
return issuer, nil
}
func (tbs *TBSCertificate) ParseSubjectCommonNames() ([]string, error) {
subject, err := tbs.ParseSubject()
if err != nil {
return nil, err
}
cns, err := subject.ParseCNs()
if err != nil {
return nil, errors.New("failed to process certificate subject: " + err.Error())
}
return cns, nil
}
func (tbs *TBSCertificate) ParseSubjectAltNames() ([]SubjectAltName, error) {
sans := []SubjectAltName{}
for _, sanExt := range tbs.GetExtension(oidExtensionSubjectAltName) {
var err error
sans, err = ParseSANExtension(sans, sanExt.Value)
if err != nil {
return nil, err
}
}
return sans, nil
}
func (tbs *TBSCertificate) GetExtension(id asn1.ObjectIdentifier) []Extension {
var exts []Extension
for _, ext := range tbs.Extensions {
if ext.Id.Equal(id) {
exts = append(exts, ext)
}
}
return exts
}
func ParseCertificate(certBytes []byte) (*Certificate, error) {
var cert Certificate
if rest, err := asn1.Unmarshal(certBytes, &cert); err != nil {
return nil, errors.New("failed to parse certificate: " + err.Error())
} else if len(rest) > 0 {
return nil, fmt.Errorf("trailing data after certificate: %v", rest) // XXX: too strict?
}
return &cert, nil
}
func (cert *Certificate) GetRawTBSCertificate() []byte {
return cert.TBSCertificate.FullBytes
}
func (cert *Certificate) ParseTBSCertificate() (*TBSCertificate, error) {
return ParseTBSCertificate(cert.GetRawTBSCertificate())
}
func (cert *Certificate) ParseSignatureAlgorithm() (*pkix.AlgorithmIdentifier, error) {
signatureAlgorithm := new(pkix.AlgorithmIdentifier)
if rest, err := asn1.Unmarshal(cert.SignatureAlgorithm.FullBytes, signatureAlgorithm); err != nil {
return nil, errors.New("failed to parse signature algorithm: " + err.Error())
} else if len(rest) > 0 {
return nil, fmt.Errorf("trailing data after signature algorithm: %v", rest)
}
return signatureAlgorithm, nil
}
func (cert *Certificate) ParseSignatureValue() ([]byte, error) {
var signatureValue asn1.BitString
if rest, err := asn1.Unmarshal(cert.SignatureValue.FullBytes, &signatureValue); err != nil {
return nil, errors.New("failed to parse signature value: " + err.Error())
} else if len(rest) > 0 {
return nil, fmt.Errorf("trailing data after signature value: %v", rest)
}
return signatureValue.RightAlign(), nil
}
func ParseSANExtension(sans []SubjectAltName, value []byte) ([]SubjectAltName, error) {
var seq asn1.RawValue
if rest, err := asn1.Unmarshal(value, &seq); err != nil {
return nil, errors.New("failed to parse subjectAltName extension: " + err.Error())
} else if len(rest) != 0 {
// Don't complain if the SAN is followed by exactly one zero byte,
// which is a common error.
if !(len(rest) == 1 && rest[0] == 0) {
return nil, fmt.Errorf("trailing data in subjectAltName extension: %v", rest) // XXX: too strict?
}
}
if !seq.IsCompound || seq.Tag != 16 || seq.Class != 0 {
return nil, errors.New("failed to parse subjectAltName extension: bad SAN sequence") // XXX: too strict?
}
rest := seq.Bytes
for len(rest) > 0 {
var val asn1.RawValue
var err error
rest, err = asn1.Unmarshal(rest, &val)
if err != nil {
return nil, errors.New("failed to parse subjectAltName extension item: " + err.Error())
}
sans = append(sans, SubjectAltName{Type: val.Tag, Value: val.Bytes})
}
return sans, nil
}
third_party/certspotter/x509_test.go
Vendored
+135
View File
@@ -0,0 +1,135 @@
// Copyright (C) 2026 Opsmate, Inc.
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License, v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
//
// This software is distributed WITHOUT A WARRANTY OF ANY KIND.
// See the Mozilla Public License for details.
package certspotter
import (
"encoding/asn1"
"os/exec"
"slices"
"strings"
"testing"
)
// printableString builds an asn1.RawValue holding a PrintableString, as
// decodeASN1String expects to see it.
func printableString(s string) asn1.RawValue {
return asn1.RawValue{Class: 0, Tag: 19, IsCompound: false, Bytes: []byte(s)}
}
func atv(oid asn1.ObjectIdentifier, value string) AttributeTypeAndValue {
return AttributeTypeAndValue{Type: oid, Value: printableString(value)}
}
// A commonName that is not the first attribute of a multi-valued RDN must still
// be extracted, otherwise a watched name could escape detection.
func TestParseCNsMultiValuedRDN(t *testing.T) {
rdns := RDNSequence{
{
atv(oidOrganization, "Example Org"),
atv(oidCommonName, "watched.example.com"),
},
}
cns, err := rdns.ParseCNs()
if err != nil {
t.Fatal(err)
}
if want := []string{"watched.example.com"}; !slices.Equal(cns, want) {
t.Errorf("ParseCNs() = %q, want %q", cns, want)
}
}
func TestParseCNsMultipleCNs(t *testing.T) {
rdns := RDNSequence{
{atv(oidCommonName, "a.example.com")},
{
atv(oidOrganizationalUnit, "Dept"),
atv(oidCommonName, "b.example.com"),
},
}
cns, err := rdns.ParseCNs()
if err != nil {
t.Fatal(err)
}
if want := []string{"a.example.com", "b.example.com"}; !slices.Equal(cns, want) {
t.Errorf("ParseCNs() = %q, want %q", cns, want)
}
}
func TestRDNSequenceStringEscaping(t *testing.T) {
tests := []struct {
name string
rdns RDNSequence
want string
}{
{
name: "nul byte",
rdns: RDNSequence{{atv(oidCommonName, "ev\x00il")}},
want: `CN=ev\x00il`,
},
{
name: "backslash and comma",
rdns: RDNSequence{{atv(oidOrganization, `a\b,c`)}},
want: `O=a\\b\,c`,
},
{
name: "control characters",
rdns: RDNSequence{{atv(oidCommonName, "a\tb\nc")}},
want: `CN=a\x09b\x0ac`,
},
{
name: "multi-valued rdn joined with plus",
rdns: RDNSequence{{
atv(oidOrganization, "Org"),
atv(oidCommonName, "host.example.com"),
}},
want: "O=Org+CN=host.example.com",
},
{
name: "separate rdns joined with comma",
rdns: RDNSequence{
{atv(oidCommonName, "host.example.com")},
{atv(oidOrganization, "Org")},
},
want: "CN=host.example.com, O=Org",
},
{
name: "printable unicode passes through",
rdns: RDNSequence{{AttributeTypeAndValue{
Type: oidOrganization,
Value: asn1.RawValue{Class: 0, Tag: 12, Bytes: []byte("café")}, // UTF8String
}}},
want: "O=café",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := tt.rdns.String(); got != tt.want {
t.Errorf("String() = %q, want %q", got, tt.want)
}
})
}
}
// Regression test for the denial of service in which a NUL byte in a certificate
// Subject/Issuer DN, placed into a hook script's environment, was rejected by
// os/exec and aborted notification (crashing the whole monitor). The escaped DN
// must be usable as an environment variable value.
func TestRDNSequenceStringIsExecSafe(t *testing.T) {
rdns := RDNSequence{{atv(oidCommonName, "ev\x00il\x01\x02")}}
dn := rdns.String()
if strings.ContainsRune(dn, 0) {
t.Fatalf("DN string still contains a NUL byte: %q", dn)
}
cmd := exec.Command("/bin/sh", "-c", "exit 0")
cmd.Env = []string{"SUBJECT_DN=" + dn}
if err := cmd.Run(); err != nil {
t.Fatalf("exec with SUBJECT_DN=%q failed: %v", dn, err)
}
}