In A Nutshell
About Android OS
Some parts of Android will be familiar, such as the Linux Kernel, OpenGL, and the SQL database. Others may be completely foreign, such as Android's idea of the application life cycle. You'll need a good understanding of these key concepts in order to write well-behaved Android applications. Let's start off by taking a look at the overall system architecture--the key layers and components that make up the Android stack. Read More
Linux From Scratch
There are always many ways to accomplish a single task. The same can be said about Linux distributions. A great many have existed over the years. Some still exist, some have morphed into something else, yet others have been relegated to our memories. They all do things differently to suit the needs of their target audience. Because so many different ways to accomplish the same end goal exist, I began to realize I no longer had to be limited by any one implementation. Prior to discovering Linux, we simply put up with issues in other Operating Systems as you had no choice. It was what it was, whether you liked it or not. With Linux, the concept of choice began to emerge. If you didn't like something, you were free, even encouraged, to change it. Linux From Scratch
Creating a Raspberry Pi-Based Beowulf Cluster
Raspberry Pis have really taken the embedded Linux community by storm. For those unfamiliar, however, a Raspberry Pi (RPi) is a small (credit card sized), inexpensive single-board computer that is capable of running Linux and other lightweight operating systems which run on ARM processors. For those who may not have heard of a Beowulf cluster before, a Beowulf cluster is simply a collection of identical, (typically) commodity computer hardware based systems, networked together and running some kind of parallel processing software that allows each node in the cluster to share data and computation. Joshua Kiepert, Boise State University
Let's Encrypt News
Takeaways from Tailscale’s Adoption of ARI
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Wed, 01 May 2024 00:00:00 +0000
An Engineer’s Guide to Integrating ARI into Existing ACME Clients
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Thu, 25 Apr 2024 00:00:00 +0000
Deploying Let's Encrypt's New Issuance Chains
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Fri, 12 Apr 2024 00:00:00 +0000
New Intermediate Certificates
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.
First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.
Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.
The New Certificates
We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.
We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.
You can see details of all of the certificates on our updated hierarchy page.
Rotating Issuance
Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.
Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.
We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.
These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.
Providing Smaller Chains
When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.
ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.
This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.
Other Minor Changes
We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:
-
We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.
-
We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.
Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.
Deployment
We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.
Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.
We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.
We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Tue, 19 Mar 2024 00:00:00 +0000
Introducing Sunlight, a CT implementation built for scalability, ease of operation, and reduced cost
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.
First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.
Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.
The New Certificates
We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.
We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.
You can see details of all of the certificates on our updated hierarchy page.
Rotating Issuance
Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.
Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.
We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.
These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.
Providing Smaller Chains
When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.
ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.
This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.
Other Minor Changes
We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:
-
We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.
-
We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.
Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.
Deployment
We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.
Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.
We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.
We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.
CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.
In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.
Bottlenecks from the Database
Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.
The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.
Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.
Serving Tiles
Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.
Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.
Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.
Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!
The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.
Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.
Running More Logs
The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.
The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective of a CT log.
The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.
No More Merge Delay
One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.
Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.
It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.
A Sunny Future
Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.
We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”
If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).
We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.
Thu, 14 Mar 2024 00:00:00 +0000
A Year-End Letter from our Vice President
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.
First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.
Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.
The New Certificates
We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.
We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.
You can see details of all of the certificates on our updated hierarchy page.
Rotating Issuance
Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.
Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.
We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.
These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.
Providing Smaller Chains
When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.
ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.
This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.
Other Minor Changes
We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:
-
We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.
-
We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.
Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.
Deployment
We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.
Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.
We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.
We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.
CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.
In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.
Bottlenecks from the Database
Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.
The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.
Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.
Serving Tiles
Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.
Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.
Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.
Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!
The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.
Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.
Running More Logs
The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.
The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective of a CT log.
The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.
No More Merge Delay
One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.
Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.
It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.
A Sunny Future
Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.
We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”
If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).
We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.
This letter was originally published in our 2023 Annual Report.
We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.
One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.
Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.
Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with
Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.
We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.
Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.
Support Our Work
ISRG is a 501(c)(3) nonprofit organization that is 100% supported through the generosity of those who share our vision for ubiquitous, open Internet security. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Thu, 28 Dec 2023 00:00:00 +0000
Our role in supporting the nonprofit ecosystem
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.
First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.
Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.
The New Certificates
We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.
We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.
You can see details of all of the certificates on our updated hierarchy page.
Rotating Issuance
Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.
Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.
We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.
These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.
Providing Smaller Chains
When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.
ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.
This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.
Other Minor Changes
We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:
-
We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.
-
We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.
Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.
Deployment
We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.
Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.
We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.
We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.
CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.
In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.
Bottlenecks from the Database
Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.
The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.
Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.
Serving Tiles
Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.
Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.
Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.
Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!
The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.
Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.
Running More Logs
The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.
The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective of a CT log.
The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.
No More Merge Delay
One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.
Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.
It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.
A Sunny Future
Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.
We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”
If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).
We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.
This letter was originally published in our 2023 Annual Report.
We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.
One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.
Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.
Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with
Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.
We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.
Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.
Support Our Work
ISRG is a 501(c)(3) nonprofit organization that is 100% supported through the generosity of those who share our vision for ubiquitous, open Internet security. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
For more than ten years, we at the nonprofit Internet Security Research Group (ISRG) have been focused on our mission of building a more secure and privacy-respecting Internet for everyone, everywhere. As we touch on in our 2023 Annual Report, we now serve more than 360 million domains with free TLS certificates.
Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.
Serving .org at Internet scale
Let’s Encrypt serves 57% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations are varied, all of them rely on the Internet in some capacity.
When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits' content could be changed without their knowledge or their visitors' private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.
In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.
Zooming in on three nonprofits we serve
The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.
Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.
The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”
Serving philanthropic foundations
In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.
We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, Yield Giving, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.
One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”
Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.
Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Wed, 13 Dec 2023 00:00:00 +0000
Increase your security governance with CAA
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.
First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.
Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.
The New Certificates
We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.
We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.
You can see details of all of the certificates on our updated hierarchy page.
Rotating Issuance
Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.
Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.
We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.
These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.
Providing Smaller Chains
When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.
ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.
This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.
Other Minor Changes
We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:
-
We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.
-
We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.
Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.
Deployment
We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.
Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.
We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.
We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.
CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.
In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.
Bottlenecks from the Database
Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.
The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.
Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.
Serving Tiles
Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.
Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.
Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.
Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!
The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.
Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.
Running More Logs
The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.
The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective of a CT log.
The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.
No More Merge Delay
One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.
Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.
It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.
A Sunny Future
Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.
We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”
If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).
We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.
This letter was originally published in our 2023 Annual Report.
We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.
One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.
Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.
Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with
Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.
We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.
Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.
Support Our Work
ISRG is a 501(c)(3) nonprofit organization that is 100% supported through the generosity of those who share our vision for ubiquitous, open Internet security. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
For more than ten years, we at the nonprofit Internet Security Research Group (ISRG) have been focused on our mission of building a more secure and privacy-respecting Internet for everyone, everywhere. As we touch on in our 2023 Annual Report, we now serve more than 360 million domains with free TLS certificates.
Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.
Serving .org at Internet scale
Let’s Encrypt serves 57% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations are varied, all of them rely on the Internet in some capacity.
When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits' content could be changed without their knowledge or their visitors' private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.
In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.
Zooming in on three nonprofits we serve
The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.
Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.
The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”
Serving philanthropic foundations
In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.
We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, Yield Giving, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.
One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”
Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.
Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
According to Cloudflare’s Merkle Town, 257,036 certificates are issued every hour. We at Let’s Encrypt are issuing close to 70% of those certs. Being a Certificate Authority that operates as a nonprofit for the public’s benefit means we are constantly considering how we can improve our Subscribers' experience and security. One simple innovation to do just that is by using CAA (Certificate Authority Authorization), and two CAA extensions for Account and Method Binding.
What is CAA?
CAA is a type of DNS record that allows site owners to specify which Certificate Authorities are allowed to issue certificates containing their domain names. Using CAA is a proactive way to ensure that your domain(s) and subdomain(s) are under your control—you’re able to add a layer of security to your DNS governance. (By contrast, Certificate Transparency (CT) logs are a reactive way to monitor your DNS governance—by publicly publishing certificates issued to domains, Subscribers can verify that their domain(s) are using the intended CA(s).)
We think CAA is important for every Subscriber, but it’s all the more important if you’re handling TLS at scale. This is particularly true if a team or multiple teams have access to your integration.
Account and Method Binding is another layer of CAA that can improve your security even further. Method binding allows Subscribers to limit the sets of domain control validation methods—DNS-01, HTTP-01, or TLS-ALPN-01— which can be used to demonstrate control over their domain. Account binding allows a Subscriber to limit issuance to a specific ACME account. For further technical details, review our community post or take a look at RFC 8657.
CAA Adoption
Famedly, a German healthcare company, set up CAA as part of updating their overall ACME setup. In hearing more about why they chose to turn on CAA now, the answer was simple: because it was easy to do and low hanging fruit to enhance their security.
“The biggest benefit of using CAA along with account and method binding is closing the DV loophole,” said Jan Christian Grünhage, Famedly’s Head of Infrastructure. “By using DNSSEC with DNS-01 challenges, we’ve got cryptographic signatures all the way through the stack.”
The team at Famedly set up CAA over the course of a few days. “The larger project was to transition our issuance to a single ACME account ID, so adopting CAA as part of that work only added marginal effort,” remarked Jan. “The added security benefit was absolutely worth the effort.”
Getting started with CAA
If you manage TLS at scale, consider adopting CAA and Account and Method Binding. To get started, review our documentation, RFC 8659 and RFC 8657, and check out the community forum for more from Subscribers who’ve set up or are using CAA.
As with anything with DNS, there are some potential hiccups to avoid. The most important to highlight is that CAA will always respect the CAA record closest to the domain name it is issuing a certificate for. For more on this, check out this section of the CAA documentation. You’ll also want to ensure that your DNS provider supports setting CAA records.
Thanks to Famedly
We’re grateful for Famedly taking the time to share with us more about their experience in setting up CAA. What’s more, Famedly financially supported ISRG this year as part of our tenth anniversary campaign.
As a project of the Internet Security Research Group (ISRG), 100% of the funding for Let’s Encrypt comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org. If you or your organization can support us with a donation of any size, we ask that you consider a contribution.
Thu, 07 Sep 2023 00:00:00 +0000
Shortening the Let's Encrypt Chain of Trust
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.
First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.
Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.
The New Certificates
We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.
We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.
You can see details of all of the certificates on our updated hierarchy page.
Rotating Issuance
Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.
Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.
We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.
These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.
Providing Smaller Chains
When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.
ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.
This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.
Other Minor Changes
We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:
-
We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.
-
We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.
Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.
Deployment
We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.
Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.
We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.
We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.
CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.
In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.
Bottlenecks from the Database
Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.
The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.
Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.
Serving Tiles
Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.
Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.
Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.
Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!
The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.
Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.
Running More Logs
The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.
The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective of a CT log.
The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.
No More Merge Delay
One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.
Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.
It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.
A Sunny Future
Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.
We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”
If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).
We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.
This letter was originally published in our 2023 Annual Report.
We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.
One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.
Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.
Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with
Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.
We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.
Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.
Support Our Work
ISRG is a 501(c)(3) nonprofit organization that is 100% supported through the generosity of those who share our vision for ubiquitous, open Internet security. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
For more than ten years, we at the nonprofit Internet Security Research Group (ISRG) have been focused on our mission of building a more secure and privacy-respecting Internet for everyone, everywhere. As we touch on in our 2023 Annual Report, we now serve more than 360 million domains with free TLS certificates.
Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.
Serving .org at Internet scale
Let’s Encrypt serves 57% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations are varied, all of them rely on the Internet in some capacity.
When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits' content could be changed without their knowledge or their visitors' private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.
In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.
Zooming in on three nonprofits we serve
The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.
Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.
The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”
Serving philanthropic foundations
In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.
We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, Yield Giving, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.
One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”
Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.
Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
According to Cloudflare’s Merkle Town, 257,036 certificates are issued every hour. We at Let’s Encrypt are issuing close to 70% of those certs. Being a Certificate Authority that operates as a nonprofit for the public’s benefit means we are constantly considering how we can improve our Subscribers' experience and security. One simple innovation to do just that is by using CAA (Certificate Authority Authorization), and two CAA extensions for Account and Method Binding.
What is CAA?
CAA is a type of DNS record that allows site owners to specify which Certificate Authorities are allowed to issue certificates containing their domain names. Using CAA is a proactive way to ensure that your domain(s) and subdomain(s) are under your control—you’re able to add a layer of security to your DNS governance. (By contrast, Certificate Transparency (CT) logs are a reactive way to monitor your DNS governance—by publicly publishing certificates issued to domains, Subscribers can verify that their domain(s) are using the intended CA(s).)
We think CAA is important for every Subscriber, but it’s all the more important if you’re handling TLS at scale. This is particularly true if a team or multiple teams have access to your integration.
Account and Method Binding is another layer of CAA that can improve your security even further. Method binding allows Subscribers to limit the sets of domain control validation methods—DNS-01, HTTP-01, or TLS-ALPN-01— which can be used to demonstrate control over their domain. Account binding allows a Subscriber to limit issuance to a specific ACME account. For further technical details, review our community post or take a look at RFC 8657.
CAA Adoption
Famedly, a German healthcare company, set up CAA as part of updating their overall ACME setup. In hearing more about why they chose to turn on CAA now, the answer was simple: because it was easy to do and low hanging fruit to enhance their security.
“The biggest benefit of using CAA along with account and method binding is closing the DV loophole,” said Jan Christian Grünhage, Famedly’s Head of Infrastructure. “By using DNSSEC with DNS-01 challenges, we’ve got cryptographic signatures all the way through the stack.”
The team at Famedly set up CAA over the course of a few days. “The larger project was to transition our issuance to a single ACME account ID, so adopting CAA as part of that work only added marginal effort,” remarked Jan. “The added security benefit was absolutely worth the effort.”
Getting started with CAA
If you manage TLS at scale, consider adopting CAA and Account and Method Binding. To get started, review our documentation, RFC 8659 and RFC 8657, and check out the community forum for more from Subscribers who’ve set up or are using CAA.
As with anything with DNS, there are some potential hiccups to avoid. The most important to highlight is that CAA will always respect the CAA record closest to the domain name it is issuing a certificate for. For more on this, check out this section of the CAA documentation. You’ll also want to ensure that your DNS provider supports setting CAA records.
Thanks to Famedly
We’re grateful for Famedly taking the time to share with us more about their experience in setting up CAA. What’s more, Famedly financially supported ISRG this year as part of our tenth anniversary campaign.
As a project of the Internet Security Research Group (ISRG), 100% of the funding for Let’s Encrypt comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org. If you or your organization can support us with a donation of any size, we ask that you consider a contribution.
When Let’s Encrypt first launched, we needed to ensure that our certificates were widely trusted. To that end, we arranged to have our intermediate certificates cross-signed by IdenTrust’s DST Root CA X3. This meant that all certificates issued by those intermediates would be trusted, even while our own ISRG Root X1 wasn’t yet. During subsequent years, our Root X1 became widely trusted on its own.
Come late 2021, our cross-signed intermediates and DST Root CA X3 itself were expiring. And while all up-to-date browsers at that time trusted our root, over a third of Android devices were still running old versions of the OS which would suddenly stop trusting websites using our certificates. That breakage would have been too widespread, so we arranged for a new cross-sign – this time directly onto our root rather than our intermediates – which would outlive DST Root CA X3 itself. This stopgap allowed those old Android devices to continue trusting our certificates for three more years.
On September 30th, 2024, that cross-sign too will expire.
In the last three years, the percentage of Android devices which trust our ISRG Root X1 has risen from 66% to 93.9%. That percentage will increase further over the next year, especially as Android releases version 14, which has the ability to update its trust store without a full OS update. In addition, dropping the cross-sign will reduce the number of certificate bytes sent in a TLS handshake by over 40%. Finally, it will significantly reduce our operating costs, allowing us to focus our funding on continuing to improve your privacy and security.
For these reasons, we will not be getting a new cross-sign to extend compatibility any further.
The transition will roll out as follows:
-
On Thursday, Feb 8th, 2024, we stopped providing the cross-sign by default in requests made to our /acme/certificate API endpoint. For most Subscribers, this means that your ACME client will configure a chain which terminates at ISRG Root X1, and your webserver will begin providing this shorter chain in all TLS handshakes. The longer chain, terminating at the soon-to-expire cross-sign, will still be available as an alternate chain which you can configure your client to request.
-
On Thursday, June 6th, 2024, we will stop providing the longer cross-signed chain entirely. This is just over 90 days (the lifetime of one certificate) before the cross-sign expires, and we need to make sure subscribers have had at least one full issuance cycle to migrate off of the cross-signed chain.
-
On Monday, September 30th, 2024, the cross-signed certificate will expire. This should be a non-event for most people, as any client breakages should have occurred over the preceding six months.
If you use Android 7.0 or earlier, you may need to take action to ensure you can still access websites secured by Let’s Encrypt certificates. We recommend installing and using Firefox Mobile, which uses its own trust store instead of the Android OS trust store, and therefore trusts ISRG Root X1.
If you are a site operator, you should keep an eye on your website usage statistics and active user-agent strings during Q2 and Q3 of 2024. If you see a sudden drop in visits from Android, it is likely because you have a significant population of users on Android 7.0 or earlier. We encourage you to provide the same advice to them as we provided above.
If you are an ACME client author, please make sure that your client correctly downloads and installs the certificate chain provided by our API during every certificate issuance, including renewals. Failure modes we have seen in the past include a) never downloading the chain at all and only serving the end-entity certificate; b) never downloading the chain and instead serving a hard-coded chain; and c) only downloading the chain at first issuance and not re-downloading during renewals. Please ensure that your client does not fall into any of these buckets.
We appreciate your understanding and support, both now and in the years to come as we provide safe and secure communication to everyone who uses the web. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We’d like to thank IdenTrust for their years of partnership. They played an important role in helping Let’s Encrypt get to where we are today and their willingness to arrange a stopgap cross sign in 2021 demonstrated a true commitment to creating a secure Web.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Mon, 10 Jul 2023 00:00:00 +0000
ISRG’s 10th Anniversary
Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.
In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.
In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.
Tailscale noted that ARI was especially useful to add before certificates' validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.
One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.
Tailscale’s roadmap for getting ARI in production:
-
Updated their fork of golang.org/x/crypto to support ARI
-
Updated the renewal code in the Tailscale client
-
Tested it locally by requesting certificates for a dev domain
-
Tested renewal by stubbing out ARI response with hardcoded data
-
Tested fallback by blocking ARI requests
-
Shipped it!
The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!
We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.
We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.
Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.
To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!
Integrating ARI Into an Existing ACME Client
In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.
Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.
Step 1: Detecting support for ARI
While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.
In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.
In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:
type Directory struct {
NewNonceURL string `json:"newNonce"`
NewAccountURL string `json:"newAccount"`
NewOrderURL string `json:"newOrder"`
NewAuthzURL string `json:"newAuthz"`
RevokeCertURL string `json:"revokeCert"`
KeyChangeURL string `json:"keyChange"`
Meta Meta `json:"meta"`
RenewalInfo string `json:"renewalInfo"`
}
As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:
func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
if c.core.GetDirectory().RenewalInfo == "" {
return nil, ErrNoARI
}
}
Step 2: Determining where ARI fits into the renewal lifecycle of your client
The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.
In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.
Step 3: Constructing the ARI CertID
The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.
Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.
In the case of Lego, we implemented the above logic in the following function:
// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.
func MakeARICertID(leaf *x509.Certificate) (string, error) {
if leaf == nil {
return "", errors.New("leaf certificate is nil")
}
// Marshal the Serial Number into DER.
der, err := asn1.Marshal(leaf.SerialNumber)
if err != nil {
return "", err
}
// Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
// length, and value).
if len(der) < 3 {
return "", errors.New("invalid DER encoding of serial number")
}
// Extract only the integer bytes from the DER encoded Serial Number
// Skipping the first 2 bytes (tag and length). The result is base64url
// encoded without padding.
serial := base64.RawURLEncoding.EncodeToString(der[2:])
// Convert the Authority Key Identifier to base64url encoding without
// padding.
aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)
// Construct the final identifier by concatenating AKI and Serial Number.
return fmt.Sprintf("%s.%s", aki, serial), nil
}
Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.
Step 4: Requesting a suggested renewal window
With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.
GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE
The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ‘end’ timestamps indicating the recommended renewal period, and optionally, an ‘explanationURL’ providing additional context about the renewal suggestion.
{
"suggestedWindow": {
"start": "2021-01-03T00:00:00Z",
"end": "2021-01-07T00:00:00Z"
},
"explanationURL": "https://example.com/docs/ari"
}
The ‘explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ‘explanationURL’ might link to a page explaining the incident.
Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.
Step 5: Selecting a specific renewal time
draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.
-
Select a uniform random time within the suggested window.
-
If the selected time is in the past, attempt renewal immediately.
-
Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
-
Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
-
Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”
For Lego, we implemented the above logic in the following function:
func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {
// Explicitly convert all times to UTC.
now = now.UTC()
start := r.SuggestedWindow.Start.UTC()
end := r.SuggestedWindow.End.UTC()
// Select a uniform random time within the suggested window.
window := end.Sub(start)
randomDuration := time.Duration(rand.Int63n(int64(window)))
rt := start.Add(randomDuration)
// If the selected time is in the past, attempt renewal immediately.
if rt.Before(now) {
return &now
}
// Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
willingToSleepUntil := now.Add(willingToSleep)
if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
return &rt
}
// TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.
// Otherwise, sleep until the next normal wake time.
return nil
}
Step 6: Indicating which certificate is replaced by this new order
To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:
{
"protected": base64url({
"alg": "ES256",
"kid": "https://example.com/acme/acct/evOfKhNU60wg",
"nonce": "5XJ1L3lEkMG7tR6pA00clA",
"url": "https://example.com/acme/new-order"
}),
"payload": base64url({
"identifiers": [
{ "type": "dns", "value": "example.com" }
],
"replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
}),
"signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}
Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:
// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3
type Order struct {
...
// replaces (optional, string):
// a string uniquely identifying a previously-issued
// certificate which this order is intended to replace.
// - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
Replaces string `json:"replaces,omitempty"`
}
...
// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
...
if o.core.GetDirectory().RenewalInfo != "" {
orderReq.Replaces = opts.ReplacesCertID
}
}
When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!
Moving Forward
The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.
As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.
We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.
Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.
Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).
The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.
New Intermediate Certificates
Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.
All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.
The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.
It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.
A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.
Removing DST Root CA X3 Cross-sign
The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.
ECDSA Intermediates as Default for ECDSA Certificates
Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.
Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.
We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.
Risks of Pinning or Hard-Coding Intermediates
We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).
Questions?
We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.
First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.
Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.
The New Certificates
We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.
We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.
You can see details of all of the certificates on our updated hierarchy page.
Rotating Issuance
Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.
Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.
We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.
These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.
Providing Smaller Chains
When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.
ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.
This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.
Other Minor Changes
We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:
-
We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.
-
We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.
Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.
Deployment
We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.
Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.
We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.
We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.
CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.
In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.
Bottlenecks from the Database
Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.
The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.
Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.
Serving Tiles
Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.
Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.
Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.
Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!
The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.
Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.
Running More Logs
The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.
The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective of a CT log.
The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.
No More Merge Delay
One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.
Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.
It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.
A Sunny Future
Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.
We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”
If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).
We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.
This letter was originally published in our 2023 Annual Report.
We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.
One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.
Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.
Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with
Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.
We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.
Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.
Support Our Work
ISRG is a 501(c)(3) nonprofit organization that is 100% supported through the generosity of those who share our vision for ubiquitous, open Internet security. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
For more than ten years, we at the nonprofit Internet Security Research Group (ISRG) have been focused on our mission of building a more secure and privacy-respecting Internet for everyone, everywhere. As we touch on in our 2023 Annual Report, we now serve more than 360 million domains with free TLS certificates.
Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.
Serving .org at Internet scale
Let’s Encrypt serves 57% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations are varied, all of them rely on the Internet in some capacity.
When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits' content could be changed without their knowledge or their visitors' private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.
In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.
Zooming in on three nonprofits we serve
The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.
Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.
The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”
Serving philanthropic foundations
In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.
We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, Yield Giving, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.
One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”
Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.
Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.
According to Cloudflare’s Merkle Town, 257,036 certificates are issued every hour. We at Let’s Encrypt are issuing close to 70% of those certs. Being a Certificate Authority that operates as a nonprofit for the public’s benefit means we are constantly considering how we can improve our Subscribers' experience and security. One simple innovation to do just that is by using CAA (Certificate Authority Authorization), and two CAA extensions for Account and Method Binding.
What is CAA?
CAA is a type of DNS record that allows site owners to specify which Certificate Authorities are allowed to issue certificates containing their domain names. Using CAA is a proactive way to ensure that your domain(s) and subdomain(s) are under your control—you’re able to add a layer of security to your DNS governance. (By contrast, Certificate Transparency (CT) logs are a reactive way to monitor your DNS governance—by publicly publishing certificates issued to domains, Subscribers can verify that their domain(s) are using the intended CA(s).)
We think CAA is important for every Subscriber, but it’s all the more important if you’re handling TLS at scale. This is particularly true if a team or multiple teams have access to your integration.
Account and Method Binding is another layer of CAA that can improve your security even further. Method binding allows Subscribers to limit the sets of domain control validation methods—DNS-01, HTTP-01, or TLS-ALPN-01— which can be used to demonstrate control over their domain. Account binding allows a Subscriber to limit issuance to a specific ACME account. For further technical details, review our community post or take a look at RFC 8657.
CAA Adoption
Famedly, a German healthcare company, set up CAA as part of updating their overall ACME setup. In hearing more about why they chose to turn on CAA now, the answer was simple: because it was easy to do and low hanging fruit to enhance their security.
“The biggest benefit of using CAA along with account and method binding is closing the DV loophole,” said Jan Christian Grünhage, Famedly’s Head of Infrastructure. “By using DNSSEC with DNS-01 challenges, we’ve got cryptographic signatures all the way through the stack.”
The team at Famedly set up CAA over the course of a few days. “The larger project was to transition our issuance to a single ACME account ID, so adopting CAA as part of that work only added marginal effort,” remarked Jan. “The added security benefit was absolutely worth the effort.”
Getting started with CAA
If you manage TLS at scale, consider adopting CAA and Account and Method Binding. To get started, review our documentation, RFC 8659 and RFC 8657, and check out the community forum for more from Subscribers who’ve set up or are using CAA.
As with anything with DNS, there are some potential hiccups to avoid. The most important to highlight is that CAA will always respect the CAA record closest to the domain name it is issuing a certificate for. For more on this, check out this section of the CAA documentation. You’ll also want to ensure that your DNS provider supports setting CAA records.
Thanks to Famedly
We’re grateful for Famedly taking the time to share with us more about their experience in setting up CAA. What’s more, Famedly financially supported ISRG this year as part of our tenth anniversary campaign.
As a project of the Internet Security Research Group (ISRG), 100% of the funding for Let’s Encrypt comes from contributions from our community of users and supporters. We depend on their support in order to provide our public benefit services. If your company or organization would like to sponsor Let’s Encrypt, please email us at sponsor@letsencrypt.org. If you or your organization can support us with a donation of any size, we ask that you consider a contribution.
When Let’s Encrypt first launched, we needed to ensure that our certificates were widely trusted. To that end, we arranged to have our intermediate certificates cross-signed by IdenTrust’s DST Root CA X3. This meant that all certificates issued by those intermediates would be trusted, even while our own ISRG Root X1 wasn’t yet. During subsequent years, our Root X1 became widely trusted on its own.
Come late 2021, our cross-signed intermediates and DST Root CA X3 itself were expiring. And while all up-to-date browsers at that time trusted our root, over a third of Android devices were still running old versions of the OS which would suddenly stop trusting websites using our certificates. That breakage would have been too widespread, so we arranged for a new cross-sign – this time directly onto our root rather than our intermediates – which would outlive DST Root CA X3 itself. This stopgap allowed those old Android devices to continue trusting our certificates for three more years.
On September 30th, 2024, that cross-sign too will expire.
In the last three years, the percentage of Android devices which trust our ISRG Root X1 has risen from 66% to 93.9%. That percentage will increase further over the next year, especially as Android releases version 14, which has the ability to update its trust store without a full OS update. In addition, dropping the cross-sign will reduce the number of certificate bytes sent in a TLS handshake by over 40%. Finally, it will significantly reduce our operating costs, allowing us to focus our funding on continuing to improve your privacy and security.
For these reasons, we will not be getting a new cross-sign to extend compatibility any further.
The transition will roll out as follows:
-
On Thursday, Feb 8th, 2024, we stopped providing the cross-sign by default in requests made to our /acme/certificate API endpoint. For most Subscribers, this means that your ACME client will configure a chain which terminates at ISRG Root X1, and your webserver will begin providing this shorter chain in all TLS handshakes. The longer chain, terminating at the soon-to-expire cross-sign, will still be available as an alternate chain which you can configure your client to request.
-
On Thursday, June 6th, 2024, we will stop providing the longer cross-signed chain entirely. This is just over 90 days (the lifetime of one certificate) before the cross-sign expires, and we need to make sure subscribers have had at least one full issuance cycle to migrate off of the cross-signed chain.
-
On Monday, September 30th, 2024, the cross-signed certificate will expire. This should be a non-event for most people, as any client breakages should have occurred over the preceding six months.
If you use Android 7.0 or earlier, you may need to take action to ensure you can still access websites secured by Let’s Encrypt certificates. We recommend installing and using Firefox Mobile, which uses its own trust store instead of the Android OS trust store, and therefore trusts ISRG Root X1.
If you are a site operator, you should keep an eye on your website usage statistics and active user-agent strings during Q2 and Q3 of 2024. If you see a sudden drop in visits from Android, it is likely because you have a significant population of users on Android 7.0 or earlier. We encourage you to provide the same advice to them as we provided above.
If you are an ACME client author, please make sure that your client correctly downloads and installs the certificate chain provided by our API during every certificate issuance, including renewals. Failure modes we have seen in the past include a) never downloading the chain at all and only serving the end-entity certificate; b) never downloading the chain and instead serving a hard-coded chain; and c) only downloading the chain at first issuance and not re-downloading during renewals. Please ensure that your client does not fall into any of these buckets.
We appreciate your understanding and support, both now and in the years to come as we provide safe and secure communication to everyone who uses the web. If you have any questions about this transition or any of the other work we do, please ask on our community forum.
We’d like to thank IdenTrust for their years of partnership. They played an important role in helping Let’s Encrypt get to where we are today and their willingness to arrange a stopgap cross sign in 2021 demonstrated a true commitment to creating a secure Web.
We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.
It’s hard to believe 10 years have passed since Eric Rescorla, Alex Halderman, Peter Eckersley and I founded ISRG as a nonprofit home for public benefit digital infrastructure. We had an ambitious vision, but we couldn’t have known then the extent to which that vision would become shared and leveraged by so much of the Internet.
Since its founding in 2013, ISRG’s Let’s Encrypt certificate authority has come to serve hundreds of millions of websites and protect just about everyone who uses the Web. Our Prossimo project has brought the urgent issue of memory safety to the fore, and Divvi Up is set to revolutionize the way apps collect metrics while preserving user privacy. I’ve tried to comprehend how much data about peoples' lives our work has and will protect, and tried even harder to comprehend what that means if one could quantify privacy. It’s simply beyond my ability.
Some of the highlights from the past ten years include:
-
May 24, 2013: ISRG is incorporated, intending to build Let’s Encrypt
-
November 18, 2014: The Let’s Encrypt project is announced publicly
-
September 14, 2015: Let’s Encrypt issues its first certificate
-
October 19, 2015: Let’s Encrypt becomes publicly trusted
-
December 3, 2015: Let’s Encrypt becomes generally available
-
March 8, 2016: Let’s Encrypt issues its millionth certificate
-
June 28, 2017: Let’s Encrypt issues its 100 millionth certificate
-
March 11, 2019: The ACME protocol becomes an IETF standard
-
February 27, 2020: Let’s Encrypt issues its billionth certificate
-
October 26, 2020: ISRG board approves a privacy preserving metrics project, now Divvi Up
-
December 9, 2020: ISRG board approves a memory safety project, now Prossimo
-
December 18, 2020: Divvi Up starts servicing COVID exposure notification
-
October 3, 2022: Support for Rust is merged into the Linux kernel
All this wouldn’t be possible without our staff, community, donors, funders, and other partners, all of whom I’d like to thank wholeheartedly.
I feel so fortunate that we’ve been able to thrive. We’re fortunate primarily because great people got involved and funders stepped up, but there’s also just a bit of good fortune involved in any success story. The world is a complicated place, there is complex context that one can’t control around every effort. Despite our best efforts, fortune has a role to play in terms of the degree to which the context swirling around us helps or hinders. We have been fortunate in every sense of the word and for that I am grateful.
Our work is far from over. Each of our three projects has challenges and opportunities ahead.
For Let’s Encrypt, which is more critical than ever and relatively mature, our focus over the next few years will be on long-term sustainability. More and more people working with certificates can’t recall a time when Let’s Encrypt didn’t exist, and most people who benefit from our service don’t need to know it exists at all (by design!). Let’s Encrypt is just part of how the Internet works now, which is great for many reasons, but it also means it’s at risk of being taken for granted. We are making sure that doesn’t happen so we can keep Let’s Encrypt running reliably and make investments in its future.
Prossimo is making a huge amount of progress moving critical software infrastructure to memory safe code, from the Linux kernel to NTP, TLS, media codecs, and even sudo/su. We have two major challenges ahead of us here. The first is to raise the money we need to complete development work. The second is to get the safer software we’ve been building adopted widely. We feel pretty good about our plans but it’s not going to be easy. Things worth doing rarely are.
Divvi Up is exciting technology with a bright future. Our biggest challenge here, like most things involving cryptography, is to make it easy to use. We also need to make sure we can provide the service at a cost that will allow for widespread adoption, so we’ll be doing a lot of optimization. Our hope is that over the next decade we can make privacy respecting metrics the norm, just like we did for HTTPS.
The internet wasn’t built with security or privacy in mind, so there is a bountiful opportunity for us to improve its infrastructure. The Internet is also constantly growing and changing, so it is also our job to look into the future and prepare for the next set of threats and challenges as best we can.
Thanks to our supporters, we’ll continue adapting and responding to help ensure the Web is more secure long into the future. Please consider becoming a sponsor or making a donation in support of our work.
Wed, 24 May 2023 00:00:00 +0000