Let's Encrypt News

Intent to End OCSP Service

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Tue, 23 Jul 2024 00:00:00 +0000

---

More Memory Safety for Let’s Encrypt: Deploying ntpd-rs

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Mon, 24 Jun 2024 00:00:00 +0000

---

Let’s Encrypt Continues Partnership with Princeton to Bolster Internet Security

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Thu, 30 May 2024 00:00:00 +0000

---

Takeaways from Tailscale’s Adoption of ARI

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Wed, 01 May 2024 00:00:00 +0000

---

An Engineer’s Guide to Integrating ARI into Existing ACME Clients

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Thu, 25 Apr 2024 00:00:00 +0000

---

Deploying Let's Encrypt's New Issuance Chains

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Fri, 12 Apr 2024 00:00:00 +0000

---

New Intermediate Certificates

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.

First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.

Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.

The New Certificates

We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.

We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.

You can see details of all of the certificates on our updated hierarchy page.

Rotating Issuance

Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.

Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.

We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.

These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.

Providing Smaller Chains

When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.

ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

Other Minor Changes

We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:

• We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.

• We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.

Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.

Deployment

We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.

Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.

We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Tue, 19 Mar 2024 00:00:00 +0000

---

Introducing Sunlight, a CT implementation built for scalability, ease of operation, and reduced cost

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.

First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.

Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.

The New Certificates

We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.

We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.

You can see details of all of the certificates on our updated hierarchy page.

Rotating Issuance

Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.

Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.

We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.

These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.

Providing Smaller Chains

When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.

ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

Other Minor Changes

We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:

• We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.

• We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.

Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.

Deployment

We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.

Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.

We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.

CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.

In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.

Bottlenecks from the Database

Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.

The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.

Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.

Serving Tiles

Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.

Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.

Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.

Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!

The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.

Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.

Running More Logs

The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.

The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective required by CT log programs.

The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.

No More Merge Delay

One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.

Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.

It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.

A Sunny Future

Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.

We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”

If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).

We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.

Thu, 14 Mar 2024 00:00:00 +0000

---

A Year-End Letter from our Vice President

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.

First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.

Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.

The New Certificates

We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.

We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.

You can see details of all of the certificates on our updated hierarchy page.

Rotating Issuance

Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.

Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.

We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.

These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.

Providing Smaller Chains

When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.

ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

Other Minor Changes

We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:

• We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.

• We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.

Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.

Deployment

We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.

Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.

We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.

CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.

In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.

Bottlenecks from the Database

Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.

The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.

Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.

Serving Tiles

Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.

Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.

Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.

Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!

The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.

Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.

Running More Logs

The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.

The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective required by CT log programs.

The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.

No More Merge Delay

One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.

Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.

It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.

A Sunny Future

Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.

We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”

If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).

We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.

This letter was originally published in our 2023 Annual Report.

We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.

One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.

Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.

Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with

Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.

We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.

Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.

Thu, 28 Dec 2023 00:00:00 +0000

---

Our role in supporting the nonprofit ecosystem

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.

First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.

Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.

The New Certificates

We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.

We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.

You can see details of all of the certificates on our updated hierarchy page.

Rotating Issuance

Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.

Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.

We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.

These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.

Providing Smaller Chains

When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.

ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

Other Minor Changes

We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:

• We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.

• We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.

Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.

Deployment

We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.

Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.

We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.

CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.

In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.

Bottlenecks from the Database

Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.

The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.

Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.

Serving Tiles

Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.

Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.

Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.

Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!

The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.

Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.

Running More Logs

The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.

The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective required by CT log programs.

The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.

No More Merge Delay

One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.

Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.

It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.

A Sunny Future

Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.

We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”

If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).

We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.

This letter was originally published in our 2023 Annual Report.

We typically open our annual report with a letter from our Executive Director and co-founder, Josh Aas, but he’s on parental leave so I’ll be filling in. I’ve run the Brand & Donor Development team at ISRG since 2016, so I’ve had the pleasure of watching our work mature, our impact grow, and I’ve had the opportunity to get to know many great people who care deeply about security and privacy on the Internet.

One of the biggest observations I’ve made during Josh’s absence is that all 23 people who work at ISRG fall into that class of folks. Of course I was a bit nervous as Josh embarked on his leave to discover just how many balls he has been keeping in the air for the last decade. Answer: it’s a lot. But the roster of staff that we’ve built up made it pretty seamless for us to keep moving forward.

Let’s Encrypt is supporting 40 million more websites than a year ago, bringing the total to over 360 million. The engineering team has grown to 12 people who are responsible for our continued reliability and ability to scale. But they’re not maintaining the status quo. Let’s Encrypt engineers are pushing forward our expectations for ourselves and for the WebPKI community. We’ve added shorter-lived certificates to our 2024 roadmap. We’re committing to this work because sub-10 day certificates significantly reduce the impact of key compromise and it broadens the universe of people who can use our certs. In addition, the team started an ambitious project to develop a new Certificate Transparency implementation because the only existing option cannot scale for the future and is prone to operational fragility. These projects are led by two excellent technical leads, Aaron Gable and James Renken, who balance our ambition with our desire for a good quality of life for our teams.

Prossimo continues to deliver highly performant and memory safe software and components in a world that is increasingly eager to address the memory safety problem. This was evidenced by participation at Tectonics, a gathering we hosted which drew industry leaders for invigorated conversation. Meanwhile, initiatives like our memory safe AV1 decoder are in line to replace a C version in Google Chrome. This change would improve security for billions of people. We’re grateful to the community that helps to guide and implement our efforts in this area, including Dirkjan Ochtman, the firms Tweede golf and Ferrous Systems, and the maintainers of the many projects we are involved with

Our newest project, Divvi Up, brought on our first two subscribers in 2023. Horizontal, a small international nonprofit serving Human Rights Defenders, will be collecting privacy-preserving telemetry metrics about the users of their Tella app, which people use to document human rights violations. Mozilla is using Divvi Up to gain insight into aspects of user behavior in the Firefox browser. It took a combination of focus and determination to get us to a production-ready state and our technical lead, Brandon Pitman played a big role in getting us there.

We hired Kristin Berdan to fill a new role as General Counsel and her impact is already apparent within our organization. She joins Sarah Heil, our CFO, Josh, and me in ISRG leadership.

Collectively, we operate three impactful and growing projects for $7 million a year. This is possible because of the amazing leadership assembled across our teams and the ongoing commitment from our community to validate the usefulness of our work. As we look toward 2024 and the challenges and opportunities that face us, I ask that you join us in building a more secure and privacy respecting Internet by sponsoring us, making a donation or gift through your DAF, or sharing with the folks you know why security and privacy matter to them.

For more than ten years, we at the nonprofit Internet Security Research Group (ISRG) have been focused on our mission of building a more secure and privacy-respecting Internet for everyone, everywhere. As we touch on in our 2023 Annual Report, we now serve more than 360 million domains with free TLS certificates.

Beyond being a big number, what does that signify? What’s the importance of having TLS being widely adopted anyways? We’ll take a closer look at these questions through the lens of one group of Subscribers we can relate to particularly well: nonprofits.

Serving .org at Internet scale

Let’s Encrypt serves 57% of all websites using the .org top level domain (TLD), which is commonly used by nonprofits. In the US alone there are 1.8M registered nonprofit organizations. And while the focus of these organizations are varied, all of them rely on the Internet in some capacity.

When a nonprofit uses a TLS certificate on their website, it protects their visitors and stakeholders from snoopers, MITM attacks, and surveillance. Without TLS, nonprofits’ content could be changed without their knowledge or their visitors’ private information could be compromised. Access to free and automated TLS via Let’s Encrypt means these nonprofits face as few barriers as possible to adopting TLS.

In short, something as fundamental as security and privacy should be as easy to access as possible. For nonprofits both large and small, Let’s Encrypt makes it easy to provide security and privacy for users of their websites, enabling them to remain focused on their missions.

Zooming in on three nonprofits we serve

The American Civil Liberties Union (ACLU) uses Let’s Encrypt as it works to realize its focus of being a “guardian of liberty” for US citizens. Using Let’s Encrypt protects ACLU’s constituents when they’re trying to know their rights or take action. With more than 4 million page views per month, ACLU’s website is a critical part of their mission.

Human Rights Watch (HRW) is an international nonprofit organization. With more than 500 individuals on staff around the world, HRW’s website is a trove of information empowering individuals and organizations alike to be informed and take action with a global perspective. Nearly 70% of HRW’s web traffic comes from people outside of the United States; that’s millions of page views per month secured by Let’s Encrypt—and by extension, millions of people around the world benefitting from a more secure and privacy-respecting Web.

The Center for Democracy & Technology (CDT) uses Let’s Encrypt to advance its mission to promote democratic values by shaping technology policy and architecture, with a focus on the rights of the individual. The CDT website offers updated and insightful information into the ways policy and innovation impact the digital space. Without a TLS certificate, the content of these pages could be intercepted and changed. What’s more, for those looking to financially support CDT, using TLS on their donation page encrypts the transaction protecting user details such as credit card and other personal information. Mallory Knodel, CTO at CDT and longtime digital rights defender and advocate commented, “Billions of people in over 60 countries access the internet with less censorship and surveillance because Let’s Encrypt hastened the adoption of web security measures by making certificates easy to obtain.”

Serving philanthropic foundations

In the United States, the work of nonprofits is made possible in large part through philanthropic foundations and organizations. When it comes to philanthropy’s web presence, Let’s Encrypt is there, too.

We provide TLS to billion dollar philanthropic organizations like the Hewlett Foundation, the Silicon Valley Community Foundation, Yield Giving, and many others. Taking a look at the top 50 philanthropic organizations around the world, Let’s Encrypt serves 36% of them. For large philanthropies, their website is the primary tool they have to communicate their focus areas for future funding as well as the impact they’ve made with past giving.

One of the leading philanthropists in the US, Craig Newmark, uses Let’s Encrypt and Digital Ocean for his website, craig newmark philanthropies. Commenting on our work, Craig recently shared, “The people at ISRG have been helping protect the Internet for over ten years, and continue to protect us all. They’re a necessary part of Cyber Civil Defense and national security.”

Overall, while Let’s Encrypt aims to build a better Internet, we’re particularly proud that our impact protects those seeking to build a better world.

Internet Security Research Group (ISRG) is the parent organization of Prossimo, Let’s Encrypt, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Wed, 13 Dec 2023 00:00:00 +0000

---

Social Media

• GitHub
• Google+
• LinkedIn
• Twitter
• Instagram