Let's Encrypt News

A Note from our Executive Director

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Wed, 11 Dec 2024 00:00:00 +0000

---

Ending OCSP Support in 2025

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Thu, 05 Dec 2024 00:00:00 +0000

---

Intent to End OCSP Service

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Tue, 23 Jul 2024 00:00:00 +0000

---

More Memory Safety for Let’s Encrypt: Deploying ntpd-rs

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Mon, 24 Jun 2024 00:00:00 +0000

---

Let’s Encrypt Continues Partnership with Princeton to Bolster Internet Security

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Thu, 30 May 2024 00:00:00 +0000

---

Takeaways from Tailscale’s Adoption of ARI

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Wed, 01 May 2024 00:00:00 +0000

---

An Engineer’s Guide to Integrating ARI into Existing ACME Clients

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Thu, 25 Apr 2024 00:00:00 +0000

---

Deploying Let's Encrypt's New Issuance Chains

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Fri, 12 Apr 2024 00:00:00 +0000

---

New Intermediate Certificates

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.

First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.

Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.

The New Certificates

We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.

We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.

You can see details of all of the certificates on our updated hierarchy page.

Rotating Issuance

Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.

Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.

We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.

These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.

Providing Smaller Chains

When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.

ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

Other Minor Changes

We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:

• We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.

• We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.

Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.

Deployment

We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.

Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.

We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Tue, 19 Mar 2024 00:00:00 +0000

---

Introducing Sunlight, a CT implementation built for scalability, ease of operation, and reduced cost

This letter was originally published in our 2024 Annual Report.

The past year at ISRG has been a great one and I couldn’t be more proud of our staff, community, funders, and other partners that made it happen. Let’s Encrypt continues to thrive, serving more websites around the world than ever before with excellent security and stability. Our understanding of what it will take to make more privacy-preserving metrics more mainstream via our Divvi Up project is evolving in important ways.

Prossimo has made important investments in making software critical infrastructure safer, from TLS and DNS to the Linux kernel.

Next year is the 10th anniversary of the launch of Let’s Encrypt. Internally things have changed dramatically from what they looked like ten years ago, but outwardly our service hasn’t changed much since launch. That’s because the vision we had for how best to do our job remains as powerful today as it ever was: free 90-day TLS certificates via an automated API. Pretty much as many as you need. More than 500,000,000 websites benefit from this offering today, and the vast majority of the web is encrypted.

Our longstanding offering won’t fundamentally change next year, but we are going to introduce a new offering that’s a big shift from anything we’ve done before - short-lived certificates. Specifically, certificates with a lifetime of six days. This is a big upgrade for the security of the TLS ecosystem because it minimizes exposure time during a key compromise event.

Because we’ve done so much to encourage automation over the past decade, most of our subscribers aren’t going to have to do much in order to switch to shorter lived certificates. We, on the other hand, are going to have to think about the possibility that we will need to issue 20x as many certificates as we do now. It’s not inconceivable that at some point in our next decade we may need to be prepared to issue 100,000,000 certificates per day.

That sounds sort of nuts to me today, but issuing 5,000,000 certificates per day would have sounded crazy to me ten years ago. Here’s the thing though, and this is what I love about the combination of our staff, partners, and funders - whatever it is we need to do to doggedly pursue our mission, we’re going to get it done. It was hard to build Let’s Encrypt. It was difficult to scale it to serve half a billion websites. Getting our Divvi Up service up and running from scratch in three months to service exposure notification applications was not easy. Our Prossimo project was a primary contributor to the creation of a TLS library that provides memory safety while outperforming its peers - a heavy lift.

Charitable contributions from people like you and organizations around the world make this stuff possible. Since 2015, tens of thousands of people have donated. They’ve made a case for corporate sponsorship, given through their DAFs, or set up recurring donations, sometimes to give $3 a month. That’s all added up to millions of dollars that we’ve used to change the Internet for nearly everyone using it. I hope you’ll join these people and help lay the foundation for another great decade.

Josh Aas
Executive Director

Earlier this year we announced our intent to provide certificate revocation information exclusively via Certificate Revocation Lists (CRLs), ending support for providing certificate revocation information via the Online Certificate Status Protocol (OCSP). Today we are providing a timeline for ending OCSP services:

• January 30, 2025
  • OCSP Must-Staple requests will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension
• May 7, 2025
  • Prior to this date we will have added CRL URLs to certificates
  • On this date we will drop OCSP URLs from certificates
  • On this date all requests including the OCSP Must Staple extension will fail
• August 6, 2025
  • On this date we will turn off our OCSP responders

Additionally, a very small percentage of our subscribers request certificates with the OCSP Must Staple Extension. If you have manually configured your ACME client to request that extension, action is required before May 7. See “Must Staple” below for details.

OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL.

Must Staple

Because of the privacy issues with OCSP, browsers and servers implement a feature called “OCSP Stapling”, where the web server sends a copy of the appropriate OCSP response during the TLS handshake, and the browser skips making a request to the CA, thus better preserving privacy.

In addition to OCSP Stapling (a TLS feature negotiated at handshake time), there’s an extension that can be added to certificates at issuance time, colloquially called “OCSP Must Staple.” This tells browsers that, if they see that extension in a certificate, they should never contact the CA about it and should instead expect to see a stapled copy in the handshake. Failing that, browsers should refuse to connect. This was designed to solve some security problems with revocation.

Let’s Encrypt has supported OCSP Must Staple for a long time, because of the potential to improve both privacy and security. However, Must Staple has failed to get wide browser support after many years. And popular web servers still implement OCSP Stapling in ways that create serious risks of downtime.

As part of removing OCSP, we’ll also be removing support for OCSP Must Staple. CRLs have wide browser support and can provide privacy benefits to all sites, without requiring special web server configuration. Thanks to all our subscribers who have helped with the OCSP Must Staple experiment.

If you are not certain whether you are using OCSP Must Staple, you can check this list of hostnames and certificate serials (11.1 MB, .zip).

As of January 30, 2025, issuance requests that include the OCSP Must Staple extension will fail, unless the requesting account has previously issued a certificate containing the OCSP Must Staple extension.

As of May 7, all issuance requests that include the OCSP Must Staple extension will fail, including renewals. Please change your ACME client configuration to not request the extension.

Today we are announcing our intent to end Online Certificate Status Protocol (OCSP) support in favor of Certificate Revocation Lists (CRLs) as soon as possible. OCSP and CRLs are both mechanisms by which CAs can communicate certificate revocation information, but CRLs have significant advantages over OCSP. Let’s Encrypt has been providing an OCSP responder since our launch nearly ten years ago. We added support for CRLs in 2022.

Websites and people who visit them will not be affected by this change, but some non-browser software might be.

We plan to end support for OCSP primarily because it represents a considerable risk to privacy on the Internet. When someone visits a website using a browser or other software that checks for certificate revocation via OCSP, the Certificate Authority (CA) operating the OCSP responder immediately becomes aware of which website is being visited from that visitor’s particular IP address. Even when a CA intentionally does not retain this information, as is the case with Let’s Encrypt, CAs could be legally compelled to collect it. CRLs do not have this issue.

We are also taking this step because keeping our CA infrastructure as simple as possible is critical for the continuity of compliance, reliability, and efficiency at Let’s Encrypt. For every year that we have existed, operating OCSP services has taken up considerable resources that can soon be better spent on other aspects of our operations. Now that we support CRLs, our OCSP service has become unnecessary.

In August of 2023 the CA/Browser Forum passed a ballot to make providing OCSP services optional for publicly trusted CAs like Let’s Encrypt. With one exception, Microsoft, the root programs themselves no longer require OCSP. As soon as the Microsoft Root Program also makes OCSP optional, which we are optimistic will happen within the next six to twelve months, Let’s Encrypt intends to announce a specific and rapid timeline for shutting down our OCSP services. We hope to serve our last OCSP response between three and six months after that announcement. The best way to stay apprised of updates on these plans is to subscribe to our API Announcements category on Discourse.

We recommend that anyone relying on OCSP services today start the process of ending that reliance as soon as possible. If you use Let’s Encrypt certificates to secure non-browser communications such as a VPN, you should ensure that your software operates correctly if certificates contain no OCSP URL. Fortunately, most OCSP implementations “fail open” which means that an inability to fetch an OCSP response will not break the system.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

When we look at the general security posture of Let’s Encrypt, one of the things that worries us most is how much of the operating system and network infrastructure is written in unsafe languages like C and C++. The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

Partially for the sake of Let’s Encrypt, and partially for the sake of the wider Internet, we started a new project called Prossimo in 2020. Prossimo’s goal is to make some of the most critical software infrastructure for the Internet memory safe. Since then we’ve invested in a range of software components including the Rustls TLS library, Hickory DNS, River reverse proxy, sudo-rs, Rust support for the Linux kernel, and ntpd-rs.

Let’s Encrypt has now taken a step that was a long time in the making: we’ve deployed ntpd-rs, the first piece of memory safe software from Prossimo that has made it into the Let’s Encrypt infrastructure.

Most operating systems use the Network Time Protocol (NTP) to accurately determine what time it is. Keeping track of time is a critical task for an operating system, and since it involves interacting with the Internet it’s important to make sure NTP implementations are secure.

In April of 2022, Prossimo started work on a memory safe and generally more secure NTP implementation called ntpd-rs. Since then, the implementation has matured and is now maintained by Project Pendulum. In April of 2024 ntpd-rs was deployed to the Let’s Encrypt staging environment, and as of now it’s in production.

Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to have been partnering with the Center for Information Technology Policy team at Princeton University since 2018 to bolster defenses against Border Gateway Protocol (BGP) attacks. We’re thrilled to continue this partnership thanks to renewed funding from the Open Technology Fund.

“Let’s Encrypt has played a pivotal role in driving our research around protecting against BGP attacks and preventing the disruption such attacks can cause. We’re grateful for the partnership with Let’s Encrypt, as the largest Certificate Authority, in this critical work.” – Jennifer Rexford, Provost, Princeton University

To date, our work with Princeton has focused on defending against BGP attacks on domain control validation via Multi-Perspective Issuance Corroboration (MPIC). This year, Let’s Encrypt is adding two new remote perspectives for domain validation. This means we will make five total validation requests, one from the primary datacenter and four from remote perspectives (previously two). Increased perspectives provide more domain validation security, thus improving visibility and protection against BGP attacks.

Additionally, we will be facilitating the adoption of ACME Renewal Information (ARI) in order to enable certificate authorities (CAs) to maintain continuity of service in a mass revocation/replacement event. If a BGP attack does occur, ARI will allow CAs to quickly and automatically revoke and replace certificates associated with the victim domain. Learn more about how to integrate ARI into an existing ACME client.

Our team will be working with the research groups of Professor Prateek Mittal to provide secure data related to increased perspectives and ARI, and contributing to research analysis and discoveries.

We’d like to thank Princeton University for their partnership on this important work, and Open Technology Fund for making it possible.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Since March 2023, Let’s Encrypt has been improving our resiliency and reliability via ACME Renewal Information (ARI). ARI makes it possible for our Subscribers to handle certificate revocation and renewal easily and automatically. A primary benefit of ARI is that it sets Subscribers up for success in terms of ideal renewal times in the event that Let’s Encrypt offers certificates with even shorter lifetimes than 90 days. We recently published a guide for engineers on how to integrate ARI into existing ACME Clients.

In this blog post, we’ll explore Let’s Encrypt Subscriber Tailscale’s experience adopting ARI.

In total, it took just two Tailscale engineers less than two days to implement ARI. Prior to ARI, the Tailscale team had made other iterations of cert renewal logic, including hardcoding renewal 14 days before expiry and hardcoding 1/3rd of remaining time until expiry. An issue with these approaches was that assumptions were made about the validity period of certificates issued by Let’s Encrypt, which will change in the future. In contrast, ARI allows Tailscale to offload the renewal decision to Let’s Encrypt without making any assumptions.

Tailscale noted that ARI was especially useful to add before certificates’ validity period starts shortening, as their client software in charge of requesting and renewing certificates is running on user machines. This makes it so they cannot easily update the whole fleet overnight if any issues come up. Thanks to ARI, they’ve reduced the risk of not rotating certificates for client machines in time, or causing excessive load on Let’s Encrypt’s infrastructure with overly-eager rotation logic.

One consideration the Tailscale team factored in deciding to adopt ARI was wanting to avoid adding a hard dependency on the Let’s Encrypt infrastructure for renewal. To remedy this, Tailscale certificate renewal logic falls back to local time-based check if the ARI endpoint cannot be reached for any reason.

Tailscale’s roadmap for getting ARI in production:

• Updated their fork of golang.org/x/crypto to support ARI

• Updated the renewal code in the Tailscale client

• Tested it locally by requesting certificates for a dev domain

• Tested renewal by stubbing out ARI response with hardcoded data

• Tested fallback by blocking ARI requests

• Shipped it!

The team reported running into one snag during the process. Because the RFC is not finalized, the upstream Go package for ACME doesn’t support ARI yet. As a solution, they added support in their fork of that Go package. Tailscale’s main piece of advice for Subscribers adopting ARI: don’t forget to put a timeout on your ARI request!

We’re grateful to the Tailscale team for taking the time to share with us their experience adopting ARI and advice for fellow Subscribers. In addition to being an ARI adopter, Tailscale is a Let’s Encrypt Sponsor! We appreciate their support of our work to build a more secure Web.

We’re also grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

Following our previous post on the foundational benefits of ACME Renewal Information (ARI), this one offers a detailed technical guide for incorporating ARI into existing ACME clients.

Since its introduction in March 2023, ARI has significantly enhanced the resiliency and reliability of certificate revocation and renewal for a growing number of Subscribers. To extend these benefits to an even broader audience, incorporating ARI into more ACME clients is essential.

To foster wider adoption, we’re excited to announce a new compelling incentive: certificate renewals that utilize ARI will now be exempt from all rate limits. To capitalize on this benefit, renewals must occur within the ARI-suggested renewal window, and the request must clearly indicate which existing certificate is being replaced. To learn how to request a suggested renewal window, select an optimal renewal time, and specify certificate replacement, continue reading!

Integrating ARI Into an Existing ACME Client

In May 2023, we contributed a pull request to the Lego ACME client, adding support for draft-ietf-acme-ari-01. In December 2023 and February 2024, we contributed two follow-up pull requests (2066, 2114) adding support for changes made in draft-ietf-acme-ari-02 and 03. These experiences provided valuable insight into the process of integrating ARI into an existing ACME client. We’ve distilled these insights into six steps, which we hope will be useful for other ACME client developers.

Note: the code snippets in this post are written in Golang. We’ve structured and contextualized them for clarity, so that they might be easily adapted to other programming languages as well.

Step 1: Detecting support for ARI

While Let’s Encrypt first enabled ARI in Staging and Production environments in March 2023, many ACME clients are used with a variety of CAs, so it’s crucial to ascertain if a CA supports ARI. This can be easily determined: if a ‘renewalInfo’ endpoint is included in the CA’s directory object, then the CA supports ARI.

In most any client you’ll find a function or method that is responsible for parsing the JSON of the ACME directory object. If this code is deserializing the JSON into a defined type, it will be necessary to modify this type to include the new ‘renewalInfo’ endpoint.

In Lego, we added a ‘renewalInfo’ field to the Directory struct, which is accessed by the GetDirectory method:

type Directory struct {
    NewNonceURL    string `json:"newNonce"`
    NewAccountURL  string `json:"newAccount"`
    NewOrderURL    string `json:"newOrder"`
    NewAuthzURL    string `json:"newAuthz"`
    RevokeCertURL  string `json:"revokeCert"`
    KeyChangeURL   string `json:"keyChange"`
    Meta           Meta   `json:"meta"`
    RenewalInfo    string `json:"renewalInfo"`
}

As we discussed above, not all ACME CAs currently implement ARI, so before we attempt to make use of the ‘renewalInfo’ endpoint we should ensure that this endpoint is actually populated before calling it:

func (c *CertificateService) GetRenewalInfo(certID string) (*http.Response, error) {
  if c.core.GetDirectory().RenewalInfo == "" {
    return nil, ErrNoARI
  }
}

Step 2: Determining where ARI fits into the renewal lifecycle of your client

The next step involves selecting the optimal place in the client’s workflow to integrate ARI support. ACME clients can either run persistently or be executed on-demand. ARI is particularly beneficial for clients that operate persistently or for on-demand clients that are scheduled to run at least daily.

In the case of Lego, it falls into the latter category. Its renew command is executed on-demand, typically through a job scheduler like cron. Therefore, incorporating ARI support into the renew command was the logical choice. Like many ACME clients, Lego already has a mechanism to decide when to renew certificates, based on the certificate’s remaining validity period and the user’s configured renewal timeframe. Introducing calls to ARI should take precedence over this mechanism, leading to a modification of the renew command to consult ARI before resorting to the built-in logic.

Step 3: Constructing the ARI CertID

The composition of the ARI CertID is a crucial part of the ARI specification. This identifier, unique to each certificate, is derived by combining the base64url encoded bytes of the certificate’s Authority Key Identifier (AKI) extension and its Serial Number, separated by a period. The approach of combining AKI and serial number is strategic: the AKI is specific to an issuing intermediate certificate, and a CA may have multiple intermediates. A certificate’s serial number is required to be unique per issuing intermediate, but serials can be reused between intermediates. Thus the combination of AKI and serial uniquely identifies a certificate. With this covered, let’s move on to constructing an ARI CertID using only the contents of the certificate being replaced.

Suppose the ‘keyIdentifier’ field of the certificate’s Authority Key Identifier (AKI) extension has the hexadecimal bytes 69:88:5B:6B:87:46:40:41:E1:B3:7B:84:7B:A0:AE:2C:DE:01:C8:D4 as its ASN.1 Octet String value. The base64url encoding of these bytes is aYhba4dGQEHhs3uEe6CuLN4ByNQ=. Additionally, the certificate’s Serial Number, when represented in its DER encoding (excluding the tag and length bytes), has the hexadecimal bytes 00:87:65:43:21. This includes a leading zero byte to ensure that the serial number is interpreted as a positive integer, as necessitated by the leading 1 bit in 0x87. The base64url encoding of these bytes is AIdlQyE=. After stripping the trailing padding characters ("=") from each encoded part and concatenating them with a period as a separator, the ARI CertID for this certificate is aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE.

In the case of Lego, we implemented the above logic in the following function:

// MakeARICertID constructs a certificate identifier as described in
// draft-ietf-acme-ari-03, section 4.1.

func MakeARICertID(leaf *x509.Certificate) (string, error) {
  if leaf == nil {
    return "", errors.New("leaf certificate is nil")
  }

  // Marshal the Serial Number into DER.
  der, err := asn1.Marshal(leaf.SerialNumber)
  if err != nil {
    return "", err
  }

  // Check if the DER encoded bytes are sufficient (at least 3 bytes: tag,
  // length, and value).
  if len(der) < 3 {
    return "", errors.New("invalid DER encoding of serial number")
  }

  // Extract only the integer bytes from the DER encoded Serial Number
  // Skipping the first 2 bytes (tag and length). The result is base64url
  // encoded without padding.
  serial := base64.RawURLEncoding.EncodeToString(der[2:])

  // Convert the Authority Key Identifier to base64url encoding without
  // padding.
  aki := base64.RawURLEncoding.EncodeToString(leaf.AuthorityKeyId)

  // Construct the final identifier by concatenating AKI and Serial Number.
  return fmt.Sprintf("%s.%s", aki, serial), nil
}

Note: In the provided code, we utilize the RawURLEncoding, which is the unpadded base64 encoding as defined in RFC 4648. This encoding is similar to URLEncoding but excludes padding characters, such as “=”. Should your programming language’s base64 package only support URLEncoding, it will be necessary to remove any trailing padding characters from the encoded strings before combining them.

Step 4: Requesting a suggested renewal window

With the ARI CertID in hand, we can now request renewal information from the CA. This is done by sending a GET request to the ‘renewalInfo’ endpoint, including the ARI CertID in the URL path.

GET https://example.com/acme/renewal-info/aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE

The ARI response is a JSON object that includes a ‘suggestedWindow’, with ‘start’ and ’end’ timestamps indicating the recommended renewal period, and optionally, an ’explanationURL’ providing additional context about the renewal suggestion.

{
  "suggestedWindow": {
    "start": "2021-01-03T00:00:00Z",
    "end": "2021-01-07T00:00:00Z"
  },
  "explanationURL": "https://example.com/docs/ari"
}

The ’explanationURL’ is optional. However, if it’s provided, it’s recommended to display it to the user or log it. For instance, in cases where ARI suggests an immediate renewal due to an incident that necessitates revocation, the ’explanationURL’ might link to a page explaining the incident.

Next, we’ll cover how to use the ‘suggestedWindow’ to determine the best time to renew the certificate.

Step 5: Selecting a specific renewal time

draft-ietf-acme-ari provides a suggested algorithm for determining when to renew a certificate. This algorithm is not mandatory, but it is recommended.

1. Select a uniform random time within the suggested window.

2. If the selected time is in the past, attempt renewal immediately.

3. Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.

4. Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

5. Otherwise, sleep until the next normal wake time, re-check ARI, and return to “1.”

For Lego, we implemented the above logic in the following function:

func (r *RenewalInfoResponse) ShouldRenewAt(now time.Time, willingToSleep time.Duration) *time.Time {

  // Explicitly convert all times to UTC.
  now = now.UTC()
  start := r.SuggestedWindow.Start.UTC()
  end := r.SuggestedWindow.End.UTC()

  // Select a uniform random time within the suggested window.
  window := end.Sub(start)
  randomDuration := time.Duration(rand.Int63n(int64(window)))
  rt := start.Add(randomDuration)

  // If the selected time is in the past, attempt renewal immediately.
  if rt.Before(now) {
    return &now
  }

  // Otherwise, if the client can schedule itself to attempt renewal at exactly the selected time, do so.
  willingToSleepUntil := now.Add(willingToSleep)
  if willingToSleepUntil.After(rt) || willingToSleepUntil.Equal(rt) {
    return &rt
  }

  // TODO: Otherwise, if the selected time is before the next time that the client would wake up normally, attempt renewal immediately.

  // Otherwise, sleep until the next normal wake time.

  return nil

}

Step 6: Indicating which certificate is replaced by this new order

To signal that a renewal was suggested by ARI, a new ‘replaces’ field has been added to the ACME Order object. The ACME client should populate this field when creating a new order, as shown in the following example:

{
  "protected": base64url({
    "alg": "ES256",
    "kid": "https://example.com/acme/acct/evOfKhNU60wg",
    "nonce": "5XJ1L3lEkMG7tR6pA00clA",
    "url": "https://example.com/acme/new-order"
  }),
  "payload": base64url({
    "identifiers": [
      { "type": "dns", "value": "example.com" }
    ],
    "replaces": "aYhba4dGQEHhs3uEe6CuLN4ByNQ.AIdlQyE"
  }),
  "signature": "H6ZXtGjTZyUnPeKn...wEA4TklBdh3e454g"
}

Many clients will have an object that the client deserializes into the JSON used for the order request. In the Lego client, this is the Order struct. It now includes a ‘replaces’ field, accessed by the NewWithOptions method:

// Order the ACME order Object.
// - https://www.rfc-editor.org/rfc/rfc8555.html#section-7.1.3

type Order struct {
  ...
  // replaces (optional, string):
  // a string uniquely identifying a previously-issued
  // certificate which this order is intended to replace.
  // - https://datatracker.ietf.org/doc/html/draft-ietf-acme-ari-03#section-5
  Replaces string `json:"replaces,omitempty"`
}

...

// NewWithOptions Creates a new order.
func (o *OrderService) NewWithOptions(domains []string, opts *OrderOptions) (acme.ExtendedOrder, error) {
  ...
  if o.core.GetDirectory().RenewalInfo != "" {
    orderReq.Replaces = opts.ReplacesCertID
  }
}

When Let’s Encrypt processes a new order request featuring a ‘replaces’ field, several important checks are conducted. First, it’s verified that the certificate indicated in this field has not been replaced previously. Next, we ensure that the certificate is linked to the same ACME account that’s making the current request. Additionally, there must be at least one domain name shared between the existing certificate and the one being requested. If these criteria are met and the new order request is submitted within the ARI-suggested renewal window, the request qualifies for exemption from all rate limits. Congratulations!

Moving Forward

The integration of ARI into more ACME clients isn’t just a technical upgrade, it’s the next step in the evolution of the ACME protocol; one where CAs and clients work together to optimize the renewal process, ensuring lapses in certificate validity are a thing of the past. The result is a more secure and privacy-respecting Internet for everyone, everywhere.

As always, we’re excited to engage with our community on this journey. Your insights, experiences, and feedback are invaluable as we continue to push the boundaries of what’s possible with ACME.

We’re grateful to be partnering with Princeton University on our ACME Renewal Information work, thanks to generous support from the Open Technology Fund.

Internet Security Research Group (ISRG) is the parent organization of Let’s Encrypt, Prossimo, and Divvi Up. ISRG is a 501(c)(3) nonprofit. If you’d like to support our work, please consider getting involved, donating, or encouraging your company to become a sponsor.

On Thursday, June 6th, 2024, we will be switching issuance to use our new intermediate certificates. Simultaneously, we are removing the DST Root CA X3 cross-sign from our API, aligning with our strategy to shorten the Let’s Encrypt chain of trust. We will begin issuing ECDSA end-entity certificates from a default chain that just contains a single ECDSA intermediate, removing a second intermediate and the option to issue an ECDSA end-entity certificate from an RSA intermediate. The Let’s Encrypt staging environment will make an equivalent change on April 24th, 2024.

Most Let’s Encrypt Subscribers will not need to take any action in response to this change because ACME clients, like certbot, will automatically configure the new intermediates when certificates are renewed. The Subscribers who will be affected are those who currently pins intermediate certificates (more on that later).

The following diagram depicts what the new hierarchy looks like. You can see details of all of the certificates on our updated Chain of Trust documentation page.

New Intermediate Certificates

Earlier this year, Let’s Encrypt generated new intermediate keys and certificates. They will replace the current intermediates, which were issued in September 2020 and are approaching their expiration.

All certificates - issued by both RSA and ECDSA intermediates - will be served with a default chain of ISRG Root X1 → (RSA or ECDSA) Intermediate → End-Entity Certificate. That is, all certificates, regardless of whether you choose to have an RSA or ECDSA end-entity certificate, will have one intermediate which is directly signed by the ISRG Root X1, which is Let’s Encrypt’s most widely trusted root.

The new ECDSA intermediates will also have an alternate chain to ISRG Root X2: ISRG Root X2 → ECDSA Intermediate → End-Entity Certificate. This is only applicable to a small number of Subscribers who prefer the smallest TLS handshake possible. To use this ECDSA-only chain, see your ACME client’s documentation on how to request alternate chains. There will not be any alternative chains for the RSA intermediates.

It is important to note that there will now be multiple active RSA and two active ECDSA intermediates at the same time. An RSA leaf certificate may be signed by any of the active RSA intermediates (a value from “R10” to “R14” in the issuer common name field of your certificate), and an ECDSA leaf certificate may be signed by any of the active ECDSA intermediates (“E5” through “E9”). Again, your ACME client should handle this automatically.

A Certificate Authority’s intermediate certificates expire every few years and need to be replaced, just like a website’s certificate is routinely renewed. Going forward, Let’s Encrypt intends to switch what intermediates are in use annually, which will help enhance the overall security of the certificates.

Removing DST Root CA X3 Cross-sign

The new intermediate chains will not include the DST Root CA X3 cross-sign, as previously announced in our post about Shortening the Let’s Encrypt Chain of Trust. By eliminating the cross-sign, we’re making our certificates leaner and more efficient, leading to faster page loads for Internet users. We already stopped providing the cross-sign in the default certificate chain on February 8th, 2024, so if your ACME client is not explicitly requesting the chain with DST Root CA X3, this will not be a change for you.

ECDSA Intermediates as Default for ECDSA Certificates

Currently, ECDSA end-entity certificates are signed by our RSA intermediates unless users opted in via a request form to use our ECDSA intermediates. With our new intermediates, we will begin issuing all ECDSA end-entity certificates from the ECDSA intermediates. The request form and allow-list will no longer be used, which we had introduced to make ECDSA intermediates available.

Earlier, the default ECDSA chain included two intermediates: both E1 and the cross-signed ISRG Root X2 (i.e. ISRG Root X1 → ISRG Root X2 → E1 → End-Entity Certificate). After the change, it will contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1 (i.e. ISRG Root X1 → E5 → End-Entity Certificate). This ensures that all of our intermediates, both RSA and ECDSA, are signed directly by our most widely-trusted ISRG Root X1.

We expect this change to benefit most users with smaller TLS handshakes. If compatibility problems with ECDSA intermediates arise, we recommend Let’s Encrypt users switch to RSA certificates. Android 7.0 is known to have a bug preventing it from working with most Elliptic Curve (EC) certificates, including our ECDSA intermediates; however, that version of Android doesn’t trust our ISRG Root X1 and thus is already incompatible.

Risks of Pinning or Hard-Coding Intermediates

We do not recommend pinning or otherwise hard-coding intermediates or roots. Pinning intermediates is especially not advisable as they change often. If you do pin intermediates, make sure you have the complete set of new intermediates (available here).

Questions?

We’re grateful for the millions of subscribers who have trusted us to carry out best practices to make the web more secure and privacy-respecting, and rotating intermediates more frequently is one of them. We’d also like to thank our great community and the funders whose support makes this work possible. If you have any questions about this transition or any of the other work we do, please ask on our community forum.

We depend on contributions from our supporters in order to provide our services. If your company or organization can help our work by becoming a sponsor of Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

On Wednesday, March 13, 2024, Let’s Encrypt generated 10 new Intermediate CA Key Pairs, and issued 15 new Intermediate CA Certificates containing the new public keys. These new intermediate certificates provide smaller and more efficient certificate chains to Let’s Encrypt Subscribers, enhancing the overall online experience in terms of speed, security, and accessibility.

First, a bit of history. In September, 2020, Let’s Encrypt issued a new root and collection of intermediate certificates. Those certificates helped us improve the privacy and efficiency of Web security by making ECDSA end-entity certificates widely available. However, those intermediates are approaching their expiration dates, so it is time to replace them.

Our new batch of intermediates are very similar to the ones we issued in 2020, with a few small changes. We’re going to go over what those changes are and why we made them.

The New Certificates

We created 5 new 2048-bit RSA intermediate certificates named in sequence from R10 through R14. These are issued by ISRG Root X1. You can think of them as direct replacements for our existing R3 and R4 intermediates.

We also created 5 new P-384 ECDSA intermediate certificates named in sequence from E5 through E9. Each of these is represented by two certificates: one issued by ISRG Root X2 (exactly like our existing E1 and E2), and one issued (or cross-signed) by ISRG Root X1.

You can see details of all of the certificates on our updated hierarchy page.

Rotating Issuance

Rotating the set of intermediates we issue from helps keep the Internet agile and more secure. It encourages automation and efficiency, and discourages outdated practices like key pinning. “Key Pinning” is a practice in which clients — either ACME clients getting certificates for their site, or apps connecting to their own backend servers — decide to trust only a single issuing intermediate certificate rather than delegating trust to the system trust store. Updating pinned keys is a manual process, which leads to an increased risk of errors and potential business continuity failures.

Intermediates usually change only every five years, so this joint is exercised infrequently and client software keeps making the same mistakes. Shortening the lifetime from five years to three years means we will be conducting another ceremony in just two years, ahead of the expiration date on these recently created certificates. This ensures we exercise the joint more frequently than in the past.

We also issued more intermediates this time around. Historically, we’ve had two of each key type (RSA and ECDSA): one for active issuance, and one held as a backup for emergencies. Moving forward we will have five: two conducting active issuance, two waiting in the wings to be introduced in about one year, and one for emergency backup. Randomizing the selected issuer for a given key type means it will be impossible to predict which intermediate a certificate will be issued from. We are very hopeful that these steps will prevent intermediate key pinning altogether, and help the WebPKI remain agile moving forward.

These shorter intermediate lifetimes and randomized intermediate issuance shouldn’t impact the online experience of the general Internet user. Subscribers may be impacted if they are pinning one of our intermediates, though this should be incredibly rare.

Providing Smaller Chains

When we issued ISRG Root X2 in 2020, we decided to cross-sign it from ISRG Root X1 so that it would be trusted even by systems that didn’t yet have ISRG Root X2 in their trust store. This meant that Subscribers who wanted issuance from our ECDSA intermediates would have a choice: they could either have a very short, ECDSA-only, but low-compatibility chain terminating at ISRG Root X2, or they could have a longer, high-compatibility chain terminating at ISRG Root X1. At the time, this tradeoff (TLS handshake size vs compatibility) seemed like a reasonable choice to provide, and we provided the high-compatibility chain by default to support the largest number of configurations.

ISRG Root X2 is now trusted by most platforms, and we can now offer an improved version of the same choice. The same very short, ECDSA-only chain will still be available for Subscribers who want to optimize their TLS handshakes at the cost of some compatibility. But the high-compatibility chain will be drastically improving: instead of containing two intermediates (both E1 and the cross-signed ISRG Root X2), it will now contain only a single intermediate: the version of one of our new ECDSA intermediates cross-signed by ISRG Root X1.

This reduces the size of our default ECDSA chain by about a third, and is an important step towards removing our ECDSA allow-list.

Other Minor Changes

We’ve made two other tiny changes that are worth mentioning, but will have no impact on how Subscribers and clients use our certificates:

• We’ve changed how the Subject Key ID field is calculated, from a SHA-1 hash of the public key, to a truncated SHA-256 hash of the same data. Although this use of SHA-1 was not cryptographically relevant, it is still nice to remove one more usage of that broken algorithm, helping move towards a world where cryptography libraries don’t need to include SHA-1 support at all.

• We have removed our CPS OID from the Certificate Policies extension. This saves a few bytes in the certificate, which can add up to a lot of bandwidth saved over the course of billions of TLS handshakes.

Both of these mirror two identical changes that we made for our Subscriber Certificates in the past year.

Deployment

We intend to put two of each of the new RSA and ECDSA keys into rotation in the next few months. Two of each will be ready to swap in at a future date, and one of each will be held in reserve in case of an emergency. Read more about the strategy in our December 2023 post on the Community Forum.

Not familiar with the forum? It’s where Let’s Encrypt publishes updates on our Issuance Tech and APIs. It’s also where you can go for troubleshooting help from community experts and Let’s Encrypt staff. Check it out and subscribe to alerts for technical updates.

We hope that this has been an interesting and informative tour around our new intermediates, and we look forward to continuing to improve the Internet, one certificate at a time.

We depend on contributions from our community of users and supporters in order to provide our services. If your company or organization would like to sponsor Let’s Encrypt please email us at sponsor@letsencrypt.org. We ask that you make an individual contribution if it is within your means.

Let’s Encrypt is proud to introduce Sunlight, a new implementation of a Certificate Transparency log that we built from the ground up with modern Web PKI opportunities and constraints in mind. In partnership with Filippo Valsorda, who led the design and implementation, we incorporated feedback from the broader transparency logging community, including the Chrome and TrustFabric teams at Google, the Sigsum project, and other CT log and monitor operators. Their insights have been instrumental in shaping the project’s direction.

CT plays an important role in the Web PKI, enhancing the ability to monitor and research certificate issuance. The operation of a CT log, however, faces growing challenges with the increasing volume of certificates. For instance, Let’s Encrypt issues over four million certificates daily, each of which must be logged in two separate CT logs. Our well-established “Oak” log currently holds over 700 million entries, reflecting the significant scale of these challenges.

In this post, we’ll explore the motivation behind Sunlight and how its design aims to improve the robustness and diversity of the CT ecosystem, while also improving the reliability and performance of Let’s Encrypt’s logs.

Bottlenecks from the Database

Let’s Encrypt has been running public CT logs since 2019, and we’ve gotten a lot of operational experience with running them, but it hasn’t been trouble-free. The biggest challenge in the architecture we’ve deployed for our “Oak” log is that the data is stored in a relational database. We’ve scaled that up by splitting each year’s worth of data into a “shard” with its own database, and then later shrinking the shards to cover six months instead of a full year.

The approach of splitting into more and more databases is not something we want to continue doing forever, as the operational burden and costs increase. The current storage size of a CT log shard is between 5 and 10 terabytes. That’s big enough to be concerning for a single database: We previously had a test log fail when we ran into a 16TiB limit in MySQL.

Scaling read capacity up requires large database instances with fast disks and lots of RAM, which are not cheap. We’ve had numerous instances of CT logs becoming overloaded by clients attempting to read all the data in the log, overloading the database in the process. When rate limits are imposed to prevent overloading, clients are forced to slowly crawl the API, diminishing CT’s efficiency as a fast mechanism for detecting mis-issued certificates.

Serving Tiles

Initially, Let’s Encrypt only planned on building a new CT log implementation. However, our discussions with Filippo made us realize that other transparency systems had improved on the original Certificate Transparency design, and we could make our logs even more robust and scalable by changing the read path APIs. In particular, the Go Checksum Database is inspired by Certificate Transparency, but uses a more efficient format for publishing its data as a series of easily stored and cached tiles.

Certificate Transparency logs are a binary tree, with every node containing a hash of its two children. The “leaf” level contains the actual entries of the log: the certificates, appended to the right side of the tree. The top of the tree is digitally signed. This forms a cryptographically verifiable structure called a Merkle Tree, which can be used to check if a certificate is in the tree, and that the tree is append-only.

Sunlight tiles are files containing 256 elements each, either hashes at a certain tree “height” or certificates (or pre-certificates) at the leaf level. Russ Cox has a great explanation of how tiles work on his blog, or you can read the relevant section of the Sunlight specification. Even Trillian, the current implementation of CT we run, uses a subtree system similar to these tiles as its internal storage.

Unlike the dynamic endpoints in previous CT APIs, serving a tree as tiles doesn’t require any dynamic computation or request processing, so we can eliminate the need for API servers. Because the tiles are static, they’re efficiently cached, in contrast with CT APIs like get-proof-by-hash which have a different response for every certificate, so there’s no shared cache. The leaf tiles can also be stored compressed, saving even more storage!

The idea of exposing the log as a series of static tiles is motivated by our desire to scale out the read path horizontally and relatively inexpensively. We can directly expose tiles in cloud object storage like S3, use a caching CDN, or use a webserver and a filesystem.

Object or file storage is readily available, can scale up easily, and costs significantly less than databases from cloud providers. It seemed like the obvious path forward. In fact, we already have an S3-backed cache in front of our existing CT logs, which means we are currently storing our data twice.

Running More Logs

The tiles API improves the read path, but we also wanted to simplify our architecture on the write path. With Trillian, we run a collection of nodes along with etcd for leader election to choose which will handle writing. This is somewhat complex, and we believe the CT ecosystem allows a different tradeoff.

The key realization is that Certificate Transparency is already a distributed system, with clients submitting certificates to multiple logs, and gracefully failing over from any unavailable ones to the others. Each individual log’s write path doesn’t require a highly available leader election system. A simple single-node writer can meet the 99% Service Level Objective required by CT log programs.

The single-node Sunlight architecture lets us run multiple independent logs with the same amount of computing power. This increases the system’s overall robustness, even if each individual log has lower potential uptime. No more leader election needed. We use a simple compare-and-swap mechanism to store checkpoints and prevent accidentally running two instances at once, which could result in a forked tree, but that has much less overhead than leader election.

No More Merge Delay

One of the goals of CT was to have limited latency for submission to the logs. A design feature called Merge Delay was added to support that. When submitting a certificate to a log, the log can return a Signed Certificate Timestamp (SCT) immediately, with a promise to include it in the log within the log’s Maximum Merge Delay, conventionally 24 hours. While this seems like a good tradeoff to not slow down issuance, there have been multiple incidents and near-misses where a log stops operating with unmerged certificates, missing its maximum merge delay, and breaking that promise.

Sunlight takes a different approach, holding submissions while it batches and integrates certificates in the log, eliminating the merge delay. While this leads to a small latency increase, we think it’s worthwhile to avoid one of the more common CT log failure cases.

It also lets us embed the final leaf index in an extension of our SCTs, bringing CT a step closer to direct client verification of Merkle tree proofs. The extension also makes it possible for clients to fetch the proof of log inclusion from the new static tile-based APIs, without requiring server-side lookup tables or databases.

A Sunny Future

Today’s announcement of Sunlight is just the beginning. We’ve released software and a specification for Sunlight, and have Sunlight CT logs running. Head to sunlight.dev to find resources to get started. We encourage CAs to start test submitting to Let’s Encrypt’s new Sunlight CT logs, for CT Monitors and Auditors to add support for consuming Sunlight logs, and for the CT programs to consider trusting logs running on this new architecture. We hope Sunlight logs will be made usable for SCTs by the CT programs run by the browsers in the future, allowing CAs to rely on them to meet the browser CT logging requirements.

We’ve gotten positive feedback so far, with comments such as “Google’s TrustFabric team, maintainers of Trillian, are supportive of this direction and the Sunlight spec. We have been working towards the same goal of cacheable tile-based logs for other ecosystems with serverless tooling, and will be folding this into Trillian and ctfe, along with adding support for the Sunlight API.”

If you have feedback on the design, please join in the conversation on the ct-policy mailing list, or in the #sunlight channel on the transparency-dev Slack (invitation to join).

We’d like to thank Chrome for supporting the development of Sunlight, and Amazon Web Services for their ongoing support for our CT log operation. If your organization monitors or values CT, please consider a financial gift of support. Learn more at https://www.abetterinternet.org/sponsor/ or contact us at: sponsor@abetterinternet.org.

Thu, 14 Mar 2024 00:00:00 +0000

---

Social Media

• GitHub
• Google+
• LinkedIn
• Twitter
• Instagram