Let's Encrypt News

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

Have you ever needed to make sure your website has a broken certificate? While many tools exist to help run an HTTPS server with valid certificates, there aren’t tools to make sure your certificate is revoked or expired. This is not a problem most people have. Tools to help manage certificates are always focused on avoiding those problems, not creating them.

Let’s Encrypt is a Certificate Authority, and so we have unusual problems we need to solve.

One of the requirements for publicly trusted Certificate Authorities is to host websites with test certificates, some of which need to be revoked or expired. This gets messed up more than you might expect, but it’s a bit tricky to get right. Test certificate sites exist to allow developers to test their clients, so it’s important that they’re done right.

We’d previously used certbot, nginx, and some shell scripts, but the shell scripts were getting a bit too complicated. So we wrote a Go program tailored to the specific needs of a CA’s test certs site.

The websites

We need to host three sites per root certificate:

• A valid certificate, like any other website.
• An expired certificate, past its expiry date.
• A revoked certificate, but it can’t be expired.

Valid is easy enough; it’s the normal case of any other website. This is a solved problem.

Expired, too, is pretty easy. Issue one certificate, wait until it expires, and then you can use it forever. Not a normal feature, but so long as your webserver doesn’t get upset at it being expired, it’s easy to set up once and leave it.

Revoked, though, is where it’s easiest to slip up. You could fail to revoke a certificate and serve a perfectly valid one, or you could let your revoked certificate expire. Making sure your website is serving a non-expired but revoked certificate is not something any of the off-the-shelf tools support.

The ingredients to bake a cake

In order to implement our program, we need a few different ingredients to mix together.

First and foremost, we need to be able to get certificates. Because we’re writing this in Go, we’re using Lego as a library to request the certificates. Obtaining a certificate requires completing a domain validation challenge. We can hook Lego up to the Go webserver we’re using to complete TLS-ALPN-01 validation. We use that challenge type because it doesn’t require any more setup beyond exposing our webserver to the internet.

To get a revoked certificate, we request a certificate and then revoke it. That’s something we can do with Lego and ACME too: The account which issued a certificate can request it be revoked. We then need a way to check that the certificate is revoked. Certificates contain an HTTP URL pointing to the Certificate Revocation List (CRL) which we poll until our certificate’s serial number appears in it.

Let’s Encrypt implements the ACME standard, which defines how clients can get certificates. In general, we think ACME clients integrated into webservers are often the best way to get certificates for websites. They can automatically handle challenges, manage and reload certificates, and overall minimize the amount of work and reduce problems.

We also need a way to wait until a certificate is in the right state. The valid certificate is ready to use right away, but that’s not true for the revoked and expired certificates. The revoked certificate needs to wait at least until it appears in a CRL, which can be up to an hour. Expired certificates need to wait even longer: Even if we request the shortest-lived certificates we offer, that’s still six days. To handle this, our program stores a “next” certificate instead of immediately overwriting the current one. We wait at least 24 hours for the revoked certificate to make sure any CRL caches or push-based CRL infrastructure have time to process the revocation. The expired certificate has to wait until it passes its expiration date. After the program decides a certificate is ready, it replaces the current certificate and passes it off to the webserver. Normal ACME tools don’t support this because they can usually start using a certificate as soon as it’s obtained.

And finally, we need a webserver to host the certificates. We’re using Go, which has a great built-in TLS and HTTP serving stack we can use. The Go TLS server takes a GetCertificate callback function that decides what certificate to use for each new connection. We have all our certificates in-memory and select the right one to serve based on the request’s SNI. This function is also where we hook up Lego to serve the challenge certificates required for TLS-ALPN-01. Because we prioritize serving the correct certificate over uptime, we refuse to handle a connection if the corresponding certificate is expired (unless it should be expired!).

Visiting the sites

If you visit one of our revoked sites, you might not get an error message. Revocation checking in browsers varies pretty widely, and has historically not worked great. Today’s state-of-the-art is Firefox’s CRLite, which is efficient and reliable. Ubuntu is deploying upki, a Rustls project based on CRLite. We hope other browsers and operating systems follow suit. The upki project is a great example of a project making use of these revoked test certificates, too.

The actual content of the website isn’t terribly important: We just have a little HTML page explaining what the site is. But since this website is meant for testing clients, there’s more than just browsers connecting. In particular, it’s pretty routine that I try connecting with curl or some other terminal http client, and getting a bunch of HTML spewed to your terminal isn’t very nice.

As a small Easter egg, we added a plain text version of the website with an ASCII art version of our logo that we serve if your HTTP client doesn’t include text/html in its Accept HTTP header. You can pass a ?txt or ?html URL parameter to specifically request one or the other version of the content, if you just want to see the ASCII art.

Let’s Encrypt has four root certificates right now. Each of them has test sites linked both here and from our documentation.

The code

As with a lot of Let’s Encrypt, the code for this project is open-source. You can find it at https://github.com/letsencrypt/test-certs-site/. Other Certificate Authorities who need to run similar test certificate sites are welcome to use it. If you need any features that would make using our test certs site easier for your TLS/HTTPS client testing, please feel free to create an issue on that repository.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

Have you ever needed to make sure your website has a broken certificate? While many tools exist to help run an HTTPS server with valid certificates, there aren’t tools to make sure your certificate is revoked or expired. This is not a problem most people have. Tools to help manage certificates are always focused on avoiding those problems, not creating them.

Let’s Encrypt is a Certificate Authority, and so we have unusual problems we need to solve.

One of the requirements for publicly trusted Certificate Authorities is to host websites with test certificates, some of which need to be revoked or expired. This gets messed up more than you might expect, but it’s a bit tricky to get right. Test certificate sites exist to allow developers to test their clients, so it’s important that they’re done right.

We’d previously used certbot, nginx, and some shell scripts, but the shell scripts were getting a bit too complicated. So we wrote a Go program tailored to the specific needs of a CA’s test certs site.

The websites

We need to host three sites per root certificate:

• A valid certificate, like any other website.
• An expired certificate, past its expiry date.
• A revoked certificate, but it can’t be expired.

Valid is easy enough; it’s the normal case of any other website. This is a solved problem.

Expired, too, is pretty easy. Issue one certificate, wait until it expires, and then you can use it forever. Not a normal feature, but so long as your webserver doesn’t get upset at it being expired, it’s easy to set up once and leave it.

Revoked, though, is where it’s easiest to slip up. You could fail to revoke a certificate and serve a perfectly valid one, or you could let your revoked certificate expire. Making sure your website is serving a non-expired but revoked certificate is not something any of the off-the-shelf tools support.

The ingredients to bake a cake

In order to implement our program, we need a few different ingredients to mix together.

First and foremost, we need to be able to get certificates. Because we’re writing this in Go, we’re using Lego as a library to request the certificates. Obtaining a certificate requires completing a domain validation challenge. We can hook Lego up to the Go webserver we’re using to complete TLS-ALPN-01 validation. We use that challenge type because it doesn’t require any more setup beyond exposing our webserver to the internet.

To get a revoked certificate, we request a certificate and then revoke it. That’s something we can do with Lego and ACME too: The account which issued a certificate can request it be revoked. We then need a way to check that the certificate is revoked. Certificates contain an HTTP URL pointing to the Certificate Revocation List (CRL) which we poll until our certificate’s serial number appears in it.

Let’s Encrypt implements the ACME standard, which defines how clients can get certificates. In general, we think ACME clients integrated into webservers are often the best way to get certificates for websites. They can automatically handle challenges, manage and reload certificates, and overall minimize the amount of work and reduce problems.

We also need a way to wait until a certificate is in the right state. The valid certificate is ready to use right away, but that’s not true for the revoked and expired certificates. The revoked certificate needs to wait at least until it appears in a CRL, which can be up to an hour. Expired certificates need to wait even longer: Even if we request the shortest-lived certificates we offer, that’s still six days. To handle this, our program stores a “next” certificate instead of immediately overwriting the current one. We wait at least 24 hours for the revoked certificate to make sure any CRL caches or push-based CRL infrastructure have time to process the revocation. The expired certificate has to wait until it passes its expiration date. After the program decides a certificate is ready, it replaces the current certificate and passes it off to the webserver. Normal ACME tools don’t support this because they can usually start using a certificate as soon as it’s obtained.

And finally, we need a webserver to host the certificates. We’re using Go, which has a great built-in TLS and HTTP serving stack we can use. The Go TLS server takes a GetCertificate callback function that decides what certificate to use for each new connection. We have all our certificates in-memory and select the right one to serve based on the request’s SNI. This function is also where we hook up Lego to serve the challenge certificates required for TLS-ALPN-01. Because we prioritize serving the correct certificate over uptime, we refuse to handle a connection if the corresponding certificate is expired (unless it should be expired!).

Visiting the sites

If you visit one of our revoked sites, you might not get an error message. Revocation checking in browsers varies pretty widely, and has historically not worked great. Today’s state-of-the-art is Firefox’s CRLite, which is efficient and reliable. Ubuntu is deploying upki, a Rustls project based on CRLite. We hope other browsers and operating systems follow suit. The upki project is a great example of a project making use of these revoked test certificates, too.

The actual content of the website isn’t terribly important: We just have a little HTML page explaining what the site is. But since this website is meant for testing clients, there’s more than just browsers connecting. In particular, it’s pretty routine that I try connecting with curl or some other terminal http client, and getting a bunch of HTML spewed to your terminal isn’t very nice.

As a small Easter egg, we added a plain text version of the website with an ASCII art version of our logo that we serve if your HTTP client doesn’t include text/html in its Accept HTTP header. You can pass a ?txt or ?html URL parameter to specifically request one or the other version of the content, if you just want to see the ASCII art.

Let’s Encrypt has four root certificates right now. Each of them has test sites linked both here and from our documentation.

The code

As with a lot of Let’s Encrypt, the code for this project is open-source. You can find it at https://github.com/letsencrypt/test-certs-site/. Other Certificate Authorities who need to run similar test certificate sites are welcome to use it. If you need any features that would make using our test certs site easier for your TLS/HTTPS client testing, please feel free to create an issue on that repository.

Nick Silverman is a Senior Infrastructure Engineer on the Edge Infrastructure team at Shopify, where he maintains the systems that provision, renew, and publish SSL certificates for millions of merchants’ custom domains. He is also a contributor to the Ruby acme-client gem.

The challenge

Shopify’s automated certificate management system relied on a static renewal threshold: 30 days before the end of the 90-day lifetime. To spread the load of provisioning and renewing certificates, we implemented a random 0–72 hour delay for each. While this helps evenly distribute certificate management over time, it did not take into account the Certificate Authority’s (CA) load. It was also incapable of reacting to a dynamic renewal window based on information provided by the CA.

However, this approach needed greater resilience to solve what is, in the end, a distributed coordination problem. The weaknesses are:

• No rapid revocation response: The static logic is not aware of revocations at all.

• Brittleness to lifetime changes: The static 30-day threshold is not resilient to changes in certificate lifetime, such as Let’s Encrypt’s announced plan to move to 45-day certificates.

• Imperfect load distribution: Despite the random jitter, massive renewal bursts could still occur.

Shopify needed to develop a global coordination system to balance the load and handle regular and urgent renewals. Thankfully, Let’s Encrypt has led the charge on a solution for this and other very important aspects of the certificate lifecycle.

The journey

Let’s Encrypt and the Internet Engineering Task Force (IETF) published the ACME Renewal Information (ARI) standard which makes an endpoint available that provides a recommended window of time for the renewal to occur. The endpoint returns a payload that looks something like this:

GET /renewal-info/ACME_KEY_IDENTIFIER
{
  "suggestedWindow": {
    "start": "2026-02-03T04:00:00Z",
    "end": "2026-02-04T04:00:00Z"
  }
}

Shopify’s certificate management system uses the acme-client Ruby gem originally authored by another Shopify employee. A growing number of ACME clients, including certbot, have enabled support for ARI, but the Ruby gem did not yet support this feature. Rather than building a custom solution, we decided to enable support for the ARI extension directly in the client.

Let’s Encrypt’s guide to integrating ARI provided the necessary roadmap, and the implementation was completed with one PR. This contribution means that not only Shopify, but also the wider Ruby community, can benefit from the ARI extension.

Deployment and ARI at scale

Once we shipped the gem support, integrating ARI into our certificate management system was straightforward. Instead of checking a static 30-day threshold, we now query the ARI endpoint and use the suggested renewal window as the gate for initiating renewals. Those dates are stored alongside the certificate upon its initial provisioning.

The updated Ruby gem provides a method for fetching renewal information:

renewal_info = client.renewal_info(certificate: existing_certificate_pem)

This method generates an ARI certificate identifier that can be used when making the API call. The client also includes a helper method, suggested_renewal_time, which chooses a random time between the returned start and end dates. The certificate identifier can be passed to the new_order method via the replaces key, which can grant a higher priority or bypass rate limits for renewals occurring during the window, depending on the CA’s policies.

Critically, Shopify also regularly polls the ARI endpoint for updated renewal timestamps. This allows our systems to rely on those timestamps as the primary renewal timing logic and removes the need for inflexible hard-coded expiry thresholds. This becomes the mechanism that Let’s Encrypt uses to dynamically change the renewal time due to a revocation event.

Results and rewards

Since enabling the use of the ARI extension, our certificate management system has become significantly more robust. Shopify now delegates the responsibility of determining renewal timing to Let’s Encrypt. The ARI extension has proven to be an impactful infrastructure improvement and the benefits gained are immediate. These benefits, alongside fewer manual interventions, are the operational success story:

• Future-proofing: We gained resilience against any future certificate lifetime changes and mass revocation events without needing code updates—ensuring our renewal logic is flexible.

• Optimized load: We directly benefit from the CA’s coordinated load balancing provided by the suggested renewal window, eliminating local randomness issues and the need for complex global coordination.

• Revocation readiness: ARI allows systems to quickly detect and respond to revocation events when an urgent renewal is necessary, well before certificates get close to their due dates.

• Simple implementation: The extension is mature (RFC 9773) and the implementation is straightforward, providing simplified renewal logic and CA-optimized timing.

• Good citizenship: Anyone using ARI helps the CA optimize its infrastructure, and contributes to better aggregate behavior across the entire ecosystem.

If you’re still relying on static renewal thresholds, give ARI a look—Shopify wholeheartedly encourages all ACME users and client developers to adopt the ARI extension.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

Have you ever needed to make sure your website has a broken certificate? While many tools exist to help run an HTTPS server with valid certificates, there aren’t tools to make sure your certificate is revoked or expired. This is not a problem most people have. Tools to help manage certificates are always focused on avoiding those problems, not creating them.

Let’s Encrypt is a Certificate Authority, and so we have unusual problems we need to solve.

One of the requirements for publicly trusted Certificate Authorities is to host websites with test certificates, some of which need to be revoked or expired. This gets messed up more than you might expect, but it’s a bit tricky to get right. Test certificate sites exist to allow developers to test their clients, so it’s important that they’re done right.

We’d previously used certbot, nginx, and some shell scripts, but the shell scripts were getting a bit too complicated. So we wrote a Go program tailored to the specific needs of a CA’s test certs site.

The websites

We need to host three sites per root certificate:

• A valid certificate, like any other website.
• An expired certificate, past its expiry date.
• A revoked certificate, but it can’t be expired.

Valid is easy enough; it’s the normal case of any other website. This is a solved problem.

Expired, too, is pretty easy. Issue one certificate, wait until it expires, and then you can use it forever. Not a normal feature, but so long as your webserver doesn’t get upset at it being expired, it’s easy to set up once and leave it.

Revoked, though, is where it’s easiest to slip up. You could fail to revoke a certificate and serve a perfectly valid one, or you could let your revoked certificate expire. Making sure your website is serving a non-expired but revoked certificate is not something any of the off-the-shelf tools support.

The ingredients to bake a cake

In order to implement our program, we need a few different ingredients to mix together.

First and foremost, we need to be able to get certificates. Because we’re writing this in Go, we’re using Lego as a library to request the certificates. Obtaining a certificate requires completing a domain validation challenge. We can hook Lego up to the Go webserver we’re using to complete TLS-ALPN-01 validation. We use that challenge type because it doesn’t require any more setup beyond exposing our webserver to the internet.

To get a revoked certificate, we request a certificate and then revoke it. That’s something we can do with Lego and ACME too: The account which issued a certificate can request it be revoked. We then need a way to check that the certificate is revoked. Certificates contain an HTTP URL pointing to the Certificate Revocation List (CRL) which we poll until our certificate’s serial number appears in it.

Let’s Encrypt implements the ACME standard, which defines how clients can get certificates. In general, we think ACME clients integrated into webservers are often the best way to get certificates for websites. They can automatically handle challenges, manage and reload certificates, and overall minimize the amount of work and reduce problems.

We also need a way to wait until a certificate is in the right state. The valid certificate is ready to use right away, but that’s not true for the revoked and expired certificates. The revoked certificate needs to wait at least until it appears in a CRL, which can be up to an hour. Expired certificates need to wait even longer: Even if we request the shortest-lived certificates we offer, that’s still six days. To handle this, our program stores a “next” certificate instead of immediately overwriting the current one. We wait at least 24 hours for the revoked certificate to make sure any CRL caches or push-based CRL infrastructure have time to process the revocation. The expired certificate has to wait until it passes its expiration date. After the program decides a certificate is ready, it replaces the current certificate and passes it off to the webserver. Normal ACME tools don’t support this because they can usually start using a certificate as soon as it’s obtained.

And finally, we need a webserver to host the certificates. We’re using Go, which has a great built-in TLS and HTTP serving stack we can use. The Go TLS server takes a GetCertificate callback function that decides what certificate to use for each new connection. We have all our certificates in-memory and select the right one to serve based on the request’s SNI. This function is also where we hook up Lego to serve the challenge certificates required for TLS-ALPN-01. Because we prioritize serving the correct certificate over uptime, we refuse to handle a connection if the corresponding certificate is expired (unless it should be expired!).

Visiting the sites

If you visit one of our revoked sites, you might not get an error message. Revocation checking in browsers varies pretty widely, and has historically not worked great. Today’s state-of-the-art is Firefox’s CRLite, which is efficient and reliable. Ubuntu is deploying upki, a Rustls project based on CRLite. We hope other browsers and operating systems follow suit. The upki project is a great example of a project making use of these revoked test certificates, too.

The actual content of the website isn’t terribly important: We just have a little HTML page explaining what the site is. But since this website is meant for testing clients, there’s more than just browsers connecting. In particular, it’s pretty routine that I try connecting with curl or some other terminal http client, and getting a bunch of HTML spewed to your terminal isn’t very nice.

As a small Easter egg, we added a plain text version of the website with an ASCII art version of our logo that we serve if your HTTP client doesn’t include text/html in its Accept HTTP header. You can pass a ?txt or ?html URL parameter to specifically request one or the other version of the content, if you just want to see the ASCII art.

Let’s Encrypt has four root certificates right now. Each of them has test sites linked both here and from our documentation.

The code

As with a lot of Let’s Encrypt, the code for this project is open-source. You can find it at https://github.com/letsencrypt/test-certs-site/. Other Certificate Authorities who need to run similar test certificate sites are welcome to use it. If you need any features that would make using our test certs site easier for your TLS/HTTPS client testing, please feel free to create an issue on that repository.

Nick Silverman is a Senior Infrastructure Engineer on the Edge Infrastructure team at Shopify, where he maintains the systems that provision, renew, and publish SSL certificates for millions of merchants’ custom domains. He is also a contributor to the Ruby acme-client gem.

The challenge

Shopify’s automated certificate management system relied on a static renewal threshold: 30 days before the end of the 90-day lifetime. To spread the load of provisioning and renewing certificates, we implemented a random 0–72 hour delay for each. While this helps evenly distribute certificate management over time, it did not take into account the Certificate Authority’s (CA) load. It was also incapable of reacting to a dynamic renewal window based on information provided by the CA.

However, this approach needed greater resilience to solve what is, in the end, a distributed coordination problem. The weaknesses are:

• No rapid revocation response: The static logic is not aware of revocations at all.

• Brittleness to lifetime changes: The static 30-day threshold is not resilient to changes in certificate lifetime, such as Let’s Encrypt’s announced plan to move to 45-day certificates.

• Imperfect load distribution: Despite the random jitter, massive renewal bursts could still occur.

Shopify needed to develop a global coordination system to balance the load and handle regular and urgent renewals. Thankfully, Let’s Encrypt has led the charge on a solution for this and other very important aspects of the certificate lifecycle.

The journey

Let’s Encrypt and the Internet Engineering Task Force (IETF) published the ACME Renewal Information (ARI) standard which makes an endpoint available that provides a recommended window of time for the renewal to occur. The endpoint returns a payload that looks something like this:

GET /renewal-info/ACME_KEY_IDENTIFIER
{
  "suggestedWindow": {
    "start": "2026-02-03T04:00:00Z",
    "end": "2026-02-04T04:00:00Z"
  }
}

Shopify’s certificate management system uses the acme-client Ruby gem originally authored by another Shopify employee. A growing number of ACME clients, including certbot, have enabled support for ARI, but the Ruby gem did not yet support this feature. Rather than building a custom solution, we decided to enable support for the ARI extension directly in the client.

Let’s Encrypt’s guide to integrating ARI provided the necessary roadmap, and the implementation was completed with one PR. This contribution means that not only Shopify, but also the wider Ruby community, can benefit from the ARI extension.

Deployment and ARI at scale

Once we shipped the gem support, integrating ARI into our certificate management system was straightforward. Instead of checking a static 30-day threshold, we now query the ARI endpoint and use the suggested renewal window as the gate for initiating renewals. Those dates are stored alongside the certificate upon its initial provisioning.

The updated Ruby gem provides a method for fetching renewal information:

renewal_info = client.renewal_info(certificate: existing_certificate_pem)

This method generates an ARI certificate identifier that can be used when making the API call. The client also includes a helper method, suggested_renewal_time, which chooses a random time between the returned start and end dates. The certificate identifier can be passed to the new_order method via the replaces key, which can grant a higher priority or bypass rate limits for renewals occurring during the window, depending on the CA’s policies.

Critically, Shopify also regularly polls the ARI endpoint for updated renewal timestamps. This allows our systems to rely on those timestamps as the primary renewal timing logic and removes the need for inflexible hard-coded expiry thresholds. This becomes the mechanism that Let’s Encrypt uses to dynamically change the renewal time due to a revocation event.

Results and rewards

Since enabling the use of the ARI extension, our certificate management system has become significantly more robust. Shopify now delegates the responsibility of determining renewal timing to Let’s Encrypt. The ARI extension has proven to be an impactful infrastructure improvement and the benefits gained are immediate. These benefits, alongside fewer manual interventions, are the operational success story:

• Future-proofing: We gained resilience against any future certificate lifetime changes and mass revocation events without needing code updates—ensuring our renewal logic is flexible.

• Optimized load: We directly benefit from the CA’s coordinated load balancing provided by the suggested renewal window, eliminating local randomness issues and the need for complex global coordination.

• Revocation readiness: ARI allows systems to quickly detect and respond to revocation events when an urgent renewal is necessary, well before certificates get close to their due dates.

• Simple implementation: The extension is mature (RFC 9773) and the implementation is straightforward, providing simplified renewal logic and CA-optimized timing.

• Good citizenship: Anyone using ARI helps the CA optimize its infrastructure, and contributes to better aggregate behavior across the entire ecosystem.

If you’re still relying on static renewal thresholds, give ARI a look—Shopify wholeheartedly encourages all ACME users and client developers to adopt the ARI extension.

This was also posted on EFF’s blog.

As we announced earlier this year, Let’s Encrypt now issues IP address and six-day certificates to the general public. The Certbot team at the Electronic Frontier Foundation has been working on two improvements to support these features: the --preferred-profile flag released last year in Certbot 4.0, and the --ip-address flag, new in Certbot 5.3. With these improvements together, you can now use Certbot to get those IP address certificates!

If you want to try getting an IP address certificate using Certbot, install version 5.4 or higher (for webroot support with IP addresses), and run this command:

sudo certbot certonly --staging \
  --preferred-profile shortlived \
  --webroot \
  --webroot-path  \
  --ip-address 

Two things of note:

• This will request a non-trusted certificate from the Let’s Encrypt staging server. Once you’ve got things working the way you want, run without the --staging flag to get a publicly trusted certificate.

• This requests a certificate with Let’s Encrypt’s “shortlived” profile, which will be good for 6 days. This is a Let’s Encrypt requirement for IP address certificates.

As of right now, Certbot only supports getting IP address certificates, not yet installing them in your web server. There’s work to come on that front. In the meantime, edit your webserver configuration to load the newly issued certificate from /etc/letsencrypt/live//fullchain.pem and /etc/letsencrypt/live//privkey.pem.

The command line above uses Certbot’s “webroot” mode, which places a challenge response file in a location where your already-running webserver can serve it. This is nice since you don’t have to temporarily take down your server.

There are two other plugins that support IP address certificates today: --manual and --standalone. The manual plugin is like webroot, except Certbot pauses while you place the challenge response file manually (or runs a user-provided hook to place the file). The standalone plugin runs a simple web server that serves a challenge response. It has the advantage of being very easy to configure, but has the disadvantage that any running webserver on port 80 has to be temporarily taken down so Certbot can listen on that port. The nginx and apache plugins don’t yet support IP addresses.

You should also be sure that Certbot is set up for automatic renewal. Most installation methods for Certbot set up automatic renewal for you. However, since the webserver-specific installers don’t yet support IP address certificates, you’ll have to set a --deploy-hook that tells your webserver to load the most up-to-date certificates from disk. You can provide this --deploy-hook through the certbot reconfigure command using the rest of the flags above.

We hope you enjoy using IP address certificates with Let’s Encrypt and Certbot, and as always if you get stuck you can ask for help in our Community Forum.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

Have you ever needed to make sure your website has a broken certificate? While many tools exist to help run an HTTPS server with valid certificates, there aren’t tools to make sure your certificate is revoked or expired. This is not a problem most people have. Tools to help manage certificates are always focused on avoiding those problems, not creating them.

Let’s Encrypt is a Certificate Authority, and so we have unusual problems we need to solve.

One of the requirements for publicly trusted Certificate Authorities is to host websites with test certificates, some of which need to be revoked or expired. This gets messed up more than you might expect, but it’s a bit tricky to get right. Test certificate sites exist to allow developers to test their clients, so it’s important that they’re done right.

We’d previously used certbot, nginx, and some shell scripts, but the shell scripts were getting a bit too complicated. So we wrote a Go program tailored to the specific needs of a CA’s test certs site.

The websites

We need to host three sites per root certificate:

• A valid certificate, like any other website.
• An expired certificate, past its expiry date.
• A revoked certificate, but it can’t be expired.

Valid is easy enough; it’s the normal case of any other website. This is a solved problem.

Expired, too, is pretty easy. Issue one certificate, wait until it expires, and then you can use it forever. Not a normal feature, but so long as your webserver doesn’t get upset at it being expired, it’s easy to set up once and leave it.

Revoked, though, is where it’s easiest to slip up. You could fail to revoke a certificate and serve a perfectly valid one, or you could let your revoked certificate expire. Making sure your website is serving a non-expired but revoked certificate is not something any of the off-the-shelf tools support.

The ingredients to bake a cake

In order to implement our program, we need a few different ingredients to mix together.

First and foremost, we need to be able to get certificates. Because we’re writing this in Go, we’re using Lego as a library to request the certificates. Obtaining a certificate requires completing a domain validation challenge. We can hook Lego up to the Go webserver we’re using to complete TLS-ALPN-01 validation. We use that challenge type because it doesn’t require any more setup beyond exposing our webserver to the internet.

To get a revoked certificate, we request a certificate and then revoke it. That’s something we can do with Lego and ACME too: The account which issued a certificate can request it be revoked. We then need a way to check that the certificate is revoked. Certificates contain an HTTP URL pointing to the Certificate Revocation List (CRL) which we poll until our certificate’s serial number appears in it.

Let’s Encrypt implements the ACME standard, which defines how clients can get certificates. In general, we think ACME clients integrated into webservers are often the best way to get certificates for websites. They can automatically handle challenges, manage and reload certificates, and overall minimize the amount of work and reduce problems.

We also need a way to wait until a certificate is in the right state. The valid certificate is ready to use right away, but that’s not true for the revoked and expired certificates. The revoked certificate needs to wait at least until it appears in a CRL, which can be up to an hour. Expired certificates need to wait even longer: Even if we request the shortest-lived certificates we offer, that’s still six days. To handle this, our program stores a “next” certificate instead of immediately overwriting the current one. We wait at least 24 hours for the revoked certificate to make sure any CRL caches or push-based CRL infrastructure have time to process the revocation. The expired certificate has to wait until it passes its expiration date. After the program decides a certificate is ready, it replaces the current certificate and passes it off to the webserver. Normal ACME tools don’t support this because they can usually start using a certificate as soon as it’s obtained.

And finally, we need a webserver to host the certificates. We’re using Go, which has a great built-in TLS and HTTP serving stack we can use. The Go TLS server takes a GetCertificate callback function that decides what certificate to use for each new connection. We have all our certificates in-memory and select the right one to serve based on the request’s SNI. This function is also where we hook up Lego to serve the challenge certificates required for TLS-ALPN-01. Because we prioritize serving the correct certificate over uptime, we refuse to handle a connection if the corresponding certificate is expired (unless it should be expired!).

Visiting the sites

If you visit one of our revoked sites, you might not get an error message. Revocation checking in browsers varies pretty widely, and has historically not worked great. Today’s state-of-the-art is Firefox’s CRLite, which is efficient and reliable. Ubuntu is deploying upki, a Rustls project based on CRLite. We hope other browsers and operating systems follow suit. The upki project is a great example of a project making use of these revoked test certificates, too.

The actual content of the website isn’t terribly important: We just have a little HTML page explaining what the site is. But since this website is meant for testing clients, there’s more than just browsers connecting. In particular, it’s pretty routine that I try connecting with curl or some other terminal http client, and getting a bunch of HTML spewed to your terminal isn’t very nice.

As a small Easter egg, we added a plain text version of the website with an ASCII art version of our logo that we serve if your HTTP client doesn’t include text/html in its Accept HTTP header. You can pass a ?txt or ?html URL parameter to specifically request one or the other version of the content, if you just want to see the ASCII art.

Let’s Encrypt has four root certificates right now. Each of them has test sites linked both here and from our documentation.

The code

As with a lot of Let’s Encrypt, the code for this project is open-source. You can find it at https://github.com/letsencrypt/test-certs-site/. Other Certificate Authorities who need to run similar test certificate sites are welcome to use it. If you need any features that would make using our test certs site easier for your TLS/HTTPS client testing, please feel free to create an issue on that repository.

Nick Silverman is a Senior Infrastructure Engineer on the Edge Infrastructure team at Shopify, where he maintains the systems that provision, renew, and publish SSL certificates for millions of merchants’ custom domains. He is also a contributor to the Ruby acme-client gem.

The challenge

Shopify’s automated certificate management system relied on a static renewal threshold: 30 days before the end of the 90-day lifetime. To spread the load of provisioning and renewing certificates, we implemented a random 0–72 hour delay for each. While this helps evenly distribute certificate management over time, it did not take into account the Certificate Authority’s (CA) load. It was also incapable of reacting to a dynamic renewal window based on information provided by the CA.

However, this approach needed greater resilience to solve what is, in the end, a distributed coordination problem. The weaknesses are:

• No rapid revocation response: The static logic is not aware of revocations at all.

• Brittleness to lifetime changes: The static 30-day threshold is not resilient to changes in certificate lifetime, such as Let’s Encrypt’s announced plan to move to 45-day certificates.

• Imperfect load distribution: Despite the random jitter, massive renewal bursts could still occur.

Shopify needed to develop a global coordination system to balance the load and handle regular and urgent renewals. Thankfully, Let’s Encrypt has led the charge on a solution for this and other very important aspects of the certificate lifecycle.

The journey

Let’s Encrypt and the Internet Engineering Task Force (IETF) published the ACME Renewal Information (ARI) standard which makes an endpoint available that provides a recommended window of time for the renewal to occur. The endpoint returns a payload that looks something like this:

GET /renewal-info/ACME_KEY_IDENTIFIER
{
  "suggestedWindow": {
    "start": "2026-02-03T04:00:00Z",
    "end": "2026-02-04T04:00:00Z"
  }
}

Shopify’s certificate management system uses the acme-client Ruby gem originally authored by another Shopify employee. A growing number of ACME clients, including certbot, have enabled support for ARI, but the Ruby gem did not yet support this feature. Rather than building a custom solution, we decided to enable support for the ARI extension directly in the client.

Let’s Encrypt’s guide to integrating ARI provided the necessary roadmap, and the implementation was completed with one PR. This contribution means that not only Shopify, but also the wider Ruby community, can benefit from the ARI extension.

Deployment and ARI at scale

Once we shipped the gem support, integrating ARI into our certificate management system was straightforward. Instead of checking a static 30-day threshold, we now query the ARI endpoint and use the suggested renewal window as the gate for initiating renewals. Those dates are stored alongside the certificate upon its initial provisioning.

The updated Ruby gem provides a method for fetching renewal information:

renewal_info = client.renewal_info(certificate: existing_certificate_pem)

This method generates an ARI certificate identifier that can be used when making the API call. The client also includes a helper method, suggested_renewal_time, which chooses a random time between the returned start and end dates. The certificate identifier can be passed to the new_order method via the replaces key, which can grant a higher priority or bypass rate limits for renewals occurring during the window, depending on the CA’s policies.

Critically, Shopify also regularly polls the ARI endpoint for updated renewal timestamps. This allows our systems to rely on those timestamps as the primary renewal timing logic and removes the need for inflexible hard-coded expiry thresholds. This becomes the mechanism that Let’s Encrypt uses to dynamically change the renewal time due to a revocation event.

Results and rewards

Since enabling the use of the ARI extension, our certificate management system has become significantly more robust. Shopify now delegates the responsibility of determining renewal timing to Let’s Encrypt. The ARI extension has proven to be an impactful infrastructure improvement and the benefits gained are immediate. These benefits, alongside fewer manual interventions, are the operational success story:

• Future-proofing: We gained resilience against any future certificate lifetime changes and mass revocation events without needing code updates—ensuring our renewal logic is flexible.

• Optimized load: We directly benefit from the CA’s coordinated load balancing provided by the suggested renewal window, eliminating local randomness issues and the need for complex global coordination.

• Revocation readiness: ARI allows systems to quickly detect and respond to revocation events when an urgent renewal is necessary, well before certificates get close to their due dates.

• Simple implementation: The extension is mature (RFC 9773) and the implementation is straightforward, providing simplified renewal logic and CA-optimized timing.

• Good citizenship: Anyone using ARI helps the CA optimize its infrastructure, and contributes to better aggregate behavior across the entire ecosystem.

If you’re still relying on static renewal thresholds, give ARI a look—Shopify wholeheartedly encourages all ACME users and client developers to adopt the ARI extension.

This was also posted on EFF’s blog.

As we announced earlier this year, Let’s Encrypt now issues IP address and six-day certificates to the general public. The Certbot team at the Electronic Frontier Foundation has been working on two improvements to support these features: the --preferred-profile flag released last year in Certbot 4.0, and the --ip-address flag, new in Certbot 5.3. With these improvements together, you can now use Certbot to get those IP address certificates!

If you want to try getting an IP address certificate using Certbot, install version 5.4 or higher (for webroot support with IP addresses), and run this command:

sudo certbot certonly --staging \
  --preferred-profile shortlived \
  --webroot \
  --webroot-path  \
  --ip-address 

Two things of note:

• This will request a non-trusted certificate from the Let’s Encrypt staging server. Once you’ve got things working the way you want, run without the --staging flag to get a publicly trusted certificate.

• This requests a certificate with Let’s Encrypt’s “shortlived” profile, which will be good for 6 days. This is a Let’s Encrypt requirement for IP address certificates.

As of right now, Certbot only supports getting IP address certificates, not yet installing them in your web server. There’s work to come on that front. In the meantime, edit your webserver configuration to load the newly issued certificate from /etc/letsencrypt/live//fullchain.pem and /etc/letsencrypt/live//privkey.pem.

The command line above uses Certbot’s “webroot” mode, which places a challenge response file in a location where your already-running webserver can serve it. This is nice since you don’t have to temporarily take down your server.

There are two other plugins that support IP address certificates today: --manual and --standalone. The manual plugin is like webroot, except Certbot pauses while you place the challenge response file manually (or runs a user-provided hook to place the file). The standalone plugin runs a simple web server that serves a challenge response. It has the advantage of being very easy to configure, but has the disadvantage that any running webserver on port 80 has to be temporarily taken down so Certbot can listen on that port. The nginx and apache plugins don’t yet support IP addresses.

You should also be sure that Certbot is set up for automatic renewal. Most installation methods for Certbot set up automatic renewal for you. However, since the webserver-specific installers don’t yet support IP address certificates, you’ll have to set a --deploy-hook that tells your webserver to load the most up-to-date certificates from disk. You can provide this --deploy-hook through the certbot reconfigure command using the rest of the flags above.

We hope you enjoy using IP address certificates with Let’s Encrypt and Certbot, and as always if you get stuck you can ask for help in our Community Forum.

As previously announced, over the next two years we will be switching the default certificate lifetime from 90 days to 64 days, and then 45 days. This will ultimately double the number of certificate renewal requests each day: today we expect renewal around day 60 (of a 90-day certificate), while in the future we expect renewal around day 30 (of a 45-day certificate). If you use an ACME client that supports ARI, this will happen automatically.

The good news for subscribers is that you don’t need any changes to your rate limits, whether you are using our default limits or have requested an override. Our rate limits affect issuance for new domain names (or groups of domain names), but renewals are exempt. So, for instance, if you are managing a set of 15,000 certificates that you continually renew, and create 250 new certificates (with new domain names) each day, you will be well within our limits both before and after the transition. The 250 new certificates daily will still be well under our New Orders per Account limit of 300 per three hours. And the 15,000 existing certificates will continue to be unaffected by rate limits, whether your ACME client is renewing them every sixty days or every thirty.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

Have you ever needed to make sure your website has a broken certificate? While many tools exist to help run an HTTPS server with valid certificates, there aren’t tools to make sure your certificate is revoked or expired. This is not a problem most people have. Tools to help manage certificates are always focused on avoiding those problems, not creating them.

Let’s Encrypt is a Certificate Authority, and so we have unusual problems we need to solve.

One of the requirements for publicly trusted Certificate Authorities is to host websites with test certificates, some of which need to be revoked or expired. This gets messed up more than you might expect, but it’s a bit tricky to get right. Test certificate sites exist to allow developers to test their clients, so it’s important that they’re done right.

We’d previously used certbot, nginx, and some shell scripts, but the shell scripts were getting a bit too complicated. So we wrote a Go program tailored to the specific needs of a CA’s test certs site.

The websites

We need to host three sites per root certificate:

• A valid certificate, like any other website.
• An expired certificate, past its expiry date.
• A revoked certificate, but it can’t be expired.

Valid is easy enough; it’s the normal case of any other website. This is a solved problem.

Expired, too, is pretty easy. Issue one certificate, wait until it expires, and then you can use it forever. Not a normal feature, but so long as your webserver doesn’t get upset at it being expired, it’s easy to set up once and leave it.

Revoked, though, is where it’s easiest to slip up. You could fail to revoke a certificate and serve a perfectly valid one, or you could let your revoked certificate expire. Making sure your website is serving a non-expired but revoked certificate is not something any of the off-the-shelf tools support.

The ingredients to bake a cake

In order to implement our program, we need a few different ingredients to mix together.

First and foremost, we need to be able to get certificates. Because we’re writing this in Go, we’re using Lego as a library to request the certificates. Obtaining a certificate requires completing a domain validation challenge. We can hook Lego up to the Go webserver we’re using to complete TLS-ALPN-01 validation. We use that challenge type because it doesn’t require any more setup beyond exposing our webserver to the internet.

To get a revoked certificate, we request a certificate and then revoke it. That’s something we can do with Lego and ACME too: The account which issued a certificate can request it be revoked. We then need a way to check that the certificate is revoked. Certificates contain an HTTP URL pointing to the Certificate Revocation List (CRL) which we poll until our certificate’s serial number appears in it.

Let’s Encrypt implements the ACME standard, which defines how clients can get certificates. In general, we think ACME clients integrated into webservers are often the best way to get certificates for websites. They can automatically handle challenges, manage and reload certificates, and overall minimize the amount of work and reduce problems.

We also need a way to wait until a certificate is in the right state. The valid certificate is ready to use right away, but that’s not true for the revoked and expired certificates. The revoked certificate needs to wait at least until it appears in a CRL, which can be up to an hour. Expired certificates need to wait even longer: Even if we request the shortest-lived certificates we offer, that’s still six days. To handle this, our program stores a “next” certificate instead of immediately overwriting the current one. We wait at least 24 hours for the revoked certificate to make sure any CRL caches or push-based CRL infrastructure have time to process the revocation. The expired certificate has to wait until it passes its expiration date. After the program decides a certificate is ready, it replaces the current certificate and passes it off to the webserver. Normal ACME tools don’t support this because they can usually start using a certificate as soon as it’s obtained.

And finally, we need a webserver to host the certificates. We’re using Go, which has a great built-in TLS and HTTP serving stack we can use. The Go TLS server takes a GetCertificate callback function that decides what certificate to use for each new connection. We have all our certificates in-memory and select the right one to serve based on the request’s SNI. This function is also where we hook up Lego to serve the challenge certificates required for TLS-ALPN-01. Because we prioritize serving the correct certificate over uptime, we refuse to handle a connection if the corresponding certificate is expired (unless it should be expired!).

Visiting the sites

If you visit one of our revoked sites, you might not get an error message. Revocation checking in browsers varies pretty widely, and has historically not worked great. Today’s state-of-the-art is Firefox’s CRLite, which is efficient and reliable. Ubuntu is deploying upki, a Rustls project based on CRLite. We hope other browsers and operating systems follow suit. The upki project is a great example of a project making use of these revoked test certificates, too.

The actual content of the website isn’t terribly important: We just have a little HTML page explaining what the site is. But since this website is meant for testing clients, there’s more than just browsers connecting. In particular, it’s pretty routine that I try connecting with curl or some other terminal http client, and getting a bunch of HTML spewed to your terminal isn’t very nice.

As a small Easter egg, we added a plain text version of the website with an ASCII art version of our logo that we serve if your HTTP client doesn’t include text/html in its Accept HTTP header. You can pass a ?txt or ?html URL parameter to specifically request one or the other version of the content, if you just want to see the ASCII art.

Let’s Encrypt has four root certificates right now. Each of them has test sites linked both here and from our documentation.

The code

As with a lot of Let’s Encrypt, the code for this project is open-source. You can find it at https://github.com/letsencrypt/test-certs-site/. Other Certificate Authorities who need to run similar test certificate sites are welcome to use it. If you need any features that would make using our test certs site easier for your TLS/HTTPS client testing, please feel free to create an issue on that repository.

Nick Silverman is a Senior Infrastructure Engineer on the Edge Infrastructure team at Shopify, where he maintains the systems that provision, renew, and publish SSL certificates for millions of merchants’ custom domains. He is also a contributor to the Ruby acme-client gem.

The challenge

Shopify’s automated certificate management system relied on a static renewal threshold: 30 days before the end of the 90-day lifetime. To spread the load of provisioning and renewing certificates, we implemented a random 0–72 hour delay for each. While this helps evenly distribute certificate management over time, it did not take into account the Certificate Authority’s (CA) load. It was also incapable of reacting to a dynamic renewal window based on information provided by the CA.

However, this approach needed greater resilience to solve what is, in the end, a distributed coordination problem. The weaknesses are:

• No rapid revocation response: The static logic is not aware of revocations at all.

• Brittleness to lifetime changes: The static 30-day threshold is not resilient to changes in certificate lifetime, such as Let’s Encrypt’s announced plan to move to 45-day certificates.

• Imperfect load distribution: Despite the random jitter, massive renewal bursts could still occur.

Shopify needed to develop a global coordination system to balance the load and handle regular and urgent renewals. Thankfully, Let’s Encrypt has led the charge on a solution for this and other very important aspects of the certificate lifecycle.

The journey

Let’s Encrypt and the Internet Engineering Task Force (IETF) published the ACME Renewal Information (ARI) standard which makes an endpoint available that provides a recommended window of time for the renewal to occur. The endpoint returns a payload that looks something like this:

GET /renewal-info/ACME_KEY_IDENTIFIER
{
  "suggestedWindow": {
    "start": "2026-02-03T04:00:00Z",
    "end": "2026-02-04T04:00:00Z"
  }
}

Shopify’s certificate management system uses the acme-client Ruby gem originally authored by another Shopify employee. A growing number of ACME clients, including certbot, have enabled support for ARI, but the Ruby gem did not yet support this feature. Rather than building a custom solution, we decided to enable support for the ARI extension directly in the client.

Let’s Encrypt’s guide to integrating ARI provided the necessary roadmap, and the implementation was completed with one PR. This contribution means that not only Shopify, but also the wider Ruby community, can benefit from the ARI extension.

Deployment and ARI at scale

Once we shipped the gem support, integrating ARI into our certificate management system was straightforward. Instead of checking a static 30-day threshold, we now query the ARI endpoint and use the suggested renewal window as the gate for initiating renewals. Those dates are stored alongside the certificate upon its initial provisioning.

The updated Ruby gem provides a method for fetching renewal information:

renewal_info = client.renewal_info(certificate: existing_certificate_pem)

This method generates an ARI certificate identifier that can be used when making the API call. The client also includes a helper method, suggested_renewal_time, which chooses a random time between the returned start and end dates. The certificate identifier can be passed to the new_order method via the replaces key, which can grant a higher priority or bypass rate limits for renewals occurring during the window, depending on the CA’s policies.

Critically, Shopify also regularly polls the ARI endpoint for updated renewal timestamps. This allows our systems to rely on those timestamps as the primary renewal timing logic and removes the need for inflexible hard-coded expiry thresholds. This becomes the mechanism that Let’s Encrypt uses to dynamically change the renewal time due to a revocation event.

Results and rewards

Since enabling the use of the ARI extension, our certificate management system has become significantly more robust. Shopify now delegates the responsibility of determining renewal timing to Let’s Encrypt. The ARI extension has proven to be an impactful infrastructure improvement and the benefits gained are immediate. These benefits, alongside fewer manual interventions, are the operational success story:

• Future-proofing: We gained resilience against any future certificate lifetime changes and mass revocation events without needing code updates—ensuring our renewal logic is flexible.

• Optimized load: We directly benefit from the CA’s coordinated load balancing provided by the suggested renewal window, eliminating local randomness issues and the need for complex global coordination.

• Revocation readiness: ARI allows systems to quickly detect and respond to revocation events when an urgent renewal is necessary, well before certificates get close to their due dates.

• Simple implementation: The extension is mature (RFC 9773) and the implementation is straightforward, providing simplified renewal logic and CA-optimized timing.

• Good citizenship: Anyone using ARI helps the CA optimize its infrastructure, and contributes to better aggregate behavior across the entire ecosystem.

If you’re still relying on static renewal thresholds, give ARI a look—Shopify wholeheartedly encourages all ACME users and client developers to adopt the ARI extension.

This was also posted on EFF’s blog.

As we announced earlier this year, Let’s Encrypt now issues IP address and six-day certificates to the general public. The Certbot team at the Electronic Frontier Foundation has been working on two improvements to support these features: the --preferred-profile flag released last year in Certbot 4.0, and the --ip-address flag, new in Certbot 5.3. With these improvements together, you can now use Certbot to get those IP address certificates!

If you want to try getting an IP address certificate using Certbot, install version 5.4 or higher (for webroot support with IP addresses), and run this command:

sudo certbot certonly --staging \
  --preferred-profile shortlived \
  --webroot \
  --webroot-path  \
  --ip-address 

Two things of note:

• This will request a non-trusted certificate from the Let’s Encrypt staging server. Once you’ve got things working the way you want, run without the --staging flag to get a publicly trusted certificate.

• This requests a certificate with Let’s Encrypt’s “shortlived” profile, which will be good for 6 days. This is a Let’s Encrypt requirement for IP address certificates.

As of right now, Certbot only supports getting IP address certificates, not yet installing them in your web server. There’s work to come on that front. In the meantime, edit your webserver configuration to load the newly issued certificate from /etc/letsencrypt/live//fullchain.pem and /etc/letsencrypt/live//privkey.pem.

The command line above uses Certbot’s “webroot” mode, which places a challenge response file in a location where your already-running webserver can serve it. This is nice since you don’t have to temporarily take down your server.

There are two other plugins that support IP address certificates today: --manual and --standalone. The manual plugin is like webroot, except Certbot pauses while you place the challenge response file manually (or runs a user-provided hook to place the file). The standalone plugin runs a simple web server that serves a challenge response. It has the advantage of being very easy to configure, but has the disadvantage that any running webserver on port 80 has to be temporarily taken down so Certbot can listen on that port. The nginx and apache plugins don’t yet support IP addresses.

You should also be sure that Certbot is set up for automatic renewal. Most installation methods for Certbot set up automatic renewal for you. However, since the webserver-specific installers don’t yet support IP address certificates, you’ll have to set a --deploy-hook that tells your webserver to load the most up-to-date certificates from disk. You can provide this --deploy-hook through the certbot reconfigure command using the rest of the flags above.

We hope you enjoy using IP address certificates with Let’s Encrypt and Certbot, and as always if you get stuck you can ask for help in our Community Forum.

As previously announced, over the next two years we will be switching the default certificate lifetime from 90 days to 64 days, and then 45 days. This will ultimately double the number of certificate renewal requests each day: today we expect renewal around day 60 (of a 90-day certificate), while in the future we expect renewal around day 30 (of a 45-day certificate). If you use an ACME client that supports ARI, this will happen automatically.

The good news for subscribers is that you don’t need any changes to your rate limits, whether you are using our default limits or have requested an override. Our rate limits affect issuance for new domain names (or groups of domain names), but renewals are exempt. So, for instance, if you are managing a set of 15,000 certificates that you continually renew, and create 250 new certificates (with new domain names) each day, you will be well within our limits both before and after the transition. The 250 new certificates daily will still be well under our New Orders per Account limit of 300 per three hours. And the 15,000 existing certificates will continue to be unaffected by rate limits, whether your ACME client is renewing them every sixty days or every thirty.

When you request a certificate from Let’s Encrypt, our servers validate that you control the hostnames in that certificate using ACME challenges. For subscribers who need wildcard certificates or who prefer not to expose infrastructure to the public Internet, the DNS-01 challenge type has long been the only choice. DNS-01 works well. It is widely supported and battle-tested, but it comes with operational costs: DNS propagation delays, recurring DNS updates at renewal time, and automation that often requires distributing DNS credentials throughout your infrastructure.

We are implementing support for a new ACME challenge type, DNS-PERSIST-01, based on a new IETF draft specification. As the name implies, it uses DNS as the validation mechanism, but replaces repeated demonstrations of control with a persistent authorization record bound to a specific ACME account and CA. The draft describes this method as being “particularly suited for environments where traditional challenge methods are impractical, such as IoT deployments, multi-tenant platforms, and scenarios requiring batch certificate operations”.

DNS-01 Proves Control Repeatedly

With DNS-01, validation relies on a one-time token generated by us. Your ACME client publishes a TXT record containing that token at _acme-challenge., and we query DNS to confirm that it matches the expected value. Because each authorization requires a new token, DNS updates become part of the issuance workflow. The benefit is that each successful validation provides fresh proof that you currently control DNS for the name being issued.

In practice, this often means DNS API credentials live somewhere in your issuance pipeline, validation attempts involve waiting for DNS propagation, and DNS changes happen frequently — sometimes many times per day in large deployments. Many subscribers accept these tradeoffs, but others would prefer to keep DNS updates and sensitive credentials out of their issuance path.

DNS-PERSIST-01 Authorizes Persistently

DNS-PERSIST-01 approaches validation differently. Instead of publishing a new challenge record for each issuance, you publish a standing authorization in the form of a TXT record that identifies both the CA and the specific ACME account you authorize to issue for this domain.

For the hostname example.com, the record would live at _validation-persist.example.com:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890"
)

Once this record exists, it can be reused for new issuance and all subsequent renewals. Operationally, this removes DNS changes from the critical path.

Security and Operational Tradeoffs

With DNS-01, the sensitive asset is DNS write access. In many deployments, DNS API credentials are distributed throughout issuance and renewal pipelines, increasing the number of places an attacker might compromise them. DNS-PERSIST-01 instead binds authorization directly to an ACME account, allowing DNS write access to remain more tightly controlled after initial setup. The tradeoff is that, because the authorization record persists over time, protecting the ACME account key becomes the central concern.

Controlling Scope and Lifetime

DNS-PERSIST-01 also introduces explicit scope controls. Without additional parameters, authorization applies only to the validated Fully Qualified Domain Name (FQDN) and remains valid indefinitely.

Wildcard Certificates

Adding policy=wildcard broadens the authorization scope to include the validated FQDN, wildcard certificates such as *.example.com, and subdomains whose suffix matches the validated FQDN:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " policy=wildcard"
)

Optional Expiration

Subscribers who aren’t comfortable with authorization persisting indefinitely can include an optional persistUntil timestamp. This limits how long the record may be used for new validations, but also means it must be updated or replaced before it expires. Anyone using this feature should ensure they have adequate reminders or monitoring in place so that authorization does not expire unexpectedly. The timestamp is expressed as UTC seconds since 1970-01-01:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " persistUntil=1767225600"
)

Authorizing Multiple CAs

Multiple CAs can be simultaneously authorized by publishing multiple TXT records at _validation-persist., each containing the issuer-domain-name of the CA you intend to authorize. During validation, each CA queries the same DNS label and evaluates only the records that match its own issuer-domain-name.

Rollout Timeline

The CA/Browser Forum ballot SC-088v3, defining “3.2.2.4.22 DNS TXT Record with Persistent Value”, passed unanimously in October 2025, and the IETF ACME working group adopted the draft that same month. While the document remains an active IETF draft, the core mechanisms described here are not expected to change substantially.

Support for the draft specification is available now in Pebble, a miniature version of Boulder, our production CA software. Work is also in progress on a lego-cli client implementation to make it easier for subscribers to experiment with and adopt. Staging rollout is planned for late Q1 2026, with a production rollout targeted for some time in Q2 2026.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

Have you ever needed to make sure your website has a broken certificate? While many tools exist to help run an HTTPS server with valid certificates, there aren’t tools to make sure your certificate is revoked or expired. This is not a problem most people have. Tools to help manage certificates are always focused on avoiding those problems, not creating them.

Let’s Encrypt is a Certificate Authority, and so we have unusual problems we need to solve.

One of the requirements for publicly trusted Certificate Authorities is to host websites with test certificates, some of which need to be revoked or expired. This gets messed up more than you might expect, but it’s a bit tricky to get right. Test certificate sites exist to allow developers to test their clients, so it’s important that they’re done right.

We’d previously used certbot, nginx, and some shell scripts, but the shell scripts were getting a bit too complicated. So we wrote a Go program tailored to the specific needs of a CA’s test certs site.

The websites

We need to host three sites per root certificate:

• A valid certificate, like any other website.
• An expired certificate, past its expiry date.
• A revoked certificate, but it can’t be expired.

Valid is easy enough; it’s the normal case of any other website. This is a solved problem.

Expired, too, is pretty easy. Issue one certificate, wait until it expires, and then you can use it forever. Not a normal feature, but so long as your webserver doesn’t get upset at it being expired, it’s easy to set up once and leave it.

Revoked, though, is where it’s easiest to slip up. You could fail to revoke a certificate and serve a perfectly valid one, or you could let your revoked certificate expire. Making sure your website is serving a non-expired but revoked certificate is not something any of the off-the-shelf tools support.

The ingredients to bake a cake

In order to implement our program, we need a few different ingredients to mix together.

First and foremost, we need to be able to get certificates. Because we’re writing this in Go, we’re using Lego as a library to request the certificates. Obtaining a certificate requires completing a domain validation challenge. We can hook Lego up to the Go webserver we’re using to complete TLS-ALPN-01 validation. We use that challenge type because it doesn’t require any more setup beyond exposing our webserver to the internet.

To get a revoked certificate, we request a certificate and then revoke it. That’s something we can do with Lego and ACME too: The account which issued a certificate can request it be revoked. We then need a way to check that the certificate is revoked. Certificates contain an HTTP URL pointing to the Certificate Revocation List (CRL) which we poll until our certificate’s serial number appears in it.

Let’s Encrypt implements the ACME standard, which defines how clients can get certificates. In general, we think ACME clients integrated into webservers are often the best way to get certificates for websites. They can automatically handle challenges, manage and reload certificates, and overall minimize the amount of work and reduce problems.

We also need a way to wait until a certificate is in the right state. The valid certificate is ready to use right away, but that’s not true for the revoked and expired certificates. The revoked certificate needs to wait at least until it appears in a CRL, which can be up to an hour. Expired certificates need to wait even longer: Even if we request the shortest-lived certificates we offer, that’s still six days. To handle this, our program stores a “next” certificate instead of immediately overwriting the current one. We wait at least 24 hours for the revoked certificate to make sure any CRL caches or push-based CRL infrastructure have time to process the revocation. The expired certificate has to wait until it passes its expiration date. After the program decides a certificate is ready, it replaces the current certificate and passes it off to the webserver. Normal ACME tools don’t support this because they can usually start using a certificate as soon as it’s obtained.

And finally, we need a webserver to host the certificates. We’re using Go, which has a great built-in TLS and HTTP serving stack we can use. The Go TLS server takes a GetCertificate callback function that decides what certificate to use for each new connection. We have all our certificates in-memory and select the right one to serve based on the request’s SNI. This function is also where we hook up Lego to serve the challenge certificates required for TLS-ALPN-01. Because we prioritize serving the correct certificate over uptime, we refuse to handle a connection if the corresponding certificate is expired (unless it should be expired!).

Visiting the sites

If you visit one of our revoked sites, you might not get an error message. Revocation checking in browsers varies pretty widely, and has historically not worked great. Today’s state-of-the-art is Firefox’s CRLite, which is efficient and reliable. Ubuntu is deploying upki, a Rustls project based on CRLite. We hope other browsers and operating systems follow suit. The upki project is a great example of a project making use of these revoked test certificates, too.

The actual content of the website isn’t terribly important: We just have a little HTML page explaining what the site is. But since this website is meant for testing clients, there’s more than just browsers connecting. In particular, it’s pretty routine that I try connecting with curl or some other terminal http client, and getting a bunch of HTML spewed to your terminal isn’t very nice.

As a small Easter egg, we added a plain text version of the website with an ASCII art version of our logo that we serve if your HTTP client doesn’t include text/html in its Accept HTTP header. You can pass a ?txt or ?html URL parameter to specifically request one or the other version of the content, if you just want to see the ASCII art.

Let’s Encrypt has four root certificates right now. Each of them has test sites linked both here and from our documentation.

The code

As with a lot of Let’s Encrypt, the code for this project is open-source. You can find it at https://github.com/letsencrypt/test-certs-site/. Other Certificate Authorities who need to run similar test certificate sites are welcome to use it. If you need any features that would make using our test certs site easier for your TLS/HTTPS client testing, please feel free to create an issue on that repository.

Nick Silverman is a Senior Infrastructure Engineer on the Edge Infrastructure team at Shopify, where he maintains the systems that provision, renew, and publish SSL certificates for millions of merchants’ custom domains. He is also a contributor to the Ruby acme-client gem.

The challenge

Shopify’s automated certificate management system relied on a static renewal threshold: 30 days before the end of the 90-day lifetime. To spread the load of provisioning and renewing certificates, we implemented a random 0–72 hour delay for each. While this helps evenly distribute certificate management over time, it did not take into account the Certificate Authority’s (CA) load. It was also incapable of reacting to a dynamic renewal window based on information provided by the CA.

However, this approach needed greater resilience to solve what is, in the end, a distributed coordination problem. The weaknesses are:

• No rapid revocation response: The static logic is not aware of revocations at all.

• Brittleness to lifetime changes: The static 30-day threshold is not resilient to changes in certificate lifetime, such as Let’s Encrypt’s announced plan to move to 45-day certificates.

• Imperfect load distribution: Despite the random jitter, massive renewal bursts could still occur.

Shopify needed to develop a global coordination system to balance the load and handle regular and urgent renewals. Thankfully, Let’s Encrypt has led the charge on a solution for this and other very important aspects of the certificate lifecycle.

The journey

Let’s Encrypt and the Internet Engineering Task Force (IETF) published the ACME Renewal Information (ARI) standard which makes an endpoint available that provides a recommended window of time for the renewal to occur. The endpoint returns a payload that looks something like this:

GET /renewal-info/ACME_KEY_IDENTIFIER
{
  "suggestedWindow": {
    "start": "2026-02-03T04:00:00Z",
    "end": "2026-02-04T04:00:00Z"
  }
}

Shopify’s certificate management system uses the acme-client Ruby gem originally authored by another Shopify employee. A growing number of ACME clients, including certbot, have enabled support for ARI, but the Ruby gem did not yet support this feature. Rather than building a custom solution, we decided to enable support for the ARI extension directly in the client.

Let’s Encrypt’s guide to integrating ARI provided the necessary roadmap, and the implementation was completed with one PR. This contribution means that not only Shopify, but also the wider Ruby community, can benefit from the ARI extension.

Deployment and ARI at scale

Once we shipped the gem support, integrating ARI into our certificate management system was straightforward. Instead of checking a static 30-day threshold, we now query the ARI endpoint and use the suggested renewal window as the gate for initiating renewals. Those dates are stored alongside the certificate upon its initial provisioning.

The updated Ruby gem provides a method for fetching renewal information:

renewal_info = client.renewal_info(certificate: existing_certificate_pem)

This method generates an ARI certificate identifier that can be used when making the API call. The client also includes a helper method, suggested_renewal_time, which chooses a random time between the returned start and end dates. The certificate identifier can be passed to the new_order method via the replaces key, which can grant a higher priority or bypass rate limits for renewals occurring during the window, depending on the CA’s policies.

Critically, Shopify also regularly polls the ARI endpoint for updated renewal timestamps. This allows our systems to rely on those timestamps as the primary renewal timing logic and removes the need for inflexible hard-coded expiry thresholds. This becomes the mechanism that Let’s Encrypt uses to dynamically change the renewal time due to a revocation event.

Results and rewards

Since enabling the use of the ARI extension, our certificate management system has become significantly more robust. Shopify now delegates the responsibility of determining renewal timing to Let’s Encrypt. The ARI extension has proven to be an impactful infrastructure improvement and the benefits gained are immediate. These benefits, alongside fewer manual interventions, are the operational success story:

• Future-proofing: We gained resilience against any future certificate lifetime changes and mass revocation events without needing code updates—ensuring our renewal logic is flexible.

• Optimized load: We directly benefit from the CA’s coordinated load balancing provided by the suggested renewal window, eliminating local randomness issues and the need for complex global coordination.

• Revocation readiness: ARI allows systems to quickly detect and respond to revocation events when an urgent renewal is necessary, well before certificates get close to their due dates.

• Simple implementation: The extension is mature (RFC 9773) and the implementation is straightforward, providing simplified renewal logic and CA-optimized timing.

• Good citizenship: Anyone using ARI helps the CA optimize its infrastructure, and contributes to better aggregate behavior across the entire ecosystem.

If you’re still relying on static renewal thresholds, give ARI a look—Shopify wholeheartedly encourages all ACME users and client developers to adopt the ARI extension.

This was also posted on EFF’s blog.

As we announced earlier this year, Let’s Encrypt now issues IP address and six-day certificates to the general public. The Certbot team at the Electronic Frontier Foundation has been working on two improvements to support these features: the --preferred-profile flag released last year in Certbot 4.0, and the --ip-address flag, new in Certbot 5.3. With these improvements together, you can now use Certbot to get those IP address certificates!

If you want to try getting an IP address certificate using Certbot, install version 5.4 or higher (for webroot support with IP addresses), and run this command:

sudo certbot certonly --staging \
  --preferred-profile shortlived \
  --webroot \
  --webroot-path  \
  --ip-address 

Two things of note:

• This will request a non-trusted certificate from the Let’s Encrypt staging server. Once you’ve got things working the way you want, run without the --staging flag to get a publicly trusted certificate.

• This requests a certificate with Let’s Encrypt’s “shortlived” profile, which will be good for 6 days. This is a Let’s Encrypt requirement for IP address certificates.

As of right now, Certbot only supports getting IP address certificates, not yet installing them in your web server. There’s work to come on that front. In the meantime, edit your webserver configuration to load the newly issued certificate from /etc/letsencrypt/live//fullchain.pem and /etc/letsencrypt/live//privkey.pem.

The command line above uses Certbot’s “webroot” mode, which places a challenge response file in a location where your already-running webserver can serve it. This is nice since you don’t have to temporarily take down your server.

There are two other plugins that support IP address certificates today: --manual and --standalone. The manual plugin is like webroot, except Certbot pauses while you place the challenge response file manually (or runs a user-provided hook to place the file). The standalone plugin runs a simple web server that serves a challenge response. It has the advantage of being very easy to configure, but has the disadvantage that any running webserver on port 80 has to be temporarily taken down so Certbot can listen on that port. The nginx and apache plugins don’t yet support IP addresses.

You should also be sure that Certbot is set up for automatic renewal. Most installation methods for Certbot set up automatic renewal for you. However, since the webserver-specific installers don’t yet support IP address certificates, you’ll have to set a --deploy-hook that tells your webserver to load the most up-to-date certificates from disk. You can provide this --deploy-hook through the certbot reconfigure command using the rest of the flags above.

We hope you enjoy using IP address certificates with Let’s Encrypt and Certbot, and as always if you get stuck you can ask for help in our Community Forum.

As previously announced, over the next two years we will be switching the default certificate lifetime from 90 days to 64 days, and then 45 days. This will ultimately double the number of certificate renewal requests each day: today we expect renewal around day 60 (of a 90-day certificate), while in the future we expect renewal around day 30 (of a 45-day certificate). If you use an ACME client that supports ARI, this will happen automatically.

The good news for subscribers is that you don’t need any changes to your rate limits, whether you are using our default limits or have requested an override. Our rate limits affect issuance for new domain names (or groups of domain names), but renewals are exempt. So, for instance, if you are managing a set of 15,000 certificates that you continually renew, and create 250 new certificates (with new domain names) each day, you will be well within our limits both before and after the transition. The 250 new certificates daily will still be well under our New Orders per Account limit of 300 per three hours. And the 15,000 existing certificates will continue to be unaffected by rate limits, whether your ACME client is renewing them every sixty days or every thirty.

When you request a certificate from Let’s Encrypt, our servers validate that you control the hostnames in that certificate using ACME challenges. For subscribers who need wildcard certificates or who prefer not to expose infrastructure to the public Internet, the DNS-01 challenge type has long been the only choice. DNS-01 works well. It is widely supported and battle-tested, but it comes with operational costs: DNS propagation delays, recurring DNS updates at renewal time, and automation that often requires distributing DNS credentials throughout your infrastructure.

We are implementing support for a new ACME challenge type, DNS-PERSIST-01, based on a new IETF draft specification. As the name implies, it uses DNS as the validation mechanism, but replaces repeated demonstrations of control with a persistent authorization record bound to a specific ACME account and CA. The draft describes this method as being “particularly suited for environments where traditional challenge methods are impractical, such as IoT deployments, multi-tenant platforms, and scenarios requiring batch certificate operations”.

DNS-01 Proves Control Repeatedly

With DNS-01, validation relies on a one-time token generated by us. Your ACME client publishes a TXT record containing that token at _acme-challenge., and we query DNS to confirm that it matches the expected value. Because each authorization requires a new token, DNS updates become part of the issuance workflow. The benefit is that each successful validation provides fresh proof that you currently control DNS for the name being issued.

In practice, this often means DNS API credentials live somewhere in your issuance pipeline, validation attempts involve waiting for DNS propagation, and DNS changes happen frequently — sometimes many times per day in large deployments. Many subscribers accept these tradeoffs, but others would prefer to keep DNS updates and sensitive credentials out of their issuance path.

DNS-PERSIST-01 Authorizes Persistently

DNS-PERSIST-01 approaches validation differently. Instead of publishing a new challenge record for each issuance, you publish a standing authorization in the form of a TXT record that identifies both the CA and the specific ACME account you authorize to issue for this domain.

For the hostname example.com, the record would live at _validation-persist.example.com:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890"
)

Once this record exists, it can be reused for new issuance and all subsequent renewals. Operationally, this removes DNS changes from the critical path.

Security and Operational Tradeoffs

With DNS-01, the sensitive asset is DNS write access. In many deployments, DNS API credentials are distributed throughout issuance and renewal pipelines, increasing the number of places an attacker might compromise them. DNS-PERSIST-01 instead binds authorization directly to an ACME account, allowing DNS write access to remain more tightly controlled after initial setup. The tradeoff is that, because the authorization record persists over time, protecting the ACME account key becomes the central concern.

Controlling Scope and Lifetime

DNS-PERSIST-01 also introduces explicit scope controls. Without additional parameters, authorization applies only to the validated Fully Qualified Domain Name (FQDN) and remains valid indefinitely.

Wildcard Certificates

Adding policy=wildcard broadens the authorization scope to include the validated FQDN, wildcard certificates such as *.example.com, and subdomains whose suffix matches the validated FQDN:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " policy=wildcard"
)

Optional Expiration

Subscribers who aren’t comfortable with authorization persisting indefinitely can include an optional persistUntil timestamp. This limits how long the record may be used for new validations, but also means it must be updated or replaced before it expires. Anyone using this feature should ensure they have adequate reminders or monitoring in place so that authorization does not expire unexpectedly. The timestamp is expressed as UTC seconds since 1970-01-01:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " persistUntil=1767225600"
)

Authorizing Multiple CAs

Multiple CAs can be simultaneously authorized by publishing multiple TXT records at _validation-persist., each containing the issuer-domain-name of the CA you intend to authorize. During validation, each CA queries the same DNS label and evaluates only the records that match its own issuer-domain-name.

Rollout Timeline

The CA/Browser Forum ballot SC-088v3, defining “3.2.2.4.22 DNS TXT Record with Persistent Value”, passed unanimously in October 2025, and the IETF ACME working group adopted the draft that same month. While the document remains an active IETF draft, the core mechanisms described here are not expected to change substantially.

Support for the draft specification is available now in Pebble, a miniature version of Boulder, our production CA software. Work is also in progress on a lego-cli client implementation to make it easier for subscribers to experiment with and adopt. Staging rollout is planned for late Q1 2026, with a production rollout targeted for some time in Q2 2026.

In a recent conversation with a Let’s Encrypt subscriber, we asked them to guess how many people work at ISRG, the nonprofit behind Let’s Encrypt (and Prossimo and Divvi Up). Their guess was about 100; they’d overestimated by 72.5 people. We’re a pretty small team, and we get a lot done, but most of that work is entirely remote, distributed, and automated. 

That is a big part of what makes FOSDEM special. For the last few years, we’ve had a stand at this annual conference in Belgium, where a few folks from our team have the opportunity to speak directly with thousands of conference-goers. We continue to learn so much from these conversations! 

That’s where the “Hello” part of this blog post comes in. At this year’s FOSDEM, we met so many Let’s Encrypt subscribers, and each of them has a unique relationship to Let’s Encrypt. We were pleasantly surprised by how many people told us they were using IP-address certificates, a new option we just made generally available in January. We had a lot of conversations about our plans to shorten certificate lifetimes. There were a few folks who asked about S/MIME (still no plans to do that). We invited people to continue to stay in touch by signing up for our newsletter. 

The most meaningful part of FOSDEM is being able to say “thank you”. Our goal in starting Let’s Encrypt was to improve security and privacy for people using the Internet, but that could not be achieved without the now millions of folks who decided to get a certificate. Our impact is predicated on this symbiotic exchange. While we were only able to directly express our gratitude to a few thousand people at FOSDEM, it was a reminder of how important the community is.

When the scripts that generate the data for letsencrypt.org/stats broke yet again, we decided to retire it rather than repair it. Let’s Encrypt issues six to ten million certificates each day, producing a large volume of logs that keeps growing. It became increasingly time-consuming and difficult to answer questions about our own issuance like “how many certificates use the ‘shortlived’ profile.” Using raw logs, this requires finding, parsing and extracting relevant portions of loglines. Querying the database behind our issuance API is not a practical option, as it’s built for transactions rather than analysis. We’d also used a log search SaaS product, but our bills were growing much faster than we’d like, and while it was fine for searching, it wasn’t able to do the analytic workloads we needed. We knew we could dream bigger and better.

This led us to seek a self-hosted solution with efficient storage for structured data in addition to logs. We chose ClickHouse because of its potential as a data warehouse; the combination of cost-efficient storage and fast aggregation over large datasets appealed to us. A bonus of ClickHouse is that it is open source, a key principle valued by Let’s Encrypt.

The first step in building our new infrastructure was to purchase new hardware. To back our ClickHouse warehouse, we bought three PowerEdge R7715 servers. Each is equipped with a 32-core AMD EPYC 9355P 3.55GHz processor, 384 GB of RAM and 32 × 3.2TB NVMe drives, working out to roughly 100TB raw storage. With structured data and 100 days’ worth of logs already in our database, we are only at ~14% of total capacity, leaving lots of room for future growth.

Logs are the bulk of our storage use and the foundation of our structured data, as every other table we build is derived from them. When it comes to log search, ClickHouse covers our basic needs with quick ingest and interactive SQL. However, there are query ergonomics that we want to improve, like using OpenTelemetry’s tracing features and ClickHouse’s tokenization settings.

Our primary target for structured data are our issuance records. A materialized view extracts those records from logs into their own table, and further views pre-aggregate from there. One such view counts issuance by day per profile. Now, questions like “what is our issuance by profile over the last 180 days” can be answered within milliseconds.

We used this approach to completely rebuild the pipeline for our public stats page. The scripts we abandoned used to take hours each day to read and process dozens of compressed data files. The Rube-Goldberg-Machine-like collection of steps failed several times a year, requiring us to intervene and fix it. ClickHouse now computes those same stats in less than 10 seconds. Aside from pre-aggregated issuance tables, we can accomplish this because ClickHouse’s native functions are capable of quick and complex aggregations. Below is a simplified snippet of our materialized view for daily stats, counting the unique set of active domains over months across millions of rows. Two things to point out about this query: array handling means we can query nested fields without reshaping the underlying data, and uniq uses approximations to stay fast at scale.

SELECT
uniq(arrayJoin(arrayMap(x -> x.value, arrayFilter(x -> x.type = 'dns', identifiers)))) AS fqdns_active,
uniq(arrayJoin(etld_plus_one)) AS reg_domains_active
FROM boulder.cert_issuances
WHERE not_before >= yesterday() - 90
 AND not_after >= yesterday()
 AND not_before <= yesterday()

Our ClickHouse issuance data also addresses the previous headache of identifying affected certificates during incidents. It used to take hours of engineer time to correctly scan and parse logs to find the affected set of serials or hours of computing time to return results from database queries, competing with production transactional load. Now, however, there is no log wrestling required, and because queries return quickly, we can iterate toward the right one in minutes rather than hours.

Not everything we needed came with ClickHouse. For scheduled reports, we built our own custom tool that queries ClickHouse and exports formatted results. It cost additional engineering time to design it to fit our needs, but we’ve seen payoffs already. Old reports were clunky to read and required chasing down context by hand. New reports, on the other hand, streamline security reviews via neat formatting and linking directly to relevant pages. The same reporting tool was repurposed for our revitalized stats pipeline as well.

The biggest challenge was backfilling data. While OTel collector handles live log ingestion well for us, it didn’t suit the task of backfilling historical logs. We couldn’t find throttling settings that handled the bulk import reliably – a large portion of files were silently dropped – and the alternative meant breaking up the import by hand. We instead switched to ClickHouse’s native S3 import method, standing up an S3-compatible gateway in front of our old logs to do so. While this avoided the earlier obstacles, this process required trial and error to match ingestion rules to OTel collector’s parsing to ensure the ingested logs matched regardless of whether they were backfilled or streamed in. Two lessons: don’t run bulk historical imports through a streaming collector, and match parsing rules across paths before you start.

One schema decision made these iterations cheap. For tables we anticipated requiring backfilling or recalculation, we deliberately chose the ReplacingMergeTree engine, so re-ingesting corrected rows simply replaced the old ones. This also came in handy for a materialized view computing aggregations across other rows. We got the calculations wrong more than once, and each time all we had to do was re-run the query rather than surgically remove bad rows. For tables without the engine, we did OPTIMIZE TABLE ... DEDUPLICATE BY, which was expensive, but only needed to be run once.

We hope that this is just the start, especially with our shorter certificate lifetimes and post-quantum certificates on the horizon. We plan to take full advantage of our new warehouse, extracting and pre-aggregating data that answers questions other teams care about, the way we did for issuance. Better analysis improves our operations and makes transparency cheaper, as our rebuilt stats page shows.

With powerful analytics in hand and only ~14% of our storage in use, we have room to store and analyze more than ever before.

Ten years ago, we printed one of the nerdiest t-shirts we’ve ever made. On the front was the entire PEM encoding of ISRG Root X1 in base64. Back then, it represented a future we were working toward. Today, that same design tells the story of just how far Let’s Encrypt has come.

Let’s Encrypt was already issuing publicly trusted certificates in 2016, but ISRG Root X1 itself was still slowly and quietly making its way into browsers and operating systems around the world. For many years, our certificates were trusted through a cross-sign from IdenTrust. ISRG Root X1 itself was added to the major trust stores fairly early on; the slow part was waiting for that update to reach the browsers and devices already out in the world, since many of them only get new trust stores when they’re updated. That took years.

I remember the day we generated Root X1 and the planning and careful execution involved. We all breathed a sigh of relief when it was done but knew that we were really just crossing the starting line since our goal was, and continues to be, to get the Web to 100% encryption.

— Josh Aas, Co-Founder and Executive Director, ISRG

The Internet’s Quiet Infrastructure

When you visit a website over HTTPS, your browser follows a chain of trust that ultimately leads back to a trusted root certificate, like Root X1. If everything is working correctly, the entire process is invisible. You see a secure connection and the cryptography quietly does its job.

When we started, 39% of page loads were encrypted. Today, in much of the world, it’s over 80%. Hundreds of millions of websites rely on Let’s Encrypt certificates every day. Most of the people using those websites will never know the name “ISRG Root X1,” and that’s exactly the point. What once required optimism and patience has become something billions of people depend on without even knowing it’s there.

The Same Design, A Different Meaning

That’s what made us want to bring it back. In 2016, it represented a goal. Looking back ten years later, we realized the same design had come to represent something entirely different.

Today, it represents a decade of work and support by engineers, contributors, sponsors, donors, and advocates who believed that secure communication on the web should be free, automated, and available to everyone. Their support helped make HTTPS the default, not a privilege.

If you have one of the few original shirts, let us know how it’s treating you by dropping a line to donate@abetterinternet.org.

A Story Worth Wearing

Let’s Encrypt is run by Internet Security Research Group (ISRG), a nonprofit funded by the generosity of our community. Every certificate we issue and every new challenge we take on is made possible by people who believe the Internet should be more secure and privacy-respecting for everyone.

If you donate $75 or more this summer we’ll send you a limited-edition ISRG Root X1 t-shirt and you can help share our story.

Then you’ll have the chance to tell the story of how you support one small piece of Internet infrastructure that went from an ambitious idea to something a large part of the web quietly depends on every day.

Let’s Encrypt is committed to a post-quantum-safe Web PKI. The path we’re planning to take is Merkle Tree Certificates (“MTCs”), a new approach that adds post-quantum authentication to the web without sacrificing the speed and reliability that have made TLS universal.

This post is about these plans and why we believe MTCs are worth pursuing as a key to a post-quantum future.

An increasingly urgent problem

For much of the last several years, the conversation about post-quantum cryptography has been a conversation about encryption. The reasoning was straightforward: an attacker who records encrypted traffic today might be able to decrypt it years from now once quantum computers can break the underlying math. Authentication, the part of TLS that indicates a server is who it says it is, has been a less urgent problem. A quantum computer needs to forge a signature in real time, not retroactively, so threats to authentication hinge on the existence of a cryptographically relevant quantum computer (CRQC).

That comfort has been eroding for a while. In the United States, the NSA’s CNSA 2.0 suite has directed national security systems toward post-quantum algorithms on a 2030-to-2035 schedule since 2022, and NIST’s draft transition guidance would deprecate RSA-2048 and P-256 after 2030 and disallow them after 2035. The European Union’s roadmap targets high-risk systems by the end of 2030 and broad migration by 2035. These mandates don’t bind the public Web PKI directly, but they set the end-of-decade timeline that the vendors, libraries, and standards bodies it relies on are already working toward.

This year, the timeline shortened further. Google announced that it would migrate its services by 2029, citing tightening estimates for the potential arrival of a CRQC. Cloudflare followed with a parallel commitment. In addition, Go 1.27 adds ML-DSA, a NIST-standardized post-quantum signature scheme, to the standard library, a sign that post-quantum signatures are becoming practical infrastructure.

Post-quantum authentication is no longer a problem the Web PKI ecosystem should defer. Long-lived keys (root certificate authorities, code-signing keys, identity systems) are particularly valuable targets, and new technology takes years to gain broad adoption, so the work has to start early.

The Web PKI’s unique circumstances

The Web PKI is one of the trickiest places to deploy post-quantum signatures. The reason is size.

ML-DSA-44, one of the smaller NIST standardized post-quantum signature schemes, has a signature roughly 2,420 bytes long. The algorithms used in the Web PKI today are much smaller. RSA-2048 signatures are 256 bytes and ECDSA-P256 signatures are 64 bytes. Public keys are bigger as well: 1,312 bytes for ML-DSA-44, 256 bytes for RSA-2048, and 64 bytes for ECDSA-P256. A typical Web PKI handshake today carries five signatures and two public keys. Replacing those with ML-DSA equivalents would push a single TLS handshake well past 10 kilobytes. Cloudflare’s research has shown that, at that scale, a meaningful share of TLS connections fail on real-world networks, and the rest get slower.

Larger handshakes would affect every TLS connection, not just those that would fail. They would mean constrained bandwidth, slower connections, and a worse experience for users, all in exchange for security against a threat that hasn’t materialized yet. That’s a steep cost to enable by default, and defaults are what actually move security at web scale.

Merkle Tree Certificates

A different design called Merkle Tree Certificates (“MTCs”) has been emerging over the past year, and we believe it is a strong path forward for the post-quantum Web PKI.

Instead of issuing certificates one at a time and signing each one individually, an MTC certificate authority issues certificates in batches, with a single signature covering the entire batch. Browsers stay up to date on those batch signatures (called “landmarks”) separately from the TLS handshake.

In the common case, the entire authentication path in an MTC handshake is one signature, one public key, and one inclusion proof. That’s smaller than today’s Web PKI handshake, even though MTCs use post-quantum algorithms. The other case is the “standalone” form. It uses slightly larger handshakes as a fallback when a client’s landmark is out of date.

There is more to MTCs than size optimization. Because every certificate is part of a published Merkle tree, transparency becomes a property of issuance itself. Today’s Certificate Transparency ecosystem is bolted on after the fact: certificates are issued by CAs, then logged separately, with extra signatures riding along in the TLS handshake to attest to that logging. With MTCs, a certificate cannot exist outside the Merkle tree. Certificate Transparency is built in.

This is not entirely new ground for us. Let’s Encrypt has operated Certificate Transparency logs since 2019. Those logs are append-only Merkle trees, the same core data structure MTCs are built on, and ones we have run in production, at scale, for years.

Cloudflare and Chrome are already running a feasibility experiment with MTCs against real internet traffic. The IETF’s PLANTS working group is working on standardizing the design. Chrome has announced that MTCs are its preferred path for adding post-quantum certificates to the public web.

Our plans

We are planning to support Merkle Tree Certificates as the path forward for the post-quantum Web PKI. We are targeting late 2026 for a staging environment that issues MTCs, and 2027 for a production-ready environment.

This is not a small endeavor. Issuing MTCs at the scale of Let’s Encrypt requires meaningful changes throughout our stack: in our issuance infrastructure, in the ACME protocol our subscribers use to obtain certificates, in revocation and operational tooling, and in the transparency-log infrastructure that MTCs subsume. We have been participating in the IETF PLANTS and ACME working groups as the standards take shape.

Alongside the MTC work, we are tracking the standards for ML-DSA signatures in X.509 (RFC 9881) and TLS (draft-ietf-tls-mldsa), and the ecosystem work this depends on, like the addition of ML-DSA to the Go standard library. The Web PKI’s transition to post-quantum security needs all of this to land in browsers, libraries, and ACME clients, whether the certificates ultimately delivered are MTCs or ML-DSA signed X.509.

What this means if you use Let’s Encrypt

Nothing changes today. Your current Let’s Encrypt certificates will continue to be issued and renewed exactly as they always have been. When post-quantum certificates become available from Let’s Encrypt, they will arrive the way our service always has: free, automated, and available to anyone with an ACME client.

The transition will take time. There are standards still being finalized, root programs still defining their requirements, and engineering work that has to land in the broader ecosystem (browsers, libraries, ACME clients) before any of this matters at scale. We will keep the community informed as the work progresses and as the timelines firm up.

If you maintain an ACME client or run an ACME-driven certificate pipeline, this is a good moment to start tracking the work in the PLANTS working group and the discussions on the mtcs@chromium.org mailing list. Some of the changes coming will require client-side support, and the ecosystem will benefit from clients that are ready when the issuance side is.

A note on the wider post-quantum transition

For the broader internet community: post-quantum encryption is the more urgent problem, because any TLS connection without post-quantum key exchange is potentially harvestable for later decryption. If you operate servers, please ensure they support hybrid post-quantum key exchange (X25519MLKEM768). Major browsers and operating systems already do, and turning it on at the server is one of the highest-leverage things you can do this year.

In closing

We have been building infrastructure for the public web since 2013 on the principle that security should be available to everyone, automatically, at no cost. The quantum transition is a generational change in how that security works under the hood.

We will have more to say as the work progresses. Until then, our thanks to the cryptographers, browser engineers, IETF working groups, and CAs whose work has gotten us this far.

Have you ever needed to make sure your website has a broken certificate? While many tools exist to help run an HTTPS server with valid certificates, there aren’t tools to make sure your certificate is revoked or expired. This is not a problem most people have. Tools to help manage certificates are always focused on avoiding those problems, not creating them.

Let’s Encrypt is a Certificate Authority, and so we have unusual problems we need to solve.

One of the requirements for publicly trusted Certificate Authorities is to host websites with test certificates, some of which need to be revoked or expired. This gets messed up more than you might expect, but it’s a bit tricky to get right. Test certificate sites exist to allow developers to test their clients, so it’s important that they’re done right.

We’d previously used certbot, nginx, and some shell scripts, but the shell scripts were getting a bit too complicated. So we wrote a Go program tailored to the specific needs of a CA’s test certs site.

The websites

We need to host three sites per root certificate:

• A valid certificate, like any other website.
• An expired certificate, past its expiry date.
• A revoked certificate, but it can’t be expired.

Valid is easy enough; it’s the normal case of any other website. This is a solved problem.

Expired, too, is pretty easy. Issue one certificate, wait until it expires, and then you can use it forever. Not a normal feature, but so long as your webserver doesn’t get upset at it being expired, it’s easy to set up once and leave it.

Revoked, though, is where it’s easiest to slip up. You could fail to revoke a certificate and serve a perfectly valid one, or you could let your revoked certificate expire. Making sure your website is serving a non-expired but revoked certificate is not something any of the off-the-shelf tools support.

The ingredients to bake a cake

In order to implement our program, we need a few different ingredients to mix together.

First and foremost, we need to be able to get certificates. Because we’re writing this in Go, we’re using Lego as a library to request the certificates. Obtaining a certificate requires completing a domain validation challenge. We can hook Lego up to the Go webserver we’re using to complete TLS-ALPN-01 validation. We use that challenge type because it doesn’t require any more setup beyond exposing our webserver to the internet.

To get a revoked certificate, we request a certificate and then revoke it. That’s something we can do with Lego and ACME too: The account which issued a certificate can request it be revoked. We then need a way to check that the certificate is revoked. Certificates contain an HTTP URL pointing to the Certificate Revocation List (CRL) which we poll until our certificate’s serial number appears in it.

Let’s Encrypt implements the ACME standard, which defines how clients can get certificates. In general, we think ACME clients integrated into webservers are often the best way to get certificates for websites. They can automatically handle challenges, manage and reload certificates, and overall minimize the amount of work and reduce problems.

We also need a way to wait until a certificate is in the right state. The valid certificate is ready to use right away, but that’s not true for the revoked and expired certificates. The revoked certificate needs to wait at least until it appears in a CRL, which can be up to an hour. Expired certificates need to wait even longer: Even if we request the shortest-lived certificates we offer, that’s still six days. To handle this, our program stores a “next” certificate instead of immediately overwriting the current one. We wait at least 24 hours for the revoked certificate to make sure any CRL caches or push-based CRL infrastructure have time to process the revocation. The expired certificate has to wait until it passes its expiration date. After the program decides a certificate is ready, it replaces the current certificate and passes it off to the webserver. Normal ACME tools don’t support this because they can usually start using a certificate as soon as it’s obtained.

And finally, we need a webserver to host the certificates. We’re using Go, which has a great built-in TLS and HTTP serving stack we can use. The Go TLS server takes a GetCertificate callback function that decides what certificate to use for each new connection. We have all our certificates in-memory and select the right one to serve based on the request’s SNI. This function is also where we hook up Lego to serve the challenge certificates required for TLS-ALPN-01. Because we prioritize serving the correct certificate over uptime, we refuse to handle a connection if the corresponding certificate is expired (unless it should be expired!).

Visiting the sites

If you visit one of our revoked sites, you might not get an error message. Revocation checking in browsers varies pretty widely, and has historically not worked great. Today’s state-of-the-art is Firefox’s CRLite, which is efficient and reliable. Ubuntu is deploying upki, a Rustls project based on CRLite. We hope other browsers and operating systems follow suit. The upki project is a great example of a project making use of these revoked test certificates, too.

The actual content of the website isn’t terribly important: We just have a little HTML page explaining what the site is. But since this website is meant for testing clients, there’s more than just browsers connecting. In particular, it’s pretty routine that I try connecting with curl or some other terminal http client, and getting a bunch of HTML spewed to your terminal isn’t very nice.

As a small Easter egg, we added a plain text version of the website with an ASCII art version of our logo that we serve if your HTTP client doesn’t include text/html in its Accept HTTP header. You can pass a ?txt or ?html URL parameter to specifically request one or the other version of the content, if you just want to see the ASCII art.

Let’s Encrypt has four root certificates right now. Each of them has test sites linked both here and from our documentation.

The code

As with a lot of Let’s Encrypt, the code for this project is open-source. You can find it at https://github.com/letsencrypt/test-certs-site/. Other Certificate Authorities who need to run similar test certificate sites are welcome to use it. If you need any features that would make using our test certs site easier for your TLS/HTTPS client testing, please feel free to create an issue on that repository.

Nick Silverman is a Senior Infrastructure Engineer on the Edge Infrastructure team at Shopify, where he maintains the systems that provision, renew, and publish SSL certificates for millions of merchants’ custom domains. He is also a contributor to the Ruby acme-client gem.

The challenge

Shopify’s automated certificate management system relied on a static renewal threshold: 30 days before the end of the 90-day lifetime. To spread the load of provisioning and renewing certificates, we implemented a random 0–72 hour delay for each. While this helps evenly distribute certificate management over time, it did not take into account the Certificate Authority’s (CA) load. It was also incapable of reacting to a dynamic renewal window based on information provided by the CA.

However, this approach needed greater resilience to solve what is, in the end, a distributed coordination problem. The weaknesses are:

• No rapid revocation response: The static logic is not aware of revocations at all.

• Brittleness to lifetime changes: The static 30-day threshold is not resilient to changes in certificate lifetime, such as Let’s Encrypt’s announced plan to move to 45-day certificates.

• Imperfect load distribution: Despite the random jitter, massive renewal bursts could still occur.

Shopify needed to develop a global coordination system to balance the load and handle regular and urgent renewals. Thankfully, Let’s Encrypt has led the charge on a solution for this and other very important aspects of the certificate lifecycle.

The journey

Let’s Encrypt and the Internet Engineering Task Force (IETF) published the ACME Renewal Information (ARI) standard which makes an endpoint available that provides a recommended window of time for the renewal to occur. The endpoint returns a payload that looks something like this:

GET /renewal-info/ACME_KEY_IDENTIFIER
{
  "suggestedWindow": {
    "start": "2026-02-03T04:00:00Z",
    "end": "2026-02-04T04:00:00Z"
  }
}

Shopify’s certificate management system uses the acme-client Ruby gem originally authored by another Shopify employee. A growing number of ACME clients, including certbot, have enabled support for ARI, but the Ruby gem did not yet support this feature. Rather than building a custom solution, we decided to enable support for the ARI extension directly in the client.

Let’s Encrypt’s guide to integrating ARI provided the necessary roadmap, and the implementation was completed with one PR. This contribution means that not only Shopify, but also the wider Ruby community, can benefit from the ARI extension.

Deployment and ARI at scale

Once we shipped the gem support, integrating ARI into our certificate management system was straightforward. Instead of checking a static 30-day threshold, we now query the ARI endpoint and use the suggested renewal window as the gate for initiating renewals. Those dates are stored alongside the certificate upon its initial provisioning.

The updated Ruby gem provides a method for fetching renewal information:

renewal_info = client.renewal_info(certificate: existing_certificate_pem)

This method generates an ARI certificate identifier that can be used when making the API call. The client also includes a helper method, suggested_renewal_time, which chooses a random time between the returned start and end dates. The certificate identifier can be passed to the new_order method via the replaces key, which can grant a higher priority or bypass rate limits for renewals occurring during the window, depending on the CA’s policies.

Critically, Shopify also regularly polls the ARI endpoint for updated renewal timestamps. This allows our systems to rely on those timestamps as the primary renewal timing logic and removes the need for inflexible hard-coded expiry thresholds. This becomes the mechanism that Let’s Encrypt uses to dynamically change the renewal time due to a revocation event.

Results and rewards

Since enabling the use of the ARI extension, our certificate management system has become significantly more robust. Shopify now delegates the responsibility of determining renewal timing to Let’s Encrypt. The ARI extension has proven to be an impactful infrastructure improvement and the benefits gained are immediate. These benefits, alongside fewer manual interventions, are the operational success story:

• Future-proofing: We gained resilience against any future certificate lifetime changes and mass revocation events without needing code updates—ensuring our renewal logic is flexible.

• Optimized load: We directly benefit from the CA’s coordinated load balancing provided by the suggested renewal window, eliminating local randomness issues and the need for complex global coordination.

• Revocation readiness: ARI allows systems to quickly detect and respond to revocation events when an urgent renewal is necessary, well before certificates get close to their due dates.

• Simple implementation: The extension is mature (RFC 9773) and the implementation is straightforward, providing simplified renewal logic and CA-optimized timing.

• Good citizenship: Anyone using ARI helps the CA optimize its infrastructure, and contributes to better aggregate behavior across the entire ecosystem.

If you’re still relying on static renewal thresholds, give ARI a look—Shopify wholeheartedly encourages all ACME users and client developers to adopt the ARI extension.

This was also posted on EFF’s blog.

As we announced earlier this year, Let’s Encrypt now issues IP address and six-day certificates to the general public. The Certbot team at the Electronic Frontier Foundation has been working on two improvements to support these features: the --preferred-profile flag released last year in Certbot 4.0, and the --ip-address flag, new in Certbot 5.3. With these improvements together, you can now use Certbot to get those IP address certificates!

If you want to try getting an IP address certificate using Certbot, install version 5.4 or higher (for webroot support with IP addresses), and run this command:

sudo certbot certonly --staging \
  --preferred-profile shortlived \
  --webroot \
  --webroot-path  \
  --ip-address 

Two things of note:

• This will request a non-trusted certificate from the Let’s Encrypt staging server. Once you’ve got things working the way you want, run without the --staging flag to get a publicly trusted certificate.

• This requests a certificate with Let’s Encrypt’s “shortlived” profile, which will be good for 6 days. This is a Let’s Encrypt requirement for IP address certificates.

As of right now, Certbot only supports getting IP address certificates, not yet installing them in your web server. There’s work to come on that front. In the meantime, edit your webserver configuration to load the newly issued certificate from /etc/letsencrypt/live//fullchain.pem and /etc/letsencrypt/live//privkey.pem.

The command line above uses Certbot’s “webroot” mode, which places a challenge response file in a location where your already-running webserver can serve it. This is nice since you don’t have to temporarily take down your server.

There are two other plugins that support IP address certificates today: --manual and --standalone. The manual plugin is like webroot, except Certbot pauses while you place the challenge response file manually (or runs a user-provided hook to place the file). The standalone plugin runs a simple web server that serves a challenge response. It has the advantage of being very easy to configure, but has the disadvantage that any running webserver on port 80 has to be temporarily taken down so Certbot can listen on that port. The nginx and apache plugins don’t yet support IP addresses.

You should also be sure that Certbot is set up for automatic renewal. Most installation methods for Certbot set up automatic renewal for you. However, since the webserver-specific installers don’t yet support IP address certificates, you’ll have to set a --deploy-hook that tells your webserver to load the most up-to-date certificates from disk. You can provide this --deploy-hook through the certbot reconfigure command using the rest of the flags above.

We hope you enjoy using IP address certificates with Let’s Encrypt and Certbot, and as always if you get stuck you can ask for help in our Community Forum.

As previously announced, over the next two years we will be switching the default certificate lifetime from 90 days to 64 days, and then 45 days. This will ultimately double the number of certificate renewal requests each day: today we expect renewal around day 60 (of a 90-day certificate), while in the future we expect renewal around day 30 (of a 45-day certificate). If you use an ACME client that supports ARI, this will happen automatically.

The good news for subscribers is that you don’t need any changes to your rate limits, whether you are using our default limits or have requested an override. Our rate limits affect issuance for new domain names (or groups of domain names), but renewals are exempt. So, for instance, if you are managing a set of 15,000 certificates that you continually renew, and create 250 new certificates (with new domain names) each day, you will be well within our limits both before and after the transition. The 250 new certificates daily will still be well under our New Orders per Account limit of 300 per three hours. And the 15,000 existing certificates will continue to be unaffected by rate limits, whether your ACME client is renewing them every sixty days or every thirty.

When you request a certificate from Let’s Encrypt, our servers validate that you control the hostnames in that certificate using ACME challenges. For subscribers who need wildcard certificates or who prefer not to expose infrastructure to the public Internet, the DNS-01 challenge type has long been the only choice. DNS-01 works well. It is widely supported and battle-tested, but it comes with operational costs: DNS propagation delays, recurring DNS updates at renewal time, and automation that often requires distributing DNS credentials throughout your infrastructure.

We are implementing support for a new ACME challenge type, DNS-PERSIST-01, based on a new IETF draft specification. As the name implies, it uses DNS as the validation mechanism, but replaces repeated demonstrations of control with a persistent authorization record bound to a specific ACME account and CA. The draft describes this method as being “particularly suited for environments where traditional challenge methods are impractical, such as IoT deployments, multi-tenant platforms, and scenarios requiring batch certificate operations”.

DNS-01 Proves Control Repeatedly

With DNS-01, validation relies on a one-time token generated by us. Your ACME client publishes a TXT record containing that token at _acme-challenge., and we query DNS to confirm that it matches the expected value. Because each authorization requires a new token, DNS updates become part of the issuance workflow. The benefit is that each successful validation provides fresh proof that you currently control DNS for the name being issued.

In practice, this often means DNS API credentials live somewhere in your issuance pipeline, validation attempts involve waiting for DNS propagation, and DNS changes happen frequently — sometimes many times per day in large deployments. Many subscribers accept these tradeoffs, but others would prefer to keep DNS updates and sensitive credentials out of their issuance path.

DNS-PERSIST-01 Authorizes Persistently

DNS-PERSIST-01 approaches validation differently. Instead of publishing a new challenge record for each issuance, you publish a standing authorization in the form of a TXT record that identifies both the CA and the specific ACME account you authorize to issue for this domain.

For the hostname example.com, the record would live at _validation-persist.example.com:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890"
)

Once this record exists, it can be reused for new issuance and all subsequent renewals. Operationally, this removes DNS changes from the critical path.

Security and Operational Tradeoffs

With DNS-01, the sensitive asset is DNS write access. In many deployments, DNS API credentials are distributed throughout issuance and renewal pipelines, increasing the number of places an attacker might compromise them. DNS-PERSIST-01 instead binds authorization directly to an ACME account, allowing DNS write access to remain more tightly controlled after initial setup. The tradeoff is that, because the authorization record persists over time, protecting the ACME account key becomes the central concern.

Controlling Scope and Lifetime

DNS-PERSIST-01 also introduces explicit scope controls. Without additional parameters, authorization applies only to the validated Fully Qualified Domain Name (FQDN) and remains valid indefinitely.

Wildcard Certificates

Adding policy=wildcard broadens the authorization scope to include the validated FQDN, wildcard certificates such as *.example.com, and subdomains whose suffix matches the validated FQDN:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " policy=wildcard"
)

Optional Expiration

Subscribers who aren’t comfortable with authorization persisting indefinitely can include an optional persistUntil timestamp. This limits how long the record may be used for new validations, but also means it must be updated or replaced before it expires. Anyone using this feature should ensure they have adequate reminders or monitoring in place so that authorization does not expire unexpectedly. The timestamp is expressed as UTC seconds since 1970-01-01:

_validation-persist.example.com. IN TXT (
  "letsencrypt.org;"
  " accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890;"
  " persistUntil=1767225600"
)

Authorizing Multiple CAs

Multiple CAs can be simultaneously authorized by publishing multiple TXT records at _validation-persist., each containing the issuer-domain-name of the CA you intend to authorize. During validation, each CA queries the same DNS label and evaluates only the records that match its own issuer-domain-name.

Rollout Timeline

The CA/Browser Forum ballot SC-088v3, defining “3.2.2.4.22 DNS TXT Record with Persistent Value”, passed unanimously in October 2025, and the IETF ACME working group adopted the draft that same month. While the document remains an active IETF draft, the core mechanisms described here are not expected to change substantially.

Support for the draft specification is available now in Pebble, a miniature version of Boulder, our production CA software. Work is also in progress on a lego-cli client implementation to make it easier for subscribers to experiment with and adopt. Staging rollout is planned for late Q1 2026, with a production rollout targeted for some time in Q2 2026.

In a recent conversation with a Let’s Encrypt subscriber, we asked them to guess how many people work at ISRG, the nonprofit behind Let’s Encrypt (and Prossimo and Divvi Up). Their guess was about 100; they’d overestimated by 72.5 people. We’re a pretty small team, and we get a lot done, but most of that work is entirely remote, distributed, and automated. 

That is a big part of what makes FOSDEM special. For the last few years, we’ve had a stand at this annual conference in Belgium, where a few folks from our team have the opportunity to speak directly with thousands of conference-goers. We continue to learn so much from these conversations! 

That’s where the “Hello” part of this blog post comes in. At this year’s FOSDEM, we met so many Let’s Encrypt subscribers, and each of them has a unique relationship to Let’s Encrypt. We were pleasantly surprised by how many people told us they were using IP-address certificates, a new option we just made generally available in January. We had a lot of conversations about our plans to shorten certificate lifetimes. There were a few folks who asked about S/MIME (still no plans to do that). We invited people to continue to stay in touch by signing up for our newsletter. 

The most meaningful part of FOSDEM is being able to say “thank you”. Our goal in starting Let’s Encrypt was to improve security and privacy for people using the Internet, but that could not be achieved without the now millions of folks who decided to get a certificate. Our impact is predicated on this symbiotic exchange. While we were only able to directly express our gratitude to a few thousand people at FOSDEM, it was a reminder of how important the community is.

Update: March 11, 2026

If you use Certbot, see Six-Day and IP Address Certificates Available in Certbot for details on requesting these certificates.

Short-lived and IP address certificates are now generally available from Let’s Encrypt. These certificates are valid for 160 hours, just over six days. In order to get a short-lived certificate subscribers simply need to select the ‘shortlived’ certificate profile in their ACME client.

Short-lived certificates improve security by requiring more frequent validation and reducing reliance on unreliable revocation mechanisms. If a certificate’s private key is exposed or compromised, revocation has historically been the way to mitigate damage prior to the certificate’s expiration. Unfortunately, revocation is an unreliable system so many relying parties continue to be vulnerable until the certificate expires, a period as long as 90 days. With short-lived certificates that vulnerability window is greatly reduced.

Short-lived certificates are opt-in and we have no plan to make them the default at this time. Subscribers that have fully automated their renewal process should be able to switch to short-lived certificates easily if they wish, but we understand that not everyone is in that position and generally comfortable with this significantly shorter lifetime. We hope that over time everyone moves to automated solutions and we can demonstrate that short-lived certificates work well.

Our default certificate lifetimes will be going from 90 days down to 45 days over the next few years, as previously announced.

IP address certificates allow server operators to authenticate TLS connections to IP addresses rather than domain names. Let’s Encrypt supports both IPv4 and IPv6. IP address certificates must be short-lived certificates, a decision we made because IP addresses are more transient than domain names, so validating more frequently is important. You can learn more about our IP address certificates and the use cases for them from our post announcing our first IP Certificate.

We’d like to thank the Open Technology Fund and Sovereign Tech Agency, along with our Sponsors and Donors, for supporting the development of this work.

Social Media

• GitHub
• Google+
• LinkedIn
• Twitter
• Instagram