reader.casz.org

 

Help

Automattic

Making the web a better place

WRONG AGAIN

12 Aug 2026, 4:40 am

WP Engine’s motion is a desperate attempt to get sanctions and other extreme remedies because Matt used ephemeral messaging in personal communications with romantic partners having nothing to do with this case.  Here’s what really happened, as the brief we filed tonight shows:

  • We preserved and collected.  A litigation hold was issued to our employees, including Matt. We collected over six terabytes of documents. More than 1.5 million documents were reviewed. Five of Matt’s personal devices were forensically imaged, including his Telegram, WhatsApp, and Signal messaging applications.
  • Nothing relevant was part of any ephemeral messaging. Our e-discovery vendor collected messages from Matt’s messaging platforms going back to 2012 (Telegram), 2015 (WhatsApp) and 2019 (Signal) through March 2026 — the entire key period in the case and years on either side. WP Engine cannot identify a single missing message about anything having to do with this case. 
  • They had every chance to find evidence of their accusations. Fact discovery produced more than 225,000 documents and over 114 hours of deposition testimony — including 21 hours of Matt across three days, with three attorneys, more than 80 exhibits, and over 1,300 questions.

Newspack Helps Hong Kong Free Press Protect Its Stories

4 Aug 2026, 8:52 pm

In 2015, following months of pro-democracy protests, Tom Grundy founded Hong Kong Free Press, which bills itself as the city’s “only independent English-language news source.” Since then, Hong Kong has slipped further down the World Press Freedom Index. Free Press journalists navigate “one of the most complicated press-freedom situations in the world,” says Grundy, the site’s editor-in-chief. “The last thing I have the capacity for and am able to worry about is the technology side.”

A few years later, as worrying new security laws took effect, Grundy turned to Automattic’s Newspack for help. The seven journalists at Free Press provide steady, on-the-ground coverage, but they’re not website developers. A bespoke content management system required constant maintenance, and conversations with developers faltered. He worried about site stability and donor engagement, but felt “underprepared and unqualified” for technical conversations. “I needed to speak to tech people who were also aware and understanding of journalistic terminology, and where we come from,” he says.

That year, Grundy began working with Newspack to migrate multiple donation management tools and the Free Press’s content management system to a single platform. “Newspack very patiently worked with me for over a year to get it right,” he says. Through Newspack’s Audience Management tools, the Free Press has developed a greater variety of member benefits, from feature previews and exclusive columns to ad-free browsing, and increased donations by 33 percent.

The security of Free Press’s reporting is a top concern for Grundy, who has seen other newsrooms raided and prominent publications closed. Newspack—which is built on WordPress and hosted by WP Cloud, both Automattic products—has helped “keep the website safe if the worst happens in Hong Kong,” he says. “We have had no downtime.” Last year, when the Free Press was selected for a potentially costly tax inspection, Grundy reached out to Newspack for help putting a targeted fundraising plan in motion. “Within a month, they’d rolled it out to everybody,” he says. 

Facing an unstable media landscape, Grundy credits Newspack with keeping Hong Kong Free Press nimble. “I can speak to someone directly who speaks my language, in terms of journalism, and gets things sorted,” he says. “That has liberated me and our news outlet to focus on navigating the red lines and getting the news out.”

Brickomattic: A Celebration of What We Have Built

4 Aug 2026, 12:00 pm

Zach Spears, a Happiness Engineer at Automattic, used our recent Radical Speed Month initiative to finish and ship a project years in the making: Brickomattic, a collection of LEGO® kits that create hangable versions of logos from all of Automattic’s products. Below is Zach’s story of how the project came to be.

This month marks my five-year anniversary at Automattic. As I look ahead to my sabattical with excitement, I can’t help but also look back on my tenure and the incredible things I have been a part of. In a similar fashion, Automattic the organization seems to find itself at a crossroads. We celebrated our 20th anniversary last year, and what it means to be Automattic is changing. We can see these exciting changes with recent acquisitions like Beeper, and changes to how we grow our existing businesses like Automattic for Agencies. As with any remarkable milestone, this is all worth noting and celebrating.

From a young age, I’ve always enjoyed the art of gift-giving; I find that it gives me a way to express my appreciation when words may not come so easily. Over the last five years, I’ve had the opportunity to be part of five different teams at Automattic that touch nearly every part of the business, and to work with incredible colleagues from across the world. Before long, my gratitude for my coworkers combined with my lifelong love of creativity and LEGO® bricks, and I decided to put together some microbuilds.

For my team meetup one year, I used BrickLink’s Studio software to design and organize mini LEGO® rocket kits for my team, which was known as Jupiter. The next year, on a team where we referred to ourselves as pirates. I constructed a small movable pirate ship, complete with custom packaging and instructions. A year later, for a team called Retrograde, I created a small diorama that featured a spaceship against a celestial backdrop.

Pirate Ship for Team Chiron
Spaceship for Team Retrograde

In addition to the team-specific sets, I wanted to build something more generic for the whole of the company. To do so, I challenged myself to recreate Automattic’s logo in brick form, which I’d never done before. I was really happy with my first design which was intended to sit flat on a desk, and I later adapted it with parts from LEGO® art sets to become suitable for wall hanging. After I shared the design internally, Matt Mullenweg asked if I could build some for the Automattic offices. If you ever find yourself at our offices in San Francisco or New York City, keep an eye out for the logo built from LEGO® pieces!

Automattic Logo after validation
Logo at our NoHo Space (please, someone rotate it 90º!)

Once I sent out the finished designs to our offices, I began crafting a series of wall hanging designs for Automattic’s primary products and business units, a process that took several months of slow (but invigorating) iteration and design. Some of the designs came together seamlessly like Day One, and some proved more complicated such as Sensei LMS. The most difficult to figure out was the Woo logo, which is a division I currently work in. I first tried a more minimalist cart design, but eventually came up with a technique to capture the bendy Woo “W.”

Original “Cart” design
Final “Woo” design

Nearly all of the final 20 designs are the same height and width and centered around the LEGO® wall hook-compatible part. This means that they can be displayed as wall art just like the official LEGO® Art sets. I’m proud of how the designs came together, and my personal favorite is the Sensei LMS design, which features the silhouette of a Bonsai tree.

Launching Brickomattic

As the 20 final designs were finally coming together, I needed a way to share to share them with my colleagues that felt more like an art gallery than a blog post. I also wanted the kits to feel three-dimensional the way a physical set would, and to experience the joy of the kit coming together, as it would in a LEGO® video game.

And so: Brickomattic.

I launched a WordPress.com site, grabbed the domain brickomattic.com, and began building. With assistance from generative coding tools, I was able to implement a fully featured and inviting gallery complete with self-constructing 3D models and an x-ray button. I also designed and exported building instructions and high-resolution renders of the designs. The site uses a library of parts from LDraw and LDR export files from the Studio software to build the immersive gallery. I’m currently working on adapting this plugin into an open-source WooCommerce plugin for MOC (“My Own Creation,” a term for custom-designed models) sellers in the broader AFOL (Adult Fans of LEGO®) community.

Building Instructions
3D Model Preview
Parts List

Why LEGO®?

At Automattic we like to say that code is poetry. This looks great on a T-shirt, but it’s also an expression of a broader philosophy. We approach what we create as a business with thought and intention. From that standpoint, it’s not much of a stretch to draw a line to the LEGO® artform; both take small parts and combine them to create something worth being proud of.

It’s also fair to point out that I’m a lifelong LEGO® fan, and this was something that I did for myself. I love creating, and I love sharing the things that I create with others. Brickomattic is as much an act of play as it is a gift to my coworkers. It forced me to step outside of my comfort zone and build things I didn’t think I was capable of.

Finally, I have a strong belief in the value of the tangible. As a business that builds software, and that works distributed, we can sometimes feel disconnected from our work. This is why meetups and team swag are important to our culture: They foster a feeling of belonging. Creating a physical representation of the software you work on every day is an important physical reminder of the things you build and the impact that they (and you) can have.

Women at Work: A Podcast for Creatives on a Platform for Creatives

27 Jul 2026, 3:00 pm

Verònica Fuerte is the founder and “creative directress” of Barcelona’s Hey Studio, where her team builds visual languages for clients and projects around the world. When she looks at the design industry, she sees something that’s always been true but too often overlooked: Women have been shaping it all along, they just haven’t always gotten the credit.

That observation became the seed for Women at Work, a podcast dedicated to amplifying the voices of female creatives across design, creativity, and innovation. The premise is simple but powerful. “If you listen to somebody that made it happen,” Verònica says, “you can have the same experience. You can follow the same path. It’s just about having examples of other people.”

The podcast’s next step was, as Verònica puts it, organic. Hey Studio’s website has always run on WordPress, so when Women at Work grew enough to need its own digital home, the choice of destination was clear. And while she has long appreciated the ease of open source software, WordPress’s longevity made her decision even simpler. “I know that WordPress is going to be forever,” she says. “So I think it makes my life easier in many, many aspects.”

The site brings everything Women at Work is about—episodes, guest stories, and its broader mission—into one place, and gives the project a professional presence that matches the quality of the conversations themselves. 

But Verònica is thinking bigger than episodes and archives. “A podcast is a community,” she says. “And I think how this community can be seen is part of the future.” With WordPress.com as the foundation, Women at Work is building toward exactly that: a growing platform that helps the next generation of creatives see themselves in the industry—and in each other.

Lights, Camera, ‘Code for the People’

13 Jul 2026, 4:15 pm

Over the past two decades, open source software has powered much of the internet’s infrastructure. Technologies like WordPress—which powers more than 40% of the web—demonstrate the impact of a model built on global collaboration and shared knowledge.

Now, though, the open web is facing an inflection point. As a handful of digital landlords and closed AI systems threaten to replace a free and open internet with algorithmic walled gardens, those of us fighting for an open-source future need to stick together.

To tell this story, Automattic partnered with acclaimed director Bao Nguyen on a new documentary short, Code for the People. The film explores this critical breaking point and serves as a rallying cry to reclaim the open-source spirit before the future of the internet is decided for good.

At a time when AI and digital platforms are redefining the rules of the internet, Code for the People invites audiences to engage in a conversation that extends beyond software. The documentary argues that the future of the web will depend on the choices the technology community makes today, positioning open source as a strategic foundation for building a more resilient, transparent, and decentralized digital infrastructure.

Earlier this month, Code for The People partnered with Variety and KQED to hold premiere screening events in New York City and San Francisco. The panels and discussions that followed made one thing obvious: These aren’t niche concerns. They’re questions that matter to anyone who builds, creates, or publishes online. Check out the full documentary above, and experience the premiere screenings through the gallery below.

Automattic Celebrates Graduates of University of Illinois Chicago’s AI Leaders Program

23 Jun 2026, 4:00 pm

On June 23, around 40 students from Illinois and Louisiana will wrap up the year at University of Illinois Chicago (UIC) as the first graduating cohort of AI Leaders—the nation’s first workforce-focused AI literacy course tied to a recognized credential.

These students built real WordPress projects. They applied generative AI to real work. For example, Stephen Poole completed the Ultimate Success track and posted a reflection after every session, and shipped his portfolio at stefanpoole.dev. Another student, Sharaheel Ali, finished WordPress 1–8 and launched sharali.blog, framing his portfolio around WordPress support engineering, cybersecurity, and AI. They earned a micro-credential backed by UIC and the WordPress Foundation. And those who completed the course also earned $1,000. Automattic is proud that we joined the University of Illinois Chicago (UIC) to make a donation to UIC Tech Solutions Open Source Fund earlier this year.

Built on Open Source

AI Leaders works because it’s built on open source. Students learn on WordPress, which powers more than 40% of the web, and contribute back to it. They use generative AI as a tool and leave with a credential employers can verify. The community they build while contributing to WordPress remains long after graduation. 

At Automattic, we believe deeply in open source; it’s the ethos that built the web. We think it should help build the next generation of AI-literate workers too—in the open, accessible to all, and with full ownership of the work they do.

Bigger Things are on the Horizon

AI Leaders is just one piece of Automattic’s broader investment in education. We’ve pledged $500 million worth of hosting accounts over the next five years to support open source learning, workforce development, and the next generation of contributors to the web.

We’re investing in this because access to AI and open source skills shouldn’t depend on where you live or what you can afford. The first cohort proves the model works. Now we want to help it grow

What’s Next

Prospective employers and agencies will attend a graduation party in Chicago to meet the students, and more job placements are in motion. The next round of AI Leaders is on the horizon; we’ll share application details for that soon. And to the inaugural AI Leaders cohort: congratulations. The web is what we make it, and we’re excited to watch you shape that legacy.

Automattic Joins the Tech Coalition

1 Jun 2026, 1:00 pm

Automattic and Tech Coalition Community logos on a blue background.

Automattic was built on the belief that publishing should be open and accessible to everyone. That mission only means something if the spaces where people create, connect, and share are also safe—especially for kids. Today, we’re proud to announce that Automattic has joined the Tech Coalition, a global alliance of technology companies working together to combat online child sexual exploitation and abuse (OCSEA).

The Tech Coalition brings together more than 60 companies (including Meta, Google, Microsoft, Apple, and Snap) to develop safer products, share threat intelligence, and support research that no single company could drive on its own. Members collaborate on detection technology, cross-platform signal sharing through programs like Lantern, and transparency standards that raise the bar across the industry.

Child safety is something we’ve thought deeply about and care deeply about across our platforms, from Tumblr to WordPress.com. But protecting kids online isn’t something any one company can solve alone. Joining the Tech Coalition helps us expand beyond what we can accomplish within our own walls, and allows us to contribute to something larger alongside people who share that commitment.

An open web requires its stewards to lead on safety. That starts with us.

Pressable, WordPress VIP, and WP Cloud Partners Earn Secure Hosting Alliance Certification

28 May 2026, 1:00 pm

We’re proud to share that Pressable and WordPress VIP have each earned official certification from the Secure Hosting Alliance (SHA), an i2Coalition initiative that recognizes hosting providers with a genuine, demonstrable commitment to security, reliability, and responsible operations. At the same time, through WP Cloud’s sponsorship of the SHA, several WP Cloud partners have earned certification of their own.

What the SHA Evaluates

Certification from the  SHA Trust Seal Program requires providers to demonstrate operational standards across four areas: 

  • Transparent customer-facing policies
  • Documented infrastructure misuse protocols with prioritized abuse response
  • Proactive network reliability and recovery planning
  • Lawful handling of government and law enforcement data requests

These principles represent the day-to-day discipline that separates hosting built for trust from hosting built for price.

Pressable: Built for Agencies, Certified for Trust

For the agencies and high-growth businesses building on Pressable, this certification provides independent, third-party verification of what our customers already expect: fast abuse response, transparent communication, and infrastructure engineered for real-world traffic demands.

“Most hosting providers compete primarily on what they charge. SHA is about what they’re willing to be accountable for,” Vik Patel, general manager of Pressable explains,  “For the agencies and businesses building on Pressable, this certification means their infrastructure has been independently reviewed and held to a documented standard. That’s the kind of assurance serious operators actually want.”

WordPress VIP: Enterprise-Grade Security, Formally Recognized

WordPress VIP serves some of the world’s largest publishers, brands, and governments. At that scale, security, abuse response, and lawful data handling are baseline requirements. SHA certification reflects the operational rigor that VIP’s enterprise customers already demand.

“Running infrastructure for enterprise clients like NASA means treating security, abuse response, and lawful data handling as core obligations every day,” says Steph Yiu, CEO, WordPress VIP. “SHA’s standards align with what our enterprise customers already require of us, and earning the accreditation reflects the operational work behind that — work that usually doesn’t get recognized publicly.”

WP Cloud: Extending High Standards Across the Ecosystem

SHA accreditation has historically been most accessible to the largest, best-resourced hosting providers. Through WP Cloud, Automattic is changing that. As an official SHA sponsor, WP Cloud is covering the cost of accreditation for its partner ecosystem, putting the same trust seal within reach of independent hosts, agencies, and specialty providers who are already doing the work.

“The Secure Hosting Alliance is one of the most credible efforts I’ve seen to name and elevate the hosts who are doing this the right way,” says Jesse Friedman, Head of WP Cloud, Automattic “We didn’t want SHA accreditation to be something only the largest providers could afford.” 

Several WP Cloud partners have already earned certification: Convesio, a containerized WordPress host with a WooCommerce focus; Ivapix, built for agencies and multi-site operators; and Infinite Source, which combines development and hosting under one roof.

A Healthier Web Starts With Accountable Infrastructure

Hosting providers don’t just sell servers; we provide the foundation that businesses, journalists, nonprofits, and creators rely on to reach the world. The Secure Hosting Alliance represents the industry choosing to hold itself to a higher standard, and we’re glad to have Pressable, WordPress VIP, and a growing cohort of WP Cloud partners standing on the right side of that line.

To learn more about the SHA Trust Seal Program, visit hostingsecurity.net. WP Cloud partners interested in starting their SHA accreditation can get in touch.

Where Automattic Meets Up

7 May 2026, 3:50 pm

Sharing Where We Stay

Automattic is fully distributed, with no headquarters. A few times a year, teams meet in person at places like a villa in Tuscany, a beach house on the Outer Banks, or a cabin outside Boulder.

Over time, we’ve built a valuable list of team-tested retreat spots: places that work for fifteen people with laptops, jet lag, and strong coffee opinions.

That list mostly lived in spreadsheets and our internal P2 system. I kept thinking it deserved a better home. So on the side, in evenings and weekends, I built one: meetomattic.com.

What it is

Meetomattic is a public directory of the venues Automattic teams have actually stayed in. Not aspirational listings, places we’ve booked, slept in, worked from, and would (or wouldn’t) book again. You can browse by city, see the venues, and read the kind of practical notes that only matter once you’ve tried to run a demo over patchy hotel Wi-Fi.

It’s small on purpose. No accounts, no booking engine, no affiliate links. Just a clean window into how one distributed company actually gathers in the physical world.

Why bother

This is a side project I’m tinkering with for fun. It isn’t going to move a metric. It probably won’t show up in a quarterly review. But distributed work is becoming more common, and the companies figuring it out are mostly figuring it out alone. Every team I talk to asks the same questions when they start planning their first meetup: Where do you go? What do you look for? What’s worth paying for (and what isn’t)?

We’ve been answering those questions internally for nearly two decades. It felt strange to keep the answers to ourselves.

What I hope happens

If a founder at a 15-person remote startup books a place from this list and their team comes home closer than they were before, that’s the win. If another distributed company publishes their own version and we can borrow ideas from each other, even better. The ambition is modest: Share what we’ve learned, lower the activation energy for in-person time at remote companies, and generate a little goodwill in a community that’s done a lot for us.

It’s a work in progress. Venues are still being added, photos are still being chased down, and I keep finding small things to polish. But it’s live, it’s real, and it’s ours to share.

Take a look: meetomattic.com. If you run a distributed team and end up booking somewhere from the list, I’d love to hear how it went.

Behind the Fiber-Optic Rope: Engineering WordCamp

7 May 2026, 2:00 pm

A screen containing various networking settings.

From time to time, our systems engineers write up a case study detailing a notable moment on the infrastructure front lines. The latest comes from Tyler Leeds, an Automattician on our Network Operations team, explaining how they ensure a rock-solid video stream from flagship WordCamp events.

Every year, Automattic sponsors and supports dozens of WordCamp meetups, where members of the WordPress open source community can connect in person, work collaboratively, and get updates on the software project’s latest developments. Of those many local and regional WordPress events, there are three (now four!) flagship events that mobilize the community at a national, regional, or continental level. WordCamp Asia, WordCamp EU, and WordCamp US—and starting in 2027, WordCamp India—bring together the best in our community. Not surprisingly, these flagship events require significant work to put together and manage. Local organizing committees don’t always have the resources of a company like Automattic, so we try to pitch in where we can.

I’ve recently returned home from WordCamp Asia in Mumbai, India. As we’re now preparing for WordCamp EU in Kraków, Poland, I thought I’d share a bit about what it takes to keep the tech flowing at a major conference. My purview extended to two key areas, Wi-Fi and streaming video, which I’ll discuss in turn.

Wi-Fi

If I were to design a challenge for a venue’s Wi-Fi system, I couldn’t dream up a better stress test than WordCamp Contributor Day. More than 1,400 community creators in a single room, all of them actively working on laptops? We’ve already seen the phenomenon cause failures at multiple WordCamp flagship events. So, in the leadup to this year’s WordCamp Asia, my network-engineering team was tasked with one simple mission: Make it go. 

Science time: The 2.4Ghz spectrum that Wi-Fi uses also happens to be the same spectrum used by the magnetron in a microwave oven. Water absorbs the spectrum so effectively that it’s perfect for vibrating the molecules in a frozen pizza until they heat up. That absorption also means that Wi-Fi becomes extremely difficult in a room filled with humans moving around, and it’s even harder over its higher-frequency 5.8Ghz cousin. To a Wi-Fi network engineer, you’re all just mobile radio attenuators with a conference lanyard. 

The limiting factor in large-scale events tends to be the availability of Wi-Fi frequency rather than a more common factor like upstream bandwidth. Imagine each channel in the Wi-Fi spectrum like a room filled with groups of people, all trying to have their own conversation with one person in each group. Eventually, too many people are all speaking at the same time and individual conversations fade into noise. The louder each person speaks, or the closer the groups, the more likely the noise from each group bleeds into the others and the conversations all collapse. 

Good Wi-Fi systems design tries to keep the rooms small, the talkers quiet, and the groups as far from one another as possible. 

At WordCamp Asia 2026, we had the pleasure to work with the Jio World Conference Centre in Mumbai. The JWCS already had the majority of work completed, with more than 100 access points in their Lotus ballroom, all managed by a controller-based wifi system. 

Conferences are one of the areas where we’ve seen controller-based Wi-Fi environments sharply outperform controllerless (cloud-based) solutions. The reason here is simple: Controller-based solutions are able to more rapidly react to the changing RF conditions in the room as attenuators conference attendees move about, create mobile hotspots, and do whatever else they can to ruin our cleverly designed system. Cloud-based solutions typically sample the RF environment periodically through the day, and then do one or more daily optimization routines to manage the RF parameters of each access point. This is fine in an office or home, but in the rapidly shifting world of a conference, it tends to be too slow to be effective. 

We’d been able to review engineering drawings and design briefs with the Jio team prior to the event, so went in with a reasonable degree of confidence that Wi-Fi would be stable. We limited users to 15Mbps to keep a lid on bandwidth, and implemented a “WordPress” SSID. The end result was a Wi-Fi environment that remained consistently good, even during the stress of contributor day.

A more interesting case, though, is what we’ve found while assisting the upcoming WordCamp EU organizers in Kraków. 

The single most common Wi-Fi problem we encounter at venues is due to a misunderstanding by venue IT staff about the use of 20/40/80/160Mhz Wi-Fi channels.

We routinely encounter venues that have assigned 40Mhz channel widths to their 5Ghz spectrum. It’s understandable. 40Mhz channels roughly double the available bandwidth for each user, which sounds like a good thing, so most venue IT staff set it that way. The issue is that (to use our prior analogy) this creates fewer, larger groups that can much more rapidly overwhelm the conversation in the room. A much better solution in these environments is to break it up into the maximum number of 20Mhz channels in order to maximise usage of the very limited Wi-Fi spectrum, even if any given conversation has a limited bandwidth.

One of the actions we’ve been able to complete for WordCamp EU is a basic site survey of the venue, ICE Kraków. You’d be surprised how many venues are deployed without ever doing a site survey or attempting to optimize the RF environment, and you can see the results of that in the below image, an overlapping channel graph for the third floor of the complex. During the site survey, we found that many APs were running on 40 and 80Mhz channels. What makes this problem worse is that the co-channel interference caused by consuming too much RF airtime negates any possible throughput advantages you could have gained from running on 40 or 80Mhz channels in the first place. It’s very much a self-defeating problem. 

Thankfully, the folks at ICE Krakow are enthusiastic about improving their Wi-Fi environment, so we also worked with them to locate bandwidth and throughput limitations in their Wi-Fi switching network, and have given recommendations and potentially equipment to solve them. We have every confidence the issues will be sorted well before the WCEU ever kicks off. 

So, that’s Wi-Fi. On to the next issue that plagues network-ops teams.

Streaming

Over three days of activities at WordCamp Asia 2026, we enjoyed a successful live stream with very few issues, despite the event being hosted in Mumbai during a period of general internet degradation across India:

A network-monitoring graph showing our connectivity from Amsterdam > India during WordCamp Asia. Normally, you’d see a flat yellow line with no orange; the orange spikes denote packet loss.

This wasn’t luck: After encountering some streaming issues during our CEO’s keynote at last year’s State of the Word, we had set out early to make sure we wouldn’t see a repeat occurrence at WordCamp Asia.

  • Starting about six weeks before the event, we met with the venue’s internet provider to establish a baseline of connectivity—100Mbps synchronous, delivered over hard line—and discuss how this connectivity actually functioned in the real world.
  • We extensively tested our streaming partner’s platform (Castr), and worked with our video and promotions teams to fully understand the backup options and understand the limitations and advantages of the various methodologies.
  • We also completed pre-conference testing on video backup streaming. We quantified what a failover to our backup stream would look like to a viewer, what a switch back to primary entailed, and how to accomplish the switch in either direction.
  • We developed an explicit set of specifications for the video stream to protect against outright failure: H.265 HEVC with an SRT transport. (More on this decision later.) 
  • We created a failure action plan, which included articulating the set of conditions under which we’d execute a change to the backup encoder/ISP and when we’d switch back to primary once it stabilized.
  • We coordinated with off-site colleagues to monitor the stream remotely and use a dedicated channel to report to on-site personnel; that way, we could ensure that what the world saw matched what we were sending from the venue.
  • We documented the process to recreate a YouTube event in case of a stream failure, and discussed with all involved parties so that everyone was clear on their duties in such a scenario.
  • Finally, we developed a comprehensive cheat-sheet that showed all the streaming destinations that would be required for the video production team to use with their encoders. We wanted to minimize any complexity the video production team would face backstage, so gave them a simple set of destinations and a simplified cue-sheet to prevent overburdening them with tasks during the show.

Technical Considerations

As mentioned, we settled on a stream spec of H.265 HEVC with an SRT transport. We set a minimum specification of a M3-based Mac computer to allow use of HEVC, the hardware encoder that’s built into the M series chips. In the end, we used M4 Mac Minis that performed solidly.

H.265 allowed us to use roughly half the bitrate required for high-quality 1080p streaming. A subjective quality assessment showed that H.265 at 4500kbps was visually equal to H.264 at 7000-8000kbps. This helped make stream issues due to bandwidth limitations less likely.

We moved to SRT (UDP-based) transport over RTMP (TCP-based) transport after observing that internet quality issues could cause the stream to disconnect from Castr, which would cascade into YouTube stream failures. SRT streams can progressively degrade quite strongly without disconnecting in situations that would have caused an RTMP failure.

Granted, the internet quality in India was sometimes variable, and so we experienced a few of these issues:

Even in these cases of video chunking, the audio remained coherent. The few times it occurred, we executed our backup failover plan for streams, and in all cases, the stream corrected immediately. We’d anticipated some level of stream degradation when we architected the backups, and we were rewarded by the smoothness of the results.

For my team, WordCamp Asia was a big win. The knowledge base we built during preparation was invaluable, and we’ll use it to inform future live streaming events like WordCamp EU. Even if the result was due to process engineering rather than technical wizardry, the outcome was the same: a solid stream that never went down.

Page processed in 0.411 seconds.