Today’s game consoles offer functionalities — such as platform crossplay — that are something of a miracle. Because every game runs differently, especially across different systems and networks, these new perks come with backend complexities that, without the proper strategies in place for monitoring and observability, can negatively impact the gaming experience.
So, how are today’s companies staying ahead of these challenges to ensure the best game experience? The answer: monitoring tools that help them observe their infrastructure in real time and provide timely feedback and internal notifications when things go wrong, so they know before the player does.
Red alert?
In my role at Sensu, I oversee the customer reliability team and regularly hear from our customers about the overwhelming number of alerts their teams are tasked with managing and addressing on a daily basis. It’s been our experience that alert fatigue can have a huge, often negative, impact on any organization, putting stress on monitoring teams dealing with them every day. In fact, in a recent report, 72% of CIOs say that alert fatigue is a big problem affecting their team.
Given the scale and complexity of gaming companies’ legacy monitoring setups, monitoring teams are constantly faced with thousands of critical errors. With so many alerts, it can be virtually impossible to tell not only which issues to escalate, but which are impacting players (and how). Under these circumstances, team morale is understandably low, and turnover is typically very high. As games continue to make the shift to online, companies are faced with the decision to entirely re-evaluate their approach to monitoring, migrating from their legacy toolset to a next-generation solution designed for scalability.
While monitoring tools may make it easier for gaming companies to stay ahead of potential bugs and server issues, what do these behind-the-scenes technologies really mean for the player and their experience? To name a few:
- Uninterrupted online streaming
- Seamless crossplay
- Quicker updates to bugs and errors
- Less downtime, more game time
Companies such as Industrial Light & Magic, Weta Digital, Verizon Media, ArenaNet, and Activision, which includes Demonware — a global online gaming company that creates and hosts online services for popular video games such as the Call of Duty series and Guitar Hero across various consoles — have introduced new monitoring solutions to their internal teams to replace legacy solutions that couldn’t keep up with the demand of today’s online gaming ecosystems.
Focusing on what matters
During Monitorama 2018, Demonware presented on how the company was able to reduce daily alerts and notifications by addressing underlying issues to ultimately create a more engaged monitoring team. Previously, the Demonware monitoring team felt overwhelmed with the volume of alerts and notifications they were receiving on a daily basis — they received literally thousands per day, each of which needed to be addressed with an antiquated system. Keeping up with these alerts for a team of their size was daunting, and the task of vetting these alerts and addressing the issues seemed impossible.
After implementing a modern monitoring platform that could keep up with the company’s scale and complexities across various game titles and consoles, Demonware reduced the volume of alerts by 99% — from thousands of alerts to only ten per day. With a proper monitoring solution in place, Demonware’s monitoring team better understands the issues affecting their games, and now has the data and insights needed to resolve long-standing bugs, resulting in a more reliable and quality experience for their players.
Detect and correct issues on the fly
Over the years, online video games companies have evolved from the traditional development model to the fast-paced DevOps model which provides the flexible and rapid “feedback” loop to detect and correct issues on the fly. With next-gen monitoring solutions in place, monitoring teams are able to quickly identify and resolve specific anomaly patterns in their applications which could have resulted in the game going down for an extended period of time while the issue was being addressed.
Monitoring tools have allowed for gaming companies to stay one step ahead of delays and downtimes for their players, ultimately boosting team morale and avoiding a throwdown on a subreddit (as reviews and conversations are integral pieces of the gaming community).
As the gaming industry continues to evolve, companies and their monitoring teams are challenged with keeping up with the changing landscape. From understanding new infrastructure and tooling, to navigating the complexities of a new gaming platform — effective monitoring tools can ensure a seamless experience for both the player and the monitoring team tasked with delivering that delightful end-user experience.
Cameron Johnston is the VP of Customer Reliability at Sensu, where he helps customers solve their stickiest monitoring challenges.