Safarian: To highlight and expand slightly on some of the things Arash mentioned — let’s just put ourselves in that situation. The attack is present at the moment. You need to, first and foremost, take a deep breath. Keep composure. The stress that it’s going to bring you—if you’re level-headed about these things you can make better decisions. You want to avoid making any kind of drastic or destructive decisions in the application.
There’s internal and external. I want to divide this up. First thing is, just evaluate the effect here. What is the net impact? How serious is this attack that’s happening today? If it’s somebody who is scamming tens of thousands in virtual currency in your system, you need to evaluate that. Is this good or bad for the system? If this is someone who’s gone in and is changing personal information for other users, you need to stop that. You need to evaluate that. How much of a concern is this? Keep your stress levels intact.
The next thing you want to discuss with your team internally is the exposure of that hack. There are times where bad actors will go out there and they like to brag about it. You want to check social media. Simple things. Even Google keywords — bots can go out there and capture these keywords so you can understand, okay, did somebody brag about a hack on a gaming system? How easily is that re-created? How much exposure is there for copycat attacks? How much information is out there? You want to manage the information going out. You want to control that message.
Let’s say the dust starts to settle after you’ve started to handle the attack. You want to understand, okay, what are the after-action tickets? There’s going to be a lot, across the board. Whether it’s something at your infrastructure layer, something in your application layer — again, that triage we mentioned previously. Is there something you need to follow up on as a fast follow? Do you need to go back and rework something, re-factor something?
Are there some things that were brought to light that start to take your focus in a different direction? You can say, “Okay, let me look at my application or my infrastructure from a different direction here.” A lot of times developers and engineers would be coding for sunny scenarios, when they’re pressured to get business functionality up and out. Taking that step back allows you to reevaluate, to put on a different pair of glasses and say, “What does my application look like? What does my infrastructure end up looking like?” Building on these after-action tickets after the dust settles.
Finally, what I would really strongly recommend is everyone give yourselves a strong customer success message and team. The team that’s sitting there and dealing directly with the customer or end user, that message needs to be clear, concise, and to the point. Obviously you want to control that message. You want to make sure the end user understands that there was an issue. It’s being resolved or has been resolved. This is what we want to do to make it up to you. Thank you for being such a strong user for us. We want to continue your business. This is what we’re doing to ensure this doesn’t happen again.
You want to control that message. You want to make sure your customer success team is out there on those front lines and doing whatever they need to do to calm the nerves.
Question: To prepare for cyberattacks, what tools or strategies do you recommend in pen testing your own systems? That’s penetration testing, where you’re basically attacking yourself to test your systems.
Haghighi: As far as strategies, as I mentioned, first of all you have your resources and then you have your staff developers, network engineers, operational engineers, systems engineers, and the team you have around them. Everyone around your company, everyone in your IT team, they should know the game architecture. Before launching any game, any service, you have to know the data flow. You have to come up with systems where you know exactly — if not A, you have to talk with B. If, under normal throughput, A talks with B, but you have more than normal throughput, that’s an alarm. You have to think about that. There could be an issue between A and B.
You have to come up with methods of testing. We’re doing tons of penetration tests, and not limited to our websites. We’re doing port scanning. We’re doing protocol scanning. We’re using hacking tools provided by a vendor we’ve been very satisfied with for testing features. We’ve done consulting. We’re participating in security events, seminars. Get as much information as you can that you can use in yoru testing area.
Safarian: One thing I would recommend right off the bat is building yourself some synthetic testing tools. You can do in-house stuff, but there’s definitely other options out there, plenty of software that we could mention. You can do simple things like CloudWatch in the AWS stack, stuff like New Relic, which is essentially an agent that’s going to sit on top of your entire application architecture and be continually monitoring. That’s on the web server side of things.
On the front end, mobile side of things, you have things like Crashlytics Fabric and so on. These things that are tuned to identify any kind of poor performance. On the back end, ELK stack, Logstash, Kibana — set yourself up so you can have the detail monitoring. You’re not going to be able to cover everything, but you can prepare yourself.
Let me go back to synthetic testing again. Set yourself up with some simple tests, some simple scripts. Unit testing. When you’re about to publish a feature, make sure you run it through your unit tests. Make sure you follow the best practices and standards. You have a service that’s sitting there and synthetically pinging your system for common, known use cases to get through your normal regression.
If you have a solid QA team, you can help train them. It’s really a learning and growing process here. We have a great QA staff, and they’re constantly reimagining their process and their regression. They’re trying to say, “Okay, here’s my script for all the sunny scenarios. Here’s my script for the bad actor scenarios.” Things we’ve taken from the past and the IQ we have from previous attacks, and then we say, “Okay, let’s roll these into our regression testing.”
These systems are all big machines. Sometimes any change you make will affect something down the line. It’s more like chemistry than physics. If there’s some kind of implementation you’ve built in system A, there might be some effect or exposure in system B. Really building that out as a growing unit — whenever you build out any kind of feature or version of your framework and your system, as it goes out just have that diligent practice to regroup and say, “Hey, what are the Q tests that surround this? What are the things we need to do? What are my scripts that I need to create or run?”
Build that strong relationship with your QA team. They are the shield for you. These are the guys that are going to sit there and do the dirty work that some developers — dirty secret — actually don’t do. Work together with them. Get these things out there. Use the tools that are available. Again, New Relic has a cool synthetic testing piece. I mentioned Crashlytics Fabric. Having an ELK stack set up on your back so you can keep that — it’s all about that instrumentation.
On your system, from an architecture standpoint, modular design. Sit down where you can limit the effects. Any time you make one change, make sure that doesn’t affect a lot of other things in that modular design pattern.
Question: I’ve seen a lot of news about ransomware. Is that something I should be concerned about? Should I consider working with any firms or vendors ?
Safarian: With ransomware, again, these are some of the questions you need to ask yourself. I go back to the audience poll, the “concerned, but not worried” people. I wonder what their take is on that. You need to evaluate what’s critical in your system, what’s available there that could affect your end user, and what’s there that could bring your end-of-day revenue to a halt. You need to prepare yourself for a lot of these things.
Answer these questions at an executive level first. Try to realize, what is the effect on the business for any of these types of attacks? What are some of the safety nets and guardrails we can put in place to avoid this. If any of this information does get out, if my system has been completely compromised, what are the things we can do? If you bring in any talented dev ops engineer, he’s going to help you put your stack into a different environment right away. “Let’s get out of here. Let’s get off this and move somewhere else.” Or you’ll be able to throw your database layer pretty deep to where you can put more security around it and avoid some of these things.
Again, best practices. I would preach that until I’m blue in the face. There’s a lot of white paper out there, a lot of great articles out there. If you’re on specific stacks, like AWS, they have a lot of — I don’t want to call their documentation amazing. It’s not the prettiest. But it’s very in-depth and very robust. You can get in there, roll up your sleeves, do a little research, do a little digging. This is your hard work that you’re putting out there into the universe.
GamesBeat: The thing about ransomware that’s interesting that it’s skyrocketed in the past year. The problem before that picking up a ransom was very hard. With the arrival of things like Bitcoin it’s become a lot harder to catch that person.
Haghighi: You guys have covered everything, but again, ransomware is a type of malicious software. It’s built around an encryption component. The only thing we can recommend is that you have to update your antivirus, update your antimalware, your Windows firewalls and patches, or Linux patches. It doesn’t matter which operating system you’re using. Make sure to never, ever trust any information you’re getting from third parties or emails. It’s easy to attach any kind of malware, even in a simple JPEG file.