Five Tips on Building Serverless Teams in an Enterprise
Streaming video provider Toons.TV — owned by Finnish enterprise Rovio Entertainment and most famous for its Angry Birds cartoon series — has amassed some 8.5 billion streaming views in the past four years. Marcia Villalba, Full Stack Developer at Rovio Entertainment spoke at the recent Serverlessconf conference in Austin to discuss how her team reoriented towards a serverless approach to meet these challenges and to speed up their backend systems.
And much like a game of Angry Birds itself, the challenge of implementing serverless in the enterprise can be seen as a source of frustration or as a challenge to reach the next level. Here are some of the lessons she picked up on the way:
1. Train the Team Together
One of the approaches Villalba took when introducing serverless ideas into the enterprise was to create small teams that would learn and work together. At Rovio, Villalba’s team held weekly workshops to review the serverless landscape of possibilities and to conduct mini-sprints.
“In one week we had a proof of concept running for identity management,” she gave as an example. Following this, her team did proof of concepts for image resizing, transcoding videos and migrating databases.
With serverless being such a new technology, everyone in the engineering team was at the same level, so setting aside time to learn, experiment and build helped create a strong serverless team at Rovio.
2. Rethink the Cloud
One of Villalba and her team’s a-ha moments with serverless came with a rethink of what cloud architecture models look like.
Villalba discussed a fairly common serverless design pattern, where an API may make calls to check if there is a new image available for content.
“We decoupled everything. The idea is to rethink architecture in a different way,” Villalba said. At Rovio, the main issue came from having a legacy system where request responses were tied up with a whole series of parameters, some of which were business critical (like knowing where the viewer was located so that viewing contracts with content suppliers could be enforced), while others were not (such as the image of the viewer’s display interface, for example). Having large request/responses intertwined with parameters meant huge costs when processing a viewer’s video content. For Rovio, this was an opportunity to rethink how to manage this application delivery using serverless.
“As our content didn’t change much, we were able to fetch the content from the database every ten minutes and put it in a bucket. In order to get all the variations in place, we chopped out all of the parameters that were not part of the business and just kept the country and the application,” Villalba said.
The team used Amazon Step Functions to preprocess all the different variations, build JSON files and store them in the bucket. “Now, all our variations are generated because our content is static, we don’t need a server to create our content over and over again,” she said. The AWS Lambda service field the request, returning a JSON file from the content delivery network.
“Every country has its own CDN, so it works like a charm and it is pretty cheap,” Villalba said.
However, with all of these proof of concepts and migrations to serverless, Villalba’s team quickly had a mess on its hands, with “millions of functions, resources, databases, transcoding and things floating around.”
Villalba said to manage this, they began organizing their serverless components into microservices. “Everything that was about content belonged to one serverless project, everything with transcoding goes into another serverless framework project. Then instead of having millions of resources floating around, we have maybe ten, and each one represents a different microservice. And then it is totally easy to manage because you can have GitHub repos, and if you need to deploy the content repo, you know all of the resources will be deployed. So we called that the Service Stack.”
Villalba said there were also a number of resources that were shared amongst services, including documentation, and SSL certificates. “But we wanted to have everything as code. So we created a permanent stack using Python and Cloud Formation and had that in a repo, and whenever we needed a new environment, we just create a few new environment variables and poof! Everything is up, and it is all on GitHub so we can track it, we can see who did what and so on.”
3. Test to Your Limits
Villalba says when her team started to implement serverless architectures in production, they tested everything. But her team quickly realized that wasn’t the way to approach testing in serverless environments.
“You need to test them in a different way,” said Villalba. “We were also running tests against managed services in the cloud. But that slows down a lot of our unit tests. We took the decision to only unit test our business logic. We had to create an abstraction and then we could properly unit test our business logic and just do integration tests for the rest.”
In serverless environments, urged Villalba, as everything is decoupled, having integration tests becomes critical.
Load testing is also an area that Villalba says is counterintuitive. The thinking may be that since it is a serverless architecture, everything scales, so load testing is unnecessary. But by doing load tests, Villalba’s team was able to identify what their soft limits were for all of the managed services they were subscribing to in AWS.
With the customer size that Toons.TV had, with the potential for peaks in traffic and usage at any time, the enterprise needs to be able to manage their costs when buying more AWS services. “You need to know your soft limits and if you are approaching them, you need to call Amazon and ask them to raise it, but you do not want to do that in advance because if you do, that means if someone wants to attack you, you are letting them spend all your money,” Villalba said. (This is what security expert Guy Podjarny describes as a billing DDoS.)
“We also used load testing to discover how much the system would cost, because we did not know much less it was, so we used load testing to find out our pricing. I don’t know if that’s the right way, but it worked for us,” Villalba said.
4. Don’t Take It Personally
Villalba says there is still a lot to learn with serverless: I do not know how to run one lambda at a time, I still cannot find out how to do that,” she said.
With the serverless market moving so fast, her team was often building solution components that then were being offered as new products from Amazon directly.
“We wrote the authorization gateway, and then Amazon released one a week later. We wrote an orchestrator lambda and then Step Functions came in. So you need to be prepared to change things and be aware of what is coming and how the code environment is changing,” recommended Villalba. “Be willing to throw a lot of things to trash.”
5. … And Have Fun
Villalba’s presentation at Servelessconf was one of the most dynamic presented and reflected much more of the real world where serverless is playing out. Villalba credits serverless as being the most fun in her developer career.
“I couldn’t just Google my questions, I had to think through how to solve problems,” she said. That is the attitude that is often needed amongst a team implementing serverless at the enterprise level. In many cases, the map has not yet been drawn.
