Sponsored By
Clearly, the way in which subscribers view content and applications is rapidly evolving. This means that the way in which service providers plan, create, monitor, fix and measure their offerings must change as well.
It is. DevOps is an umbrella term that implies a far more iterative, flexible, interactive and cross-pollinating process in which people with various skillsets, and holding a variety of roles, work together closely. Instead of having a slow process with a beginning and an end, DevOps is fast-paced, circular and aimed at constant improvement.
Telco Transformation recently chatted with Comcast Cable's Gulrukh Ahanger and John McCann. Ahanger, the operator's vice president of engineering, formerly was its executive director of application engineering. Prior to that, she was the director of engineering at Yahoo and director of the technology group at Maven Networks. McCann is Comcast's executive director of product engineering. Previously, he served as the MSO's senior director of applications engineering.
Telco Transformation: What does DevOps bring to Comcast?
Gulrukh Ahanger: It's basically helping us to improve the efficiency, reliability and productivity of our products. In DevOps mode we are using cross-disciplinary groups. For example, a product person, a designer or an engineer who writes code or an engineer who deploys it in production all sit together, know each other and work together to deliver the product. It gives them ownership of the product they are delivering. They are end-to-end vested in the process and are able to take it off the ground and into production and beyond.
John McCann: For us it is less important what we call it and what terms are used. What is more important to us is that we are as efficient as possible in our software development and deployment practices. We try to empower engineering teams to feel a real sense of ownership of the products and services they are developing, to develop strong relationships with the work they are doing and to be become invested in their work. We feel that personal investment provides the best possible quality when it comes to delivering products and services to our customers.
JM: [We] lead by example. So you demonstrate to the engineering organization as a whole that a particular process or development methodology is working. That will encourage others to follow. We identified small teams that we wanted to try the new processes with. That turned out to attract a lot of attention and encouraged other teams to look towards those initial teams as an example of how we can be more efficient and more agile.
GA: We have what we call pods now. We set up a couple of teams that were successful, and productivity and reliability and the ownership and pride developed. Everyone could see it. Eventually we had buy-in and everybody adopted the process in the past few years.
JM: A pod may consist of some developers, some QA [quality assurance] testers, product owners, maybe some UI designers. Together they have a common goal or set of goals that they are responsible for delivering.
GA: The feedback loop is very fast. If something is going right, fine, let's put it in production. If there needs to be iterations of a concept, they work together until it is right and put it in production.
TT: How do you protect quality in such a fast-paced environment?
GA: Everyone understands what the project is from its inception. While engineers go and build it, QA folks at the same time are trying to figure out how to test it and are writing a test script. As the developers are putting out code, the testers simultaneously are testing it. As soon as something is not working it goes back to the engineer and he or she fixes it. It is a fast feedback loop.
We believe in tooling. If we see there is something wrong or a bug made it into production we have tools that immediately roll it back. When we release some new software we do not give it to 100% of customers at once. We test on internal people in Comcast. When we are confident, it is given to a certain percentage of our customer base. And in a week or so it goes to the entire customer base. If we find some bug or a process that doesn't seem to work we immediately roll back to our previous software. Tools don't only help us to test the new software but also help to make things better for our customers.
TT: Is working in this environment different than traditional development work structures?
JM: [Working in] cross-functional pods is fundamentally different than being assigned work and being responsible for only a small limited scope of work. In a pod you are interacting with the product owners, with the user interface designers, with the quality assurance engineers. You are providing feedback to quality assurance engineers on the software you are testing, the nature of the testing. You are providing feedback to the product team on how to approach the design and development of the product itself.
GA: It is good to have people who are used to cross-functional disciplines and have worked in them before, but it is not necessary. We provide a lot of support and training internally to help them work in a cross-functional team. We have traditional teams within our company and we have trained them and they smoothly transitioned into the DevOps world.
TT: Are the actual skillsets different, or is it the same skillsets executed differently?
JM: It is the same skillsets. We are talking more about a team and how members of the team interact with each other. We found that engineering teams embrace working closely with the product and QA teams. It is not a different set of skills but a set of behaviors that are adopted and embraced by the team.
GA: When they are working in these pods or cross-functional teams they tend to…learn the skills of their peers. At some point some of the skills become interchangeable. They can do multiple things. When they work independently they only focus on a certain aspect of software development.
People in the teams can have their view and their views can be heard and according to that the product can be changed and become much better and stronger…Not only the teams become stronger, but the product becomes stronger and more reliable in the process.
GA: It's an organic thing. It grows. We see what works and what doesn't work. We are constantly refining the process. We are faced every day with different issues and different problems we are trying to solve. So it's a living being.
JM: We are really happy with the progress we've had over the years with the X1 platform and with how our teams have embraced the DevOps model. We truly feel that DevOps provides fundamental value in improving the working relationship among members of the team and also provides a higher quality product in the end. They are all excited every morning.
— Carl Weinschenk, Contributing Writer
