CenturyLink's Newkirk: DevOps Culture Puts Customers First

By creating a collaborative company culture, DevOps is allowing CenturyLink to put its focus on creating value for its end users. The move to DevOps has wide ramifications across a company. For CenturyLink, it means that developers must move from an approach in which their sole concern is their small piece of a project to one in which their focus is fully on the ultimate goal: serving the customer.

Other features of a DevOps environment are fast turnaround times, group responsibility -- to the extent that people are on call at certain times -- and the elimination of silos. It also means that the type of developer CenturyLink Inc. (NYSE: CTL) is now hiring has changed. The company wants developers who exhibit "a spirit of collaboration and cooperation."

Changing a company from a legacy approach to a DevOps strategy is a tall order. (See CenturyLink CTO Hussain Maps Out Virtualization.)

James Newkirk, CenturyLink's Vice President of Platform Engineering, recently responded to questions on the changes brought by DevOps.

Our conversation with Newkirk is part of a series of Telco Transformation Q&As on DevOps, all of which will largely feature the same set of questions. (See DevOps: AT&T's Saxena on Building a 'One-Team Culture'.)

Telco Transformation: What cultural transformations need to take place in order to implement a DevOps mindset across the entire workforce?

James Newkirk: The biggest mindset shift for the organization is to take a more customer-focused approach to delivery, and begin to change what they do to reflect that. Typically, developers are mostly worried about completing their code and "passing it over the wall" for it to be tested and deployed. And the operations staff are used to putting code through a formal testing process that can take days or weeks to complete, and "tossing bugs back over the wall" to be fixed and re-entered into the process. Both sides can be more concerned with completing their own pieces of the delivery puzzle than focusing on delivering value to the end user.

The big transition comes in deciding that shipping customer value is the most important thing, which requires the two sides to work together instead of focusing on their own contributions. This change has to involve the leaders of their departments, since Dev and Ops usually report up to different VPs or directors. They have to ensure that a spirit of cooperation and trust is built at that level, and that the two of them are committed to working together to optimize the experience for end users rather than for their own departments. That spirit has to flow down through each of their organizations, to the point that the developers and operations staff both take ownership of end-to-end quality: they work together to test and correct defects, learn to automate as much of that workload as possible, share pieces of the support load, and learn to work together as part of the same team.

This seems like a simple transition, but it is counter to the culture in many large corporations, where support/test and developers are separated by organization and geographic walls. To allow it to be successful, the metrics on which each department is judged may need to be changed away from measurements of each group's efforts, to the delivery of end-user value.

TT: How are employees being trained for new services and applications?

JN: As new products are ready for release to production, they have to address a checklist of items from our support engineers to make sure that the product is ready to be supported. These include having the appropriate monitoring, knowledge-based articles for the support engineers to reference as needed, training for the entire product team in how to engage with our support engineers, alerting and monitoring systems are integrated with our support engineering systems, and several smaller items.

The support engineers are trained for each of the products by receiving overviews of how each product functions prior to that product being released, and by being provided knowledge-based articles that describe the operation and support processes for each application. Some teams include a support engineer as a stakeholder in the product planning process, so that support needs are directly included during the construction of the product.

TT: How are new employees being recruited?

JN: The actual process of recruiting hasn't changed, but we definitely look for different things. We specifically screen everyone with an eye toward a spirit of collaboration and cooperation. Our organization has its own set of values, and these are considered as we hire. These (values) drive us toward hiring people who are strongly customer focused, understand and appreciate the agile way of working in small pieces, and are comfortable with the quick deployment cycles that our products demand. Our support model requires all developers and engineers to be on-call a week at a time, a duty that rotates through each team, and this requirement is explained to everyone during the process. Our operations people are expected to be able to script and interact with the product teams to solve customer issues as they arise, 24/7. All of this aims us towards recruiting and hiring people that fit within our culture of DevOps.

TT: What is the impact of DevOps on breaking down service silos? Did that lead to the creation of cross-disciplinary groups?

JN: Adopting DevOps values has definitely reduced the number and depth of silos that we have in our organization. On product teams, individual developers and engineers, are forced to eliminate silos because each one of them is expected to be on-call for the operation of their entire product periodically. Any silos of knowledge that exist on those teams can manifest themselves in a reduced customer experience should a team member be paged for an area of their system with which they are unfamiliar. Our support engineers are similarly forced to understand the operation of each product at a deep enough level to be able to begin troubleshooting as customers report incidents, both through their own knowledge and experience, and through an available knowledge base that documents operation and troubleshooting for each product.

This knowledge base is created by both operations and engineers as a way of sharing information that allows the support engineers to solve incidents more quickly than involving product team members. Finally, representatives from every team, both support and product, attend a daily standup meeting that focuses on operational incidents and upcoming production changes.

TT: Do you have some examples of how DevOps has changed, or impacted services and applications?

JN: The biggest effect of DevOps on each product is the depth of monitoring and alerting that product teams build for their systems. Each of these tools automatically notifies the support engineers should significant events occur, allowing them to begin our incident handling process to address issues before they impact customers. For specific kinds of events, these monitoring and alerting systems directly page the on-call engineer for a product team, allowing mitigations to be put into place even more quickly.

Our support engineers have embraced ChatOps with [messaging tool] Slack to create automation about spinning up the necessary infrastructure to handle customer-impacting incidents. These tools allow incidents to be started more quickly, and prevent errors and miscommunication during the initiation process.

Everyone in our organization is trained in the incident handling processes, including how to initiate and participate in incident support, the proper use of our ticketing system, and the SLAs we have committed to for our customers.

Our values of transparency and honesty are clearly seen in our practice of holding a retrospective very shortly after any issue that required starting up our incident response process. Everyone involved in the incident is invited, appreciations for positive contributions are given, and action items for improvements from any team (whether support engineers or product teams) are created for that team's backlog and implemented so that our incident response process continually improves.

Check back for more Telco Transformation Q&As on DevOps to learn how other companies are implementing and utilizing this method for rapid innovation and delivery.

— Carl Weinschenk, Contributing Writer, Telco Transformation