DevOps: AT&T's Saxena on Building a 'One-Team Culture'

Historically, teams have focused on developing a market-ready product first (Day 1), then focusing on operation and service management needs (Day 2). In our new approach, we are working the service and operational needs simultaneously (Day 1 and Day 2 requirements both prioritized from the start). We needed to adjust our culture from being artifact- and process-driven to tools-based. This required a shift in individual behavior and an overall change in our collective norms and behaviors.

Instead of doing serial creation of services components that resulted in operations needs getting "lost in interpretation," we prioritized them upfront in the scoping process.

Most importantly, we implemented a one-team culture. We have the product manager work the entire lifecycle with the ops teams, working together in a highly agile and collaborative manner.

TT: Has this transformation been difficult to achieve? If so, why?

SS: Yes. It requires people to question each other, make suggestions that they would not have made in the past, and fundamentally change their day-to-day working model and relationship maps. Instead of producing one document at a time and "throwing it over the wall," the one team culture requires asking questions in real time. In parallel, we also colocated several development and operational teams. In our primary collaboration zones, AT&T has invested in a highly interactive work space called Workplace 2020. Tools like ambient Telepresence (always on) or AT&T Collaborate for voice, video and IM collaboration are widely used for cross collaboration zone communication.

TT: How are employees being trained for new services and applications?

SS: AT&T's Vision 2020 has workforce transformation at the core. Our CEO Randall Stephenson has made participating in corporate education a pillar of AT&T's strategy. We've invested significantly in laying out multi-year structured programs. For "Capabilities Evolution," the name of our skills pivot program, the four main pillars include: sponsored corporate wide online and teacher-led programs through AT&T University, collaboration with universities like Georgia Tech, technology team-driven training, and live technology boot camps.

We have best-in-class training provided by AT&T University that spans all business units. And we recently created "Path to 2020" programs. These provide a long-term approach to reaching our overall 2020 Vision. Our collaboration with Georgia Tech's Masters of Computer Science program allows our employees access to one of the most renowned online programs to gain technical skills. We collaborated with Udacity to create "Nanodegrees," which has a focused curriculum to augment rapidly your technical abilities. In some domains, the experience and knowledge of our expert team members is the greatest asset we have. (See Advancing Cybersecurity Goes Beyond Tech and Fuetsch Shines a Light on AT&T's Digital Transformation Plans.)

Team-driven technology training occurs all across AT&T and we encourage this knowledge sharing. AT&T has coordinated boot camps to do simulated live workshops for emerging technologies and methodologies. For example, the DevOps Bootcamp provides a gaming technique that provides real-world experience, making teams work together in an open, trusting environment. By implementing training across these different avenues throughout the company, AT&T is pivoting its workforce in an innovative and unique manner.

TT: How are new employees being recruited?

SS: As one of the country's largest employers, we understand the workforce that develops, sells and maintains the technologies is one of our greatest assets. The recruiting landscape has changed. Companies rely heavily on innovative ways to both attract talented people who are passionate about technology, and retain them throughout their careers.

While we still engage in traditional recruiting strategies such as job fairs, online job postings and employee referrals, we have realized the value of real world settings that highlight candidate skills, like Hackathons and internships. The AT&T Experience Weekend, or AEW, lets us spend a weekend with a group of candidates from top colleges around the country to share what AT&T has to offer, and find out more about what the candidate is looking for. Regardless of how we learn of a candidate, they all go through the same four-step interview process that ensures technical skill and an aptitude for business acumen.

We have also grown and invested our internship and college-hire opportunities at AT&T across the country. Our college-hire program experience is an 18-month cohort. We give members different assignments across the AT&T Technology Development organization, providing an opportunity to explore the possibilities and their passions at AT&T.

Finally, it's also important to remember that AT&T has a large pool of very talented folks already. Our people are excited for the transformation, which also makes them perfect candidates for joining DevOps teams.

TT: What is the impact of DevOps on breaking down service silos? Did that lead to the creation of cross-disciplinary groups?

SS: For organizational silos, the "hand-off" point is the goal for teams, instead of the overall, common goal of delivering a product. DevOps has allowed us to move from a siloed view where each software development discipline (design, code, test, operate) takes the work product from the previous step with minimal interaction to one where we are driving people across all the disciplines to take ownership of the end product. Everyone now has skin in the game for this ultimate goal and are therefore empowered, thus encouraging greater innovation.

We become more efficient by "shifting left" and improving efficiencies through automation. Yet we often lose momentum by working via hand-offs and procedures. With everyone engaged upfront we have the ability to identify issues earlier and course correct. Additionally, teams are required to develop across disciplines. This creates a general understanding of what is required for peers to be successful and to complete work accordingly.

TT: Do you have some examples of how DevOps has changed, or impacted services and applications?

SS: DevOps principles and concepts are helping us to deliver much more quickly. Toolsets that enable continuous integration, continuous delivery, and automated deployments allow us to be much more efficient and ultimately deliver our products more quickly.

Our AT&T Integrated Cloud Team (AIC) team has completely transformed the way we work, through DevOps methodology. We have evolved from scrum teams to inviting and incorporating the operations team in the overall planning from day one. This has led to the "Check Box" -- documentation that verifies Ops affirms and supports the direction the team is taking. Not only has this change resulted in a better product, but there is greater satisfaction overall for our collaborations.

One key thing we learned recently is the importance of completing the Check Box for a change request on a project. Although they have the preliminary approval from business partners for the scope, they found it was necessary to obtain approval again, to maintain good working relationships.

At AT&T, we realize that resiliency in the architecture and every phase after that is essential to offering excellent experiences to our end customers and users of our systems. This has led to AT&T DevOps adopting a third tier resiliency function which builds on root cause analysis, improvement of mean time to restore, and availability of our current applications in production. As we build new applications, we are implementing resiliency by design, which addresses the goal of ensuring applications and systems are built resilient right from the start. DevOps personnel realize that they, too, are responsible for resiliency -- and providing training, certification and improvement metrics to back that up.

Check back for more Telco Transformation Q&As on DevOps to learn how other companies are implementing and utilizing this method for rapid innovation and delivery.

— Mike Robuck, Editor, Telco Transformation