From Sysadmin to Developer to Solution Architect at Red Hat
We also talk about:
- her experiences as a sysadmin,
- how she learned to program,
- and how she transitioned into becoming a solution architect at Red Hat.
- She also shares why she has a wall of different certifications,
- and started a bunch of different learning circles and communities that help people learn to program and reach their goals.
Michaela’s Code Review Workshops
Angela Andrews, is a solution architect at Red Hat. Angela is a curious learner who has worn many hats over the last +20 years in the tech industry.
Other episodes you'll enjoy
Read the whole episode "From Sysadmin to Developer to Solution Architect at Red Hat" (Transcript)
[If you want, you can help make the transcript better, and improve the podcast’s accessibility via Github. I’m happy to lend a hand to help you get started with pull requests, and open source work.]
Michaela: [00:00:00] Hello, and welcome to the software engineering unlocked podcast. I'm your host, Dr. McKayla, and today I have the pleasure to talk to Angela Andrews: a solution architect at RedHat. But before I start, I wanted to tell you that I have added new dates for my code review workshops. In these workshops, you learn how to get the most out of code reviews. That means you learn how to make them fast and effective at the same time. In the workshop I share with you all my knowledge I gained over the last 10 years. I worked with engineers at Microsoft, National Instruments, Metro Systems, Wix, Automatic, Flutter, and many others to make code reviews their super power. I'd love to also work with you and your team -- so check out my brand new website address: https://awesomecodereviews.com.
But now back to Angela. Angela worked as a SysAdmin for several years, but recently transitioned to become a solution architect at Red Hat. Today, she shares with us why she has so many certificates and what it means to be a solution architect and her interesting journey within tech. So, I'm super thrilled to welcome Angela. Angela -- thank you so much for being on my show!
Angela: [00:01:07] Well, thank you Mikayla for having me. I really appreciate it.
Michaela: [00:01:11] So Angela, how does this all start for you? How did you, you know, when did you put your first toe into tech, or your foot into tech, or something like that?
Angela: [00:01:21] Well, God, this started many years ago, so I started on a help desk and then, you know, at a really small company. And then I turned into a network admin. I was really a sysadmin who did network stuff, and then I became a, you know, changed jobs, became a full fledged, sysadmin. During this really long stint, I tried a lot of different things. I thought I wanted to be a developer, so I learned it and then I realized I was a better sysadmin than a developer, so then I learned and pivoted, learned a lot of cloud and other products and processes and became a solutions architect at a software company. So that's where I am right now.
Michaela: [00:02:08] Yeah, yeah. And do you want to share which company that is, where do you work right now?
Angela: [00:02:13] I work for Red Hat, and they're an international company. They're all over. So, uh, yeah, that's, I've been there since May.
Michaela: [00:02:22] Okay. And so the solution architect role was the first time that you had that was now this position at Red Hat, or have you done it before?
Angela: [00:02:31] Nope, this was the first one. I've been a sysadmin for the past 15 years.
Michaela: [00:02:37] Okay, so cool. What does a solution architect do? What does your day to day look like?
Angela: [00:02:41] Hah! So my day to day is a lot of learning. Because I'm new, I'm learning the Red Hat portfolio, learning how things work, using labs, building my own clusters and installing services, understanding how they work. And, what I do is I help customers devise solutions to their technical problems, you know, be it moving into cloud native development, or helping people organize and modernize their infrastructure is basically what I do.
Michaela: [00:03:17] Okay, very cool. Yeah so, before I actually left Microsoft, I was looking at some different positions there. And one of those positions were also solution architects, or very customer facing positions, which are similar to what you just described, but this would have involved a lot of travel. So, how does that work right now with the pandemic, how do you work with the customer? How do you work? You know, how are you a solution architect with these travel restrictions and things like that? Does it all work remote now?
Angela: [00:03:48] It's all remote. So, there is no travel. My job usually during non pandemic times would be a lot of travel, I'd be visiting people, and because of the market that I work in, I'd be visiting state and local governments and educational institutions in health care. So I'd be all over, but now everything happens via remote conferencing.
Michaela: [00:04:13] Yeah, yeah. And does it work?
Angela: [00:04:15] It does, it really does. You can do more meetings in a day, which is great -- I think the volume in which we can reach out and help people is great, but, being hands on is something that is, you know, I can't wait until it gets to that point because there's a different dynamic when you work with people face to face, you know, meetings can go over, you can, you know, whiteboard, you can brainstorm, and people are more comfortable asking questions in their environment because they're in their office on their turf, and then you can invite more people in, because usually on these calls, what I find is there's always someone missing from the conversation. And, you know, not to say that it can't happen, you know, when you're on prem, but I think being accessible is something that we're working around, but I think that's kinda missing, and I think everyone looks forward to the day when we can get that to more face to face.
Michaela: [00:05:22] Yeah, so have you been traveling before as well, so extensively as a sysadmin, or I guess --
Angela: [00:05:27] No, my travel was basically to conferences and to local meetups and things of that nature, and that was it, you know -- it was a few times a year versus, you know, being a solutions architect, it could be a few times a month.
Michaela: [00:05:46] Yes, yeah, I can imagine, yeah. So, have you ever actually experienced this travel, this heavy travel workload? Or was the pandemic already hitting before, right?
Angela: [00:05:59] No, I haven't had the opportunity to do any travel for work. I started during the pandemic, so right in the middle. I decided to step out on faith and take this job because it was interesting and it was different and I can learn all these technologies and I can travel. So it was, it seems like the perfect job at the perfect time -- just professionally, it seemed like the right thing to do.
Michaela: [00:06:29] Yeah, I think it's also cool that now you can really focus on learning everything and don't have the added stress of traveling, then once you're already confident and know your stuff, then you're starting to travel. I don't know -- that's at least how I imagine it, right now, but uh --
Angela: [00:06:44] Well, just think I'm in this house all day. And once, you know, whatever time my date ends, I can still sit here and learn. I bought a server so I can have a lab environment at home. It sounds like a plane taking off in my house, so I can't leave it on, and of course I use, you know, virtualization on, you know, a personal machine, so I've always been into learning and testing on my own, but I have so much more time to do that now, which is amazing! So, I'm studying now for my RH CSA, which is the Red Hat certified solutions systems administrator, I think. Yeah. So that's the first entry level certification at Red Hat, and all solution architects have to take it. So that's what I'm doing along with learning -- a whole bunch of other stuff. So this is perfect.
Michaela: [00:07:40] Yeah, I can imagine! It's not only the commute -- I mean, you have a long commute to your clients, right? And you're saving all on that and can actually put that into your development and your learning of things. So yeah, that sounds really cool. But so, you haven't been a solutions architect before, and you just applied for this job and they took a chance and hired you without prior knowledge or prior experience -- is that how it works and you're learning on the job?
Angela: [00:08:09] Well, what I'm learning is their technology. I've been technical for over, about 20 years. Being able, what a solution architect does -- Yes, it's technical, and yes, I'm technical. But what I'm doing is I'm communicating, and I'm listening, and I'm thinking, and I'm devising and I'm doing more than -- and I don't want to say anyone can become a solutions architect -- but what it takes, the skills in my opinion, that it takes is that you have to be a very good listener, you have to see what people are saying. Sometimes they don't know what they're saying, so you also have to be a good interviewer. You have to be able to ask those questions that get the information out of them. You have to have patience. You have to have emotional quotient. You have to be able to read people and communicate with people. So, there are a lot more, I wanna say soft skills, in my opinion, in being a solutions architect than there are technical -- because if you're technical, you can learn new tech. That's not the issue. But being a communicator and facilitating and helping a customer get to where they need to be technologically -- that's the skill in my opinion.
Michaela: [00:09:38] Yeah, I totally agree. I also see it. I see it that way, so very cool! I hear already that you're learning a lot and I also, when I looked at your website, I saw so many certificates. So, why do you do the certificates? Is it / was it required on your job, or is it out of passion for yourself or how does it, how does this happen?
Angela: [00:10:03] Well, it was never required by my job. It was never a term of employment. What I found in getting certified in certain things was that you learn this very baseline information that the vendor wants you to learn, right? So take, you know, LPI Linux, you know, if you want to learn Linux, just the essentials, just enough to understand what Linux is and to use it as an operating system, you know, they've developed this certification. So you study it, you learn what they expect you to learn. You take it and you pass it. I did the same thing for, you know, the AWS certified solutions architect associate. We were using AWS as our cloud provider. We also ran a private cloud using VMware. So what these companies do, I said well, I'm using it every day -- I'm using it in certain ways, but they've developed these certifications that they want people to know, at least this much -- we want to give you this much information about our services and how they work together, and I think studying for certification and passing it is an amazing way: one, to level up; and two, if you're really interested in any particular technology aside from using it and getting comfortable with it and, you know, seeing how it works, getting hands on, getting a certification is a really good way because you can't learn it without using it. So, it really does answer those two questions -- like, you get your hands on, yes, you do! Do you learn what they're expecting you to learn? Yes. And then what I think it does, it really broadens, you know, the product for you, because I only used certain AWS products, but studying for the solutions architect, I used products that I'd never seen before, so it wasn't a learning experience for me. Same thing with the security plus -- I studied for it, started taking practice exams, and then I got this job. So now that I got this job and I'm doing all this learning for work, I had to stop studying for my security plus. I had to stop studying for my GCP ACE exam. It's an associate cloud engineer. So I had these two things up in the air, like I'm so close! I'm taking my practice exams. I've gone through all the material and then I get this job and it's like: oh man, but I won't have enough time to, you know, finish before the actual job starts, because the job is learning. I'm going to pick them back up though!
Michaela: [00:12:50] They are a little bit on hold now and then they're going. Yeah. Yeah. Well, one step at a time, I would say!
Angela: [00:12:56] One step at a time -- and they're paying me! So I'll do, I'll take their Red Hat exams obviously. And it's fun! I'm having a good time learning. We have a study group at work that are getting certified -- a bunch of solutions architects, so it's actually really nice.
Michaela: [00:13:12] I think it's also nice that with these certificates, you have somehow a roadmap, right? So you're not out in the open because you could probably learn the same thing, getting different tutorials and different, you know, having yourself, setting yourself as different challenges and things like that, but you don't know if you have actually covered everything. There's a lot of work also to prepare that you covered everything and you did everything in the depth that you should and so on, so somehow this gives you a roadmap to let you know, actually I did it, right? It's a little bit like a checklist -- do I know everything that I should actually know?
Michaela: [00:13:44] And, so for the certificates that you did, how did you select them? Because, I mean, I don't think I have a lot of certificates -- well I mean I have a few, -- I mean, I did university, right? (That's like my certification) But I didn't pick up a lot of certifications outside -- it was never required by my job, and I did a few things like ScrumMaster, but I wasn't really -- I didn't really feel like this is as useful as it should be, and then I didn't like the machinery behind it, because if you get certified as a ScrumMaster, right, you're only certified for a certain time and then you are not certified anymore, and it felt like, I mean, I know people that took it, it's more like you're paying that money and then you're getting a certification, right, so this wasn't something that I wanted to be part of right? So, I probably did have really good ones right, but a lot of them just have to certificate, you paid $2000 or whatever euros, right -- they got their certification. Then they are ScrumMasters. And if they don't pay it (my husband had one, right). If they don't pay it in two years time, then they're not ScrumMasters anymore, right? And so I felt like, okay, this is, I didn't believe into it, right. It felt like this is not really fair, you know? You are it because you know it or you're not, right? It shouldn't be bound to the money that you're currently paying. I don't know, how are those other certificates working? Is this the same thinking -- I understand also that if technology changes, right, you have to recertify yourself because otherwise you're outdated. But for ScrumMasters, for example, I found it was very silly with soft skills basically, right? So --
Angela: [00:15:27] I agree. With the, I'll use the AWS cert as an example, once you take it, you are now certified for three years. Now, if you know anything about AWS, you know, it changes at the speed of sound. Every reinvent, their annual conference that they have at the end of the year, they produce, you know, a bunch of new services and how things work together. So at that pace, you know, your cert that you've taken -- you might want to level up and learn the new tools. So I'll tell you this. I took mine in 2019 and they just changed the cert and I looked at it and I'm like, wow, this is, it's a bunch of new stuff! So, for something that changes as rapidly as technology, meaning, you know -- I'm now an AWS certified, you know, engineer. And, but now they have, you know, a hundred new services that are a really big part of their product line, and I don't know what else is there to do other than, you know, get back in there and recertify. So for something like that, I get -- you have to stay on the pulse of what the technology is.
Michaela: [00:16:47] But when you get recertified, do you really have a test again? Or --
Angela: [00:16:52] Yes, you do --
Michaela: [00:16:53] -- is it just paid? You know, because I think these are also two really different things, right? So either people are recertified by they're really proving that they know again, their stuff, or, you know, getting recertified by paying, which I find a little bit --
Angela: [00:17:03] So I think different vendors have different means for staying certified and recertifying. Some will ask you to take continuing education credits, which means during the course of your certification, you're constantly learning the stuff that they're putting out, or someone's putting out, and you're obtaining this credit. So you're showing them that you're invested in this technology and staying informed about this particular vertical -- that you you're going to do, what it takes, so they sometimes ask you to take, you know, you need to have this many continuing education credits, or you have to recertify. Now that again, that doesn't bother me either, because if this is something that you're interested in, you should want to stay ahead of the curve and learn the new technologies and take those workshops and things like that. So I think that's very important. And the ones that are just asking you for more money, just for a cert, to each their own -- there may be something to that, especially if you're a career transitioner or you're trying to pivot into something, those certifications are the fuel in your tank to get you where you need to go. But once you're there, you know -- you can kind of say, well, I don't feel like I need to take this exam again. Maybe there's something else that, you know, kind of is in the vein, but it's with newer technology with newer understanding. Maybe that's what I'll go after. So, I think it's a very personal decision on why people decide to take certs. Sometimes their jobs just require it -- like mine does now. I'd never been in a job that had required me to get these certifications. I did it because I was curious. Now I'm doing it because yes, I'm still curious and they're paying me for it!
Michaela: [00:19:02] Yes, yeah, I think that certifications, or taking courses, right, I took a lot of courses and learned a lot of things, and certification somehow stand out for that, showing that the person is actually very curious, right, they're a learner. And so even if you did a certification and it's not valid anymore -- maybe you don't need the technology, but it shows something right, that you learned that at that time. And I think even for outdated data technologies, we are learning so many things that you can, you know, from the basic mental models and from the concepts that you can actually extract and then apply to something else, right? So, I think it also helps you speed up for new technology, right? So, your past experiences and technology, I mean, it evolves, but it's based on the same principles right?
So, yeah. I wanted to talk still a little bit more about the different hats that you've worn, and also I know that you're super passionate about security, and that you have been coding, right? So you briefly mentioned that you have been coding. How was that part of your experience? Why did you start to code? Which languages did you start to code? How did you learn it and how did you decide that you're not going to go that route further, but stay in the admin part?
I'm just going to back up a little bit. So that was in what, 2018? And, you know, then I started getting into -- I had this little coding meetup at my house. This started in 2017: so every, you know, every month or so a bunch of women would get together and come to my house and we would code. So one day I said, oh, does anybody, you know, want to deploy a server on AWS? I'm the only Linux person. No one knows Linux. No one really is delving into deploying on a cloud. So, we're sitting there and I'm teaching these women how to deploy WordPress into an EC2 instance. First time on the command line for everybody -- it was crazy how much they learned. Well, someone took a photo and posted it on Twitter. And this guy who runs a local meetup, WordPress meetup -- he said, oh man, could you do talk for us? And I was like, well, it wasn't a talk. It was just a bunch of friends. So I developed this talk and put it on for this amazing meetup in a Philly WordPress group, and then I wind up doing it as a workshop at a conference with one of the women that was in the group (she was a WordPress developer).
That was me still working in this dev space, and then, you know, cloud got real, and I really wanted to get my AWS cert, and I did that. And then I started this group in the beginning of the year, 'cause I was learning Python last year with a fella. We would meet on Zoom once a week. And it was like, wow, boy it would be nice to get more people in our group! So I took to Twitter and I went, who wants to join this group? I got so much response. So now I have this, you know, weekly group that meets on Zoom. Four different time zones, four times a week. Different people, different group leaders, all learning Python. So it's a group called Python for All. It is amazing because you know, we've got people on the West coast and you know, down South, we've got two East coast groups. They all meet at different times and everyone's learning Python together. And the reason it's called Python for All is because Python touches everything. Python touches dev ops, it's touching configuration management, it's touching ML, it's touching data science, it's touching cloud, it's touching everything. So, Python for All, even, such as security. So I started being a teaching assistant and a substitute teacher for a cybersecurity bootcamp in Philadelphia last year. And Python was a huge, it was a part of it. You have to learn how to write scripts to do certain things and understand coding and what code looks like, and how to break into web servers and things like that. So, being able to use code to solve problems is not just for developers. That skill goes through many different career paths. So, that was my awesome foray into getting into cybersecurity. And then, you know, more people want to learn how to get their cert or learn the background, get certified. So, I started a security plus study group. And it ran for X number of months, and everyone's now getting their certification and passing! And, of course, now that I have my new job, I haven't been able to finish, but again, I felt really happy to be able to be a part of this and watching all these people just: "I got my cert! I passed my security plus!" That's where my life has been in the past couple of years.
Michaela: [00:27:54] So very community driven also, and a lot of people --
Angela: [00:27:59] Community is everything.
Michaela: [00:28:01] Yeah, and I can really see that you're like, you're glowing a little bit when you're talking about it -- it's really awesome. Have you done that mostly -- you said Zoom -- so it's mostly remote, but you also had like this meetups where people are coming to your house, right? And you have been a little bit in your, in your local community as well. And does it give you the same energy boost, like this remote thing, or is it more when you're with the people in the same room, in the same city that you can connect, that you're getting this energy from --
Angela: [00:28:35] It comes from both. I think, just connecting with people -- I guess it would be harder if I didn't know them, right? But no, getting to know them has helped. Some of the people of course came in and I knew them -- a lot of the people in security plus were my students. Some people I didn't know at all. They reached out on Twitter. So, I find that remote allows you to build the comradery. I found that people were more open for whatever reason. And I liked them both. I really do. The class was in person until COVID hit (the cybersecurity bootcamp), so I was in class three nights a week, three days a week, and I'm getting to know these people as people and how, you know, helping them maneuver their way into becoming cybersecurity professionals. Am I a cyber security practitioner? No, but my job really does entail a lot of the skills needed to be that person. I mean, I have to know bash. I have to know servers. I have to know databases. I have to know port numbers. I have to know networking. Well, so do cybersecurity professionals. They take that pivot when they're getting into more penetration, and command and control and things like that -- but they're still using the same skills. The same skills that I already have. So, yeah, I think it's great.
Michaela: [00:30:12] Very cool. So what made really curious is that you mentioned you have a kid, right? So when I think about myself, I don't, I mean, my kids are really small, but it's really hard for me to get out and have meetups. We have also a meetup here and I have been, I think. I don't know, in the last two years I've been three times or something. And I was very proud if I made it there, right? So most of my social experience in my community is online mainly because I feel I'm very home bound and still in the house and they need me at night and things like that. Do you want to share, how old is your kid and has this changed over time or, you know, or is it hard to combine? I dunno.
Angela: [00:30:57] My kids are adults! I don't have that problem!
Michaela: [00:31:02] I was really wondering like, how are you doing it? Like you're everywhere. And I'm like, struggling!
Angela: [00:31:09] Well, I'm going to say, my children are probably older than you think. They're 23 and 30. This face is kind of like, very misleading!
Michaela: [00:31:28] It is! Yeah, I found it -- maybe you're straggling like me, but the two or the four year old or something?
Angela: [00:31:34] My dog is four!
Michaela: [00:31:42] Wow, I wouldn't have thought! Okay, so yeah, it's a different lifestyle.
Angela: [00:31:47] Mmhm.
Michaela: [00:31:48] I can see myself doing that when they're out of the house!
Angela: [00:31:51] Yes, of course! Well, because you have more time on your hands. So I will say this, you know, when I started getting into development and programming, my son was in high school. Before then, I would have never been able to do stuff like that. I mean, they take up so much time with their sports and activities and things like that. Me and my husband dividing time behind, you know, this one going one place and this one going somewhere else. It was -- it's hard, 'cause understanding where my free time came when they were younger, I don't think I would have been able to do this as vigorously if they were younger. I mean, I would still try, I'm going to assume that I would still want to try, but it wouldn't have been as easy.
Michaela: [00:32:42] Yeah. Yeah. As I said, I barely leave the house, now with COVID I don't leave the house at all anymore! I'm really happy if I can go into this office and nobody, you know, storms my space. So, one thing that I wanted to ask you, if you still have a little bit of time -- I'm super interested in software engineering practices, right? And I know that now you're now not a software engineer, but you're working with software engineers and you're deploying code and you're helping people really run their systems. So, my question is how do you see different software engineering practices influence your work? Do you see that? Do you see that some practices are better than others, and while you were coding, did you learn about them? And can you now apply that somehow -- we were talking about security, which is one part, right, that people are testing their system, that they are maybe code reviewing their systems. I don't know if you see the impact of those different practices and the rigor people put behind them in your day to day life.
Angela: [00:33:50] Well, not so much as a solutions architect, because again, you know, we don't have privy, we're not privy to the customers, you know, practices, -- when those practices are lacking, we try to point them into the direction, you know, IT optimization -- how can you make things work better in your environment? But as a systems administrator, what I found is, good coding practices are of course, you know, being security minded. So, there's like this three legged stool, I think that, you know, uh, applications and code sit on top of, so developers have a really big part of it because they're the ones that are writing the code. But, what I think they should really take into consideration is at the beginning, not as an afterthought, what is / how am I best securing this code? How am I best making sure that this, what I'm writing, is safe to put out there -- you know, data can't be easily extracted, I am protecting my keys... That's part of it. Another part of it being an ops person is, you know, on this other leg of the stool, you know, what are we doing? How are we protecting the systems that this code runs on? Is the operating system patched and secure? Does it have the right firewalls and ports open? How are we protecting the perimeter? What traffic can get in and out? And then of course there's another leg, and sometimes, you know, depending on the organization, that person could be coupled into another role, but, the security part -- well, how are we going to scan this for any type of vulnerabilities in the code per se, you know, looking for what kind of packages are running inside of this application? So, there's this DevSecOps thing that's going around, and I think that's really a big part of it because you can't do dev without the other, and it goes vice versa. A lot of these coding bootcamps are not teaching their students about security. I think a lot of these tutorials that you see in all these spaces, they're not touching on, "how do I secure my application?" And I think being in an environment, working at a company -- hopefully it's a part of the practices, hopefully it's a part of the culture. So when you, as a new developer, you come into this new organization -- hopefully they have those processes in place that you have to abide by. You have to do things a certain way. You have to test your code a certain way, if you think about it, most security breaches happen through human error, right? You left keys somewhere, or you didn't patch a system or something like that.
So, I think when we are: one, I think more diligent with how we're protecting our systems, and staying up to date with patches is a really big part of it; but two, is making sure that it's always a part of the conversation and never an afterthought. Like don't tack it on to the end, because what happens is, and if you've ever built / wrote any code, you know, the stuff that you start trying to put into once you've finished -- they don't integrate very well. So I think good engineering practices, you know, definitely have someone review your code, everything should go through some sort of code review. Everything should have some sort of testing framework, and testing not just if it looks great or if it's responsive, but can we hack into it? Can we get data from it? Can we do some cross site scripting and get some information from this website? So, I think everybody has to, you just expound where their reaches are just a little bit more. So, developers need to be more security conscious -- they have to also know the systems. Don't write code -- don't write bad code that eats up resources, you know, understand what happens when you write code -- it has an impact on the underlying infrastructure, and be smart about that. And, it goes both ways -- as an ops person, you should understand code, you should be able to read it and understand, well, you're missing something very key here, or be able to offer some suggestions. And of course, security is everywhere. Security needs to understand that they're this blanket over everything because they don't want the intellectual property or the information to get out of the organization. So, having that DevSecOps / DevOps means a lot of different things to a lot of different people, but if you think about it in terms of the roles that people play, those roles cross; they mesh together, and they're always overlapping. Having those practices in place in your organization, and if you don't know / recognize them, if you don't see them, bring them up, because maybe someone else was thinking about the exact same thing. So, conversations are the best way to solve problems, and if something, one thing or the other is missing from your engineering practices that you think should be there, bring it to your team. You might be that person that, you know, has this great idea, and your team can get on board with it. So, it just depends on where you are in your career. It depends on what organization you're in, but you know, learn what you can, and put your 2 cents in where you see fit. You don't know what you don't know, so I don't fault people for that, but if you do know something, it behooves you to bring that experience to where you are and help better the team. Only good can come out out of that.
Michaela: [00:40:01] Yeah, that sounds great. So one of the things that now did you say, well, communication, right. And seeing when things are not going right, is one of the things that came to my mind is incident handling, right? So postmortems of, if there are some incidents in your organization, right, maybe there is some security vulnerability that has been exploited or, you know, maybe there is some attack, you know, maybe the system is down and things like that. And, with all your experience, you've definitely lived through some of those, right?
Angela: [00:40:30] Yes.
Michaela: [00:40:31] What makes a good post-mortem and an incident investigation? What are some of the presets that you think, you know, really help the teams learn from the problems that they made?
Angela: [00:40:44] Ooh, that's a big one. So, I think again, you don't know what you don't know. So, having great logging to begin with, make sure you're monitoring your environment. This is before any incident occurs. So, you have some sort of, you know, some sort of seam, which is a system event management system where logs are going, so you can capture the type of traffic that's going in and out of your environment between systems. That is even before anything, any breach occurs, but for good incident management, I think, you know, because it's not: "Oh, well, have I been breached?", but you know -- at some point there's some sort of breach somewhere, right? So I think a good postmortem -- there is no good post-mortem -- but I think good tips would be is to, one, ask the question: why something so and so happened? Why? Well, because we didn't do, we didn't patch this vulnerability. Well, why? Well, because whatever. Well, why? -- and I've been in one of those where that constant question kept coming, and you get to the root cause. And it doesn't feel good, but nine times out of ten, sometimes it's human error, sometimes a process breaks -- you don't even know that it's broken, and something goes awry -- but being able to monitor your environment and make sure you have good logs, make sure that, you're penetration testing your own environment, you know, hopefully there's someone on board that can do this type of thing. That's why security is basically everybody's job. But hopefully if you have a security department, they're doing those kinds of things to stay proactive against it. So they're putting honeypots in and they're doing vulnerability scans and they're pen testing their own environment. So, they're trying to get ahead of any type of -- they're doing the things that the nefarious people on the outside are doing, but they're trying to do it on the inside to see, oh, we need to plug this hole, oh, we need to plug this hole. So, that was one of the most uncomfortable instances in my professional career. And you know, I have a room full of professionals sitting there and having this question asked over and over, but what it does is it exposes, you know, well where you went wrong, you know, and it helps you the next time. So there is no -- it's not a bad thing. It opens your eyes, and then you decide, well, how do we prevent this from happening? And you develop those processes and you stick to them, because you don't want to be in that hot seat again. So, I think incidents response is not comfortable, but what you get out of the end of it -- hopefully, it's more information, it helps put you in a better security posture, and it gets everybody on board because now you're talking, now you're understanding what your impact has on the environment as a whole. Sometimes you don't even know what your impact holds, but something like this will definitely tell you that you do hold a lot of power. So I just think it's funny because it was one time, and it had to be over 10 years ago, and I'll never forget it. Yeah, you're forever shaped.
Michaela: [00:44:27] I really like your focus on why, because there are also people that are focusing on who, right? So these are two things, and sometimes probably go hand in hand, but the question is what is the bigger one, right? What is the bigger question? Is it the why? Or is it the who? And also, I think, I don't know, maybe the last four, four months, four months ago or something, one of the big CEO's on Twitter that I'm following, he was sharing that -- they had like a $1 million incident or something, right? And so, some of the questions were: was the guy fired, right? And he was, why would I fire a person that just learned a $1 million lesson, right? I mean, this is the last, or the last person that I will find any situation, which I found a really nice mindset, right? So, we can learn from this, it happened, right? Probably, very often, it's human error, so there will be a queue somewhere, right, a person that has made a mistake, but often it's a chain reaction, right? It's not one person doing a mistake, but maybe several things to bleed together, so probably one person doing something, but there is a -- yeah, it's part of life so that we can actually learn from it together. But yeah, I like that you put this "why" more in focus than the "who" -- have you been in situations where there was this "who" a more bigger impact and where people were afraid of, you know, being the person that is the "who", and being fired off of making mistakes?
Angela: [00:46:00] Well, the "why" will always get to the "who" -- that's just how, because everyone knows their role, and that role in the incident tends to expose itself. So, if you're a part of a team, and, your job is part of what caused the incident. Trust me, at the end of those questions, you and your heart, you're going to know. And then, everyone else is going to know. There was no moment -- and again, like you said, sometimes it's multiple. I don't think anybody in my group felt the slightest bit fearful or threatened, because it just happened, and it can happen, you know? Yes, it's usually human error, but why? Again, the why -- so that why could be a process, or another task, or another project, or, well, what are you doing? You know, what's going on in the organization where you hold this part of this person's job into such high esteem where you say, "Okay, but wait, I need you to direct your attention here for a second."
So, it has a lot to do with not just the people, but the processes as well. At the end of the day, there's always something someone did or some process that failed, but then, what have you gotten at the end of it? Well, you've learned that, no, you shouldn't put off patches just because it's an inconvenience to a developer because they don't want this system to break, right? So you -- and this is just an example -- but, nine times out of 10, there's always a reason. There's always a reason, and it could be lack of training, it could be workload, it could be something broke somewhere else and this was like a stop gap or something. So, I think it's just anything that happens, you know, sometimes it is a million dollar mistake and I don't think, unless it was done on purpose, anyone should lose their job for something like that. I think because we're human and humans make mistakes, but humans tend to learn from them as well. So, you have to give people that type of grace and that leeway. I've never worked in an organization where -- and I think I'm very lucky. I've never worked somewhere where I felt, "oh my God, if I make a mistake, they're gonna fire me." No. So, there's a motto in my company: "Fail fast." If you're going to make a mistake, do it, get it out of the way, 'cause you've learned -- you've learned from your mistake, and now you've learned about 50 million ways to get it around it. Fail fast, get it over with! I agree.
Michaela: [00:49:03] And I think also there's this idea that's for example, proclaimed also by Google and other site reliability engineers, right? Where you say, well, if you find an arrow that can be made by one person, right? Then you're working around that, you learn from that so that this arrow can actually not be made by a human anymore, right? So it shouldn't go through if one person makes a mistake, right? This should probably be something with your process that's not completely correct, right? Or something that you could actually work, and that this problem, or this mistake, that one person makes doesn't cost a million dollars, right? There's something fundamentally wrong. It's not really the person that made this mistake, right? It's their organization that made this $1 million mistake.
Angela: [00:49:49] Right. And not every company is Google. So not everyone has the infrastructure to design that type of high availability or reliability into their systems. So, you have to build your systems as robustly and securely as you can, with the budget that you've been given. So, not everybody's Google, not everybody, but if you have that mindset, I think that goes very far -- having that mindset, that one person can't sink the ship, is a great way to look at it. And you try to do what you can in your organization, that is not of that scale, to try to circumvent those types of issues.
Michaela: [00:50:34] Yeah, that's a very good point, yeah. Wonderful! So, thank you so much, Angela, for being on my show today. I don't know if you have anything that you haven't addressed that you want to share with the listeners or people on YouTube that are watching?
Angela: [00:50:50] Oh, well, I would say, you know, find your passion. Tech is a really big field. It is not just developers. It is not just front end, backend -- it just spans so much, and I've done a lot of it. I think, like I said, I've worn a lot of hats, you know, systems and help desk and web development and, you know, a solutions architect and cloud, and... So, find where your interests lie, and go after them, and don't feel upset if you find something and say, "Well, I'd rather do this, or I'd rather integrate more into this." That's totally okay, because I think that curiosity is what makes you more marketable. 'Cause it worked for me -- I think because I'm so curious, that's what helped me get my new job, because anybody can learn certain technologies. I think that's great, but how you learn them, and how you communicate them, and how you put them into practice -- that makes a really big difference as well.
Michaela: [00:51:58] Yeah, I wholeheartedly agree. Thank you so much for being on my show!
Angela: [00:52:04] It was a pleasure! Take care of yourself.
Michaela: [00:52:06] Yeah, bye bye!
Angela: [00:52:08] Bye bye.
Michaela: [00:52:09] I hope you enjoyed another episode of the Software Engineering Unlocked Podcast! Don't forget to subscribe, and I'll talk to you again in two weeks. Bye!
Copyright 2022 Doctor McKayla