Episode 1

PodCast Feed


Links:
SysAdmin
ocd
bofh
google
apache
ssh
ssh authorized keys
scp
fedora install movie
vmware
camtasia
qemu
wine
keychain
gpg


OS:
linux
redhat
fedora
centos
freebsd
openbsd

RYOS, Episode 1 - An Introduction

Thud: In our first episode, An Introduction, why would you want to run your own server anyway? What does it take to be a good admin? What makes a bad admin? A moment of sec and how to paint your own hamster.



Thud: OK to start off our first show, we would all like to introduce ourselves just so everybody gets a feeling of who we are and the kind of things that we like as far as running servers, what OSes are and things like that. So I am going to start off introducing Gek. Tell us a little bit about yourself and then we will go on from there.

Gek: OK I am Gek, which is short for Gekitsu. I have been doing this for the past ten years or so. Professionally, I have been a sys admin for the past six. I am actually trying to make the transition into more of a development role but part of that involves developing tools for the sys admins that I work with. My OS of choice, at this moment is CentOS but I am known to change. You know, in the next 24 hours, I might like FreeBSD again, who knows. I have used everything that's open source and most things that aren't and I am mainly focusing on learning Python, Perl and Java at the moment and that's pretty much it.

Thud: all right. I guess we will move onto Seg next. Tell us a little bit about your background.

Seg: all right. Hello everyone. My name is Seg. I have been a sys admin for about six years. I like being an admin because it let's me filter packets and be a developer at the same time. My favorite operating systems are FreeBSD and Solaris but that depends on who is buying the hardware. I am forced to administer other things like Windows and Linux and even OpenBSD on occasion but I definitely like sticking with Solaris or FreeBSD. I enjoy firewall and I enjoy writing scripts. I think that about sums me up.

Thud: all right. I guess I am going to have to introduce myself now. I am Thud. I have been doing this for, I don't know, a really, really long time. I have been doing it professionally about seven years. For my favorite OSes, it really depends on what OS is best for what you are trying to do. If you are trying to do a firewall I like OpenBSD. If you are just doing a general web server, sequel database server things like that, I prefer one of the Red Hat variants either Red Hat, CentOS, Fedora. They are all pretty much the same.

Seg: So Gek, why do you want to run your own server?

Gek: Me personally, I like running my own servers just because I am kind of an information junkie and the only real way to get statistics on your email and your web is if you running your own servers and you are collecting your logs and running webalizer or something to get reports out of it. I am just, I guess, kind of a control freak.

Seg: I definitely have to agree with you there. I do run my own server because I do want to be able to read the logs and read the mail. I actually have my own mail server. Running my own web servers gives me so much more control over what I want. It allows me to create grandular access to the information that I host. How about you Thud?

Thud: Well I am pretty much the same way. I like doing things myself but also since I get paid to run servers, I would feel more comfortable screwing up my own server first before I work on a customer's server. So just having the ability to test out new things and learn new software on my own server has helped out quite a bit.

Seg: Absolutely. I definitely needed that extra IP on the Internet to do testing and things at work. People are complaining about connection problems. The ability to go out there and do a trace route from some other IP is definitely valuable.

Thud: Yeah, I have done that a whole lot myself.

Gek: I think it also becomes a standard tool kit for a sys admin. Even if it is not a box that's yours some box that you can get to and remotely run, Trace Route, NS lookup whatever.

Seg: So Gek Good. What does it take to be a good admin and where do you go to learn it?

Gek: I am not the first to ask because I am a bad admin. My first answer to everything is you are no longer allowed to have access to that server.

Thud: Wait a minute. I am a bad admin too.

Seg: I am going to have to jump on that boat definitely.

Gek: I think a good admin is more of a personality style and not really, like I said, a set of skills. It's the ability to look at a problem and analyze its parts not necessarily just its symptoms. As to where you go to learn it that's a tough question. I do a lot of reading. I try and find people who are similar minds. Just talk about it constantly and read about it as much as I can.

Seg: How important do you think imagination is when it comes to being an admin?

Gek: That's vital. [laugh] You have to be able to, if you run into a problem that you have never seen before you have to be able to say even though I do not know what it is I know that DNS has something to do with that system. Maybe it is a DNS problem, you know.

Seg: What about you Thud?

Thud: I think as far as what it takes to be a good admin, it does take a lot of imagination and also takes, I think the really, really good admins really love what they do. It just becomes part of their life. They run their own servers all the time. They are always trying to pick up new things. They are always right there trying to not only do the day to day stuff handle as an example handling tickets that they worked on. They have done the same thing 100 times. They also love working on the ones that they have no idea what's going on. They have no clues to find out what the real problem is. They just love working thrugh it. I think that is what makes a really good admin is somebody who loves what they do and that makes them good at it.

Seg: In my experience there is a bit of OCD, a bit of obsessive compulsion to some of the things that I do. You definitely have to be interested in being an admin in being interested in the systems. At least in my experience you have to be driven a little bit more than that to keep at it because parts of the job can be very meticulous and very mundane. Writing scripts, parsing files, writing code and troubleshooting things can be interesting, they could be challenging and they can also be mind numbing at times. Having a little bit of an OCD personality definitely us going to help. You also need imagination. You have to have imagination to touch on Gek's response. If you can't imagine what might be causing the problem, or if you can't think beyond what you already know about it, then you are never going to go anywhere. So you absolutely do need an imagination with that.

Thud: Yes, imagination comes in handy too when you have a horrible customer and you imagine them sitting in a pot of boiling something or other. That comes in really handy too.

Seg: Oh yes, definitely.

Gek: I usually start going over BOFH stories and thinking, "Wow! If they could just do that. It would be great!" [laughter]

Thud: Where do you think is a good place to learn how to be a good sysadmin?

Seg: You know, I don't know. Google? Google is definitely a good place to gain information.

Thud: Yes, Google is God. [laughter]

Seg: If I were to start from square one again, I am not sure where I would kick off. I guess where I started was I wanted to run a website. I wanted to have a web page, not a website. That actually started me on the process. How to work, where do I go to get hosting for it, how to I upload files, what do I name these files as they are used, and that's where it started out.


So, if I wanted to learn it over again, I didn't know anything, but I needed to imbue myself with some bit of information on how to get me started, it would definitely be start a web page. The designing of it or serving it, but I think that would be good way to kind of push you in the right direction. And then, it's just up to you to take you as far as you want to go.

Gek: That's a great example because you can really build off of that. If you want to become more of a scripting, coding type person, you can start writing PHP or Perl/CGI, and you can do stuff on that. If you want to do move more towards statistics gathering like I do, you get log files from the webserver when people are visiting your website. That's a really good suggestion, because you can really do an awful lot from just a basic website.

Thud: Yes, I have to say the same thing. That's how I got started and all of this. I wanted my own website so I setup a single web page. A few months later I discovered CGI's and Perl. I wanted to work with that. From there I started doing more complicated designs. One thing led to another, and I ended up wanting to... I got tired of the hosting provider where I was at, because there were all kinds of downtime and service kept getting hacked and everything. That's when I started even considering running my own server and everything that's involved with that.


OK, now for the next section. What makes a bad admin? Gek, what do you think makes a bad admin?

Gek: You can learn a lot looking over my shoulder. If you rush through things, if you are not somebody who pays attention to details, if you don't have proper respect for... in my situation, I am not admining boxes that are mine for the most part... if you do not have proper respect for your users, your end user, that's a really bad thing. As horrible as it is to admit, you do have to kind of bend to their whim on topics where you know better, and they are asking you to do something that you don't want to do.


For the most part, I think its just paying attention to detail. That's the big thing. You've got to watch what you are doing, and really know what you are about to touch and what systems it is going to affect.

Thud: Yes, that's actually one of the things that was brought up before, about having a little bit of OCD. I have always kind of felt that being a sysadmin, if you don't have OCD to start with, you end up developing it. Just because there are so many different things, especially if you do it on a large scale where you have multiple customer servers that you are working with. You kind of have to be able to bounce from one thing to another and be able to keep track of everything. It's this weird controlled OCD, where you concentrate on one thing just long enough to make sure you are not screwing something up and you get it to the next stage, and then you move on to the next project, or the next server, or the next ticket. It's kind of a controlled OCD.

Seg: The thing in my opinion that makes a bad admin is, assuming, when I am touching a server, whether mine or someone else's, when I am doing work on it, I try to always ask myself questions about what I am doing, double-check each command that I am typing. I have made some really dumb typo mistakes in my day, even recently. Thankfully, it was my server, where I did it last.


Being too fast in how you type and assuming that things are going to work out in a certain way, is probably the worst thing you could do. Also I have known of lot of admins in the past, take ten months or so, who happen to be customers. I have known all the bad admins, because they have expressed a sort of train of thought that would say, "Well I thought it should work this way. So that's the way I did it," not ever checking the way it does actually work. So they leave systems open, they let systems fall down, because they thought, well it would make sense if it does work this way, without actually checking.


So, without knowing more about the systems that you run is OK, but you need to investigate those things. If you don't investigate them, you don't look what you are doing, you don't question yourself, you don't assume anything, then you are a bad admin, at least in my opinion.

Thud: I have run into that quite a bit. I have had a lot of admins that I had to work with, they just assumed things. I think one of the biggest assumptions that people make is that they know what they are doing. I have had a lot of admins who swore to me up and down, that a patch would work a particular way with the configuration they were using, and if just read through the documentation, it is clear what those options do, and it doesn't create the log file in the format that they thought it was going to do.


These are just little things like that, that really show that they are not paying attention, they are making a lot of assumptions. It always ends up burning you in the end. Even if it's on your own server, because you could be trying to install a new script that requires a patch to be configured a certain way, and you are not paying enough attention to realize that the document that you are talking about was written for an earlier version of Apache, like in the 1-3 series and you are using 2-0. And the configuration files are close but they are not identical. And so then you end up having to spend, what you thought was going to be a 10 or 15 minute install, you now spend four hours trying to debug, just because you did not read the documents to start with.

Gek: When working with co-workers, I mean, this is something along the same lines. There is not a problem, I mean where I work, with admitting I don't know how to do that or I don't how what that's going to do. You are going to get more... I don't know. You are going to have people come at you with more attitude if you don't tell them ahead of time that you don't know what you are about to do or don't fully understand it, than if you just say, "Hey look, I don't know what I am doing." And most people would give you a hard time for that too, but at least then they understand.

Seg: In hindsight, it is always better to say "Hey I don't know how this is done", because if you say, "Hey I know how to do this", and then you break it, it is ten times as bad. I have never got any gruff from anybody, in the past year or so that I can remember, by saying "Hey I don't know how to do this." I usually get a nod and a bit of respect for having the guts to say that. I would definitely extend that to anyone who says it to me. I don't have any problem helping anybody out, and I am glad that most people I talk with and work with don't have a problem helping me out.

Thud: It also comes in handy when you are dealing with a customer, and you tell them up front that you don't exactly how to do what they want, and then they come back and say, do it anyway. When they have to pay to have their server rebuilt, you've got it in writing that they said it was OK. [laughter]

Seg: I had one time where it helped me out, it actually saved me a lot of work. I was talking with one customer who wanted to setup an automatic file transfer using authorized keys over SSH so an automatic SEP from one host to another without worrying about passwords. And I told him I said, "I've never done it before. I don't know how to do it" and he said "OK, well I'll just do it then." I said "Sweet!" So that actually got me out of work. I eventually ended up teaching myself how to do it but it was nice at the time. I was honest with him. I said "I don't know how to do it" and it actually got me out of work so it was pretty cool.

Thud: Yeah, any time you can get out of work is always a plus.

Seg: Definitely. That's what makes a good admin. A good admin can get himself out of work.

Thud: Yeah, and a good admin will spend four days writing a script that will save him five minutes.

Seg: Oh, yes.

Thud: If we've got time here, can you tell us a little bit about the movie you made about installing Fedora. Mr. Thud?

Seg: Well that was basically my response to yet another "how to install Fedora" fact that was out there. And I just suddenly realized that everybody keeps writing these facts and I guess people are having a difficult time doing the install and I just happened to notice that in the fact it says, "Run this, run this, wait for this output, run this other command." I figured with an actual video of an install you can actually see not only what you're supposed to be typing in and where you're supposed to be typing it in and what kind of out put you're going to get after it. That's one of the reasons I left the boot up process is it's loading all the servers and checking all the hardware. Because if it's somebody new to Lennox or even to a red hat like system, that may not be what they're expecting. So that's why I decided to do it in that level of detail and I don't know, it was my first attempt so it's not that great. Further attempts on other install videos will be much better.

Thud: I found it to be very interesting even though I'm not a Lennox red hat and I've used Fedora for years. I'm still not much of a fan of it. I did find the video to be very helpful and very informative. One thing that I am curious about being a geekwhat kind of technologies and programs and things did you use to actually record that in that kind of format?

Seg: I did it with DOS batch files.

Thud: Sweet!

Seg: [laughs] No I actually did that with VM Ware. VM ware later editions have the ability to actually record everything on the screen. I tripped on it in the past because I was trying to debug a problem and because I was going from customer to customer and problem to problem I was having problems getting the entire thing in one long process so that I could do it all the way through, have it fail and then see exactly what I did. So I started poking around in VM Ware, found the ability to record so that I could record and when I problem came up I could pause it, and not only pause the recording but pause the entire system, work on the problem that just came up, finish it, come back and continue the recording. Then when I was done I had a recording of everything that I had done and the exact errors that were coming up because the errors I was getting out of the compile were many, many screens long and it was, really, really difficult for me to log it in a way that made sense. So that was what I used to actually record the installation part of it. The rest of it to get it into QuickTime format and also to add all of the text at the bottom was with a product that I've used for years called Camtasia which basically allows you to record anything on a Windows box. There are a lot of companies out there who use Camtasia to record demos of their software packages. There used to be an ISP I had that used Camtasia to create a demo on how to set up to dial into them. So it's been around for a long time and it comes in really handy especially if you have a customer who is having a problem and says "Do these steps this way and you'll get this output and if you don't get the output you can use Camtasia to record the entire process" and you can say "see look, I typed in exactly what you said, clicked exactly what you said and this is where your program crashed."

Seg: [laughs] I just want to mention real quick that VM Ware is a virtual machine program. It will actually emulate an entire machine on your box. You can run a different OS while you're already running something. One of the most common examples is on a Windows machine or a Windows workstation you can load VM Ware and run Lennox at the same time in a window. It's actually pretty cool. You can do the reverse. You can run Windows on Lennox and stuff like that. There are ports for VM Ware in the free BSD ports. There are probably some in the open BSD ports. I don't know if they actually work or not but they are in there.

Thud: There's definitely none in the openBSD ports.

Seg: Oh, really? OK.

Thud: Open BSD is different enough that they hate VM Ware. [laughs] In the show notes I'll include links not only to VM Ware but also to Camtasia. Those are just software packages. I registered them a long time age and I found them to be so handy I just always use them.

Seg: Awesome.

Gek: I wanted to mention an Open Source free program. It's not as nice as VM Ware. It doesn't have the GUI to work with but QEMU actually does do the same thing with a little more playing and it does run with Lennox, Windows, Mac,. I think just about every OS.

Seg: Can either of you tell me the difference between WINE and VM Ware?

Thud: Yeah, VM Ware actually emulates an entire machine. So whatever OS you load on ityou can even load OS's that are not supported directly by VM Ware. Open BSD happens to be one of them. Because as far as the OS that you're loading on the virtual machine, it just sees an Intel machine with a certain processor, a certain amount of RAM, certain amount of disk space which you can all configure and say I'm going to give this amount of disk space to the machine, this much RAM. When programs run, when OSs run, it sees itself as a stand alone machine. WINE is actually a Windows emulator.

Gek: Ah, you just broke the cardinal rule.

Thud: What?

Gek: WINE is not an emulator.

Thud: Well, it is an emulator and the reason why I call it an emulator is because what it's doingit runs on UNIX systemsis when a Windows program makes a call to a DLL to tell a Windows box to generate sound out of the sound card, WINE will intercept that request and turn it into a Lennox request or a free BSD request so that the underlying operating system knows what that request is for and starts generating sound out of it's speaker.

Gek: I'm just giving you a hard time. I know the WINE guys if they ever hear this are going to write us and be nasty about the "e" word.

Thud: Well does WINE boot up and can you reboot it? If you can't then it's an emulator. [laughs]

Gek: I'm on your side on this one.

Seg: In my limited experience of use with WINE I found it to be you would run WINE and put the command that you wanted to run with it into the command line for the actual execution of WINE. WINE space [inaudible] DXE would run [inaudible] DXE in WINE. Now I've never actually got WINE to work with anything and I've also never tried VM Ware which is why I asked the questions that I did. For reference though, did I have that right? Is it WINE space [inaudible] DXE or is it something else?

Thud: That's a good question. I haven't used WINE in a long time.

Gek: I have.

Seg: That's fair. See?

Gek: I actually use WINE for the fractals I do. The program I used to generate the fractal frame file is written in Delphi so I have to use WINE to run it. That is right and most of the time that will be sufficient. That's all you have to do. There are some applications that require a little bit of tweaking like Office is a lot more complicated. So, if you try to Wind Notepad it will work but if you try to Wind Word it won't.

Seg: You have to do, there is a wind configuration utility that sometimes you have to go and play with like font mappings and things like that.

Thud: OK Cool. I also wanted to point out even though it should be obvious at this point. Here in our first episode ever it is obvious that we don't know everything and that we still ask questions and we are still learning.

Seg: I don't know if I would go that far I mean just because I am not telling everything that I know doesn't mean that I don't know everything.

OK, all right, well, I don't know everything.

Seg: I don't know everything but my wiki does.

Thud: Actually I used to say I know everything but not all at the same time.

Gek: There you go.

Thud: This episode is a moment, in a sec, which is a moment to reflect on some part of security for running your own server. I picked out passwords. Everybody says you should use a really complicated really long password that is alpha numeric preferably randomly generated. I am actually going to say that's a bad idea because if you have a 32 character randomly generated password you are going to write it down somewhere.

Seg: Not me.

Thud: Not you?


[laughter]

Gek: As long as it is randomly generated it's all zeros, right?


[laughter]
Thud: Right exactly.


[laughter]
Seg: My personal preference is what I have for my password is a stash of passwords that I have tripped across. One of them is actually the first password I have ever had to access my email account and it is usually a five or six character password that is randomly generated but because I have typed it a million times I have it memorized and if I need something that is a really strong password, I will take two or three of these five or six character passwords and just glue them together and I will change them like occasionally I will capitalize the first letter. Whatever I do, I do it in a kind of a systematic way so that I can easily keep track of it but I also don't have to worry about trying to remember a 32-character password. I have got six or seven passwords that are five characters each that I have used throughout my entire tech life so they are kind of ingrained. I don't even have to think about them. Then I just remember that it is my third or fifth one is the combined password for this particular box. So what are you guys' thoughts?

Seg: I don't think we should use passwords at all.

Thud: Well then what should we use?

Seg: Actually my own password policy is periodically, I will flush all of my passwords and start new. I will pick usually three to five passwords. I will generate them and then I will spend time memorizing them. I will go through all the systems that I have to write passwords on and I will switch them out. I have different levels of passwords. I have like one or two passwords that I use for really, really light stuff and then I have more passwords for more complex systems. The systems that I actually use that are really important to me they have their own passwords or they will have variants on existing passwords that I have. If I have five good passwords and I have seven systems that are very critical, can't let anyone get into, I will take one of those stronger ones and vary it and create a unique one for one of those systems which really is important. I am kind of following some of the things that you do. Again I think it is a nice safe way to be and it is also very manageable. What about you Gek?

Gek: I am pretty paranoid. After I got my Mac, my first iBook I started using key chain to store my passwords. So I am up to the point now where any password that is not work related, my work related stuff I don't count. Anything that is not work-related like a website like my Digg account or my de.lic.ious account I will go ahead and use the random password generator and generate something that I will never be able to remember and keep them in the key chain which has a pretty complex password that I do remember and I will change that password at least once a month.

Thud: I think that's a good system. What do you think Seg?

Seg: That's a good system. I actually I have a similar system that I used in the past. For very few systems I use my pocket pc I have got a program that encrypts all the databases where you can store not only passwords but account numbers, things like that. That comes in pretty handy. The only problem that I ever ran into with a system where you have one password to get access to all the other passwords is what happens if you forget that one password or the data file gets corrupt.

Thud: I would definitely have to agree with you. I actually helped a customer with our company a few months ago. This person's laptop had exploded and of course their password vault was on there and they had entrusted that password vault so much and they never backed it up. All of a sudden our company had no idea who they were because they couldn't remember for anything. They didn't even know their RSA pin. It was quite funny. Yes had to back it up.

Thud: What makes a good sys admin, somebody who insists on backups.

Seg: What's a backup?

Thud: It's the point where you sit there and say if I lose this data I have to commit septicu so I am going to copy this to a different disk that is somewhere safe like an in a safe or way up on the top shelf where it is really wet. You got to do backups. For stuff that. that's sensitive information, like my financial files I actually use a PGP password. I don't use because my key because my wife doesn't understand that but I use a password that she is aware of so if she ever needs to she can get into the data.

Seg: For personal backups, one of the things I used to do I haven't done it recently, I have a friend you lives on the other side of the country occasionally I would burn a DVD with the images from scans of my bills with my account numbers and stuff on there passwords sheets, stuff like that. I would encrypt it on the DVD and send it to them just in case a small planet would land on my house. I had some place else I could go and recover the data. It worked well until my friend moved and didn't tell me and I started getting these DVDs back from these people that I didn't know.


[laughter]
Seg: Whoops.


Thud: Well that about does it for our first episode. We hope you enjoyed it. We had a good time making it. We will try to make them better in the future. For show notes or any other details we talked about please remember to visit our website at runyourownserver.org. Or if you would like to send us feedback or would like questions to be answered in future episodes send us an email at podcast att runyourownserver.org.


Thud: The intro music, "I Like Caffeine" is by Tom Cody. This song "Down the Road" is by Rob Costlow.


Please visit our website for links to their websites. This episode's first sponsor is the Open BSD project, a free open and secure Linux like operating system. Be sure to check out the project at openbsd.org.




This work is covered by a Creative Commons license. Please visit our website at runyourownserver.org for details.


Transcription by CastingWords