Going from professional dishwasher to professional Linux SysAdmin

Introduction

I see it all the time – “I wish I could get a job doing Linux professionally.” This is my advice, the same advice that has worked for some of my closest friends who went from burger king to sysadmin in a year. I followed part of this path myself to get where I am now, and to some extent, so did some of my peers. If you follow my advice here, you should be able to get your foot the door doing linux work somewhere.

You can’t really go to school for this. I’ve talked to people who have taken “Linux” classes in college / university, and it’s not geared towards production servers. It’s geared towards some antiquated world where you might edit a spreadsheet in AWK or use home directories as a department document management system. This doesn’t compare to how Linux is used on The Internet.

So what does an entry level linux pro do?

  • First line of troubleshooting:
    • I can’t log in to this server
    • Disk space is getting full
    • Loss of network connectivity
    • Add / remove users
    • Reset passwords
    • Grant SUDO privileges

So, to get started in this business, I recommend undertaking a series of projects, using Google as your tutor. The project I always have my apprentices start with is this:

When I pull up your-name-lab.com, I end up at a webpage with a picture of you. This webpage is running on Apache, running on a gui-less Linux server in your house.

If you can get through all of the stuff involved in making this happen, you will have essentially bumped into every part of the usual Production Linux workflow. You won’t necessarily do it the right or best way, but you’ll have a really nice grip on the concepts involved.

The Project

Here’s what you’ll need to do to make this happen:

  • Acquire spare PC or use VirtualBox on your main workstation to install CentOS or Ubuntu Server
    • I can only recommend CentOS since that (or RHEL) is what’s used by most production environments.
    • Debian and Suse are out there too, but I think CentOS is best to start with.
  • Get through the GUI-less installation. Do a minimal install, don’t pick the “Web Server” package group during installation.
    • CentOS by default will use a windowed install mode, and that’s ok. At the end, don’t rely on the GNOME/KDE tools to make changes though, because those probably won’t be there on a production server.
  • Install Apache web server.
    • Easy mode: Use the package manager to install it.
    • Bonus points: Install from source.
  • Configure apache to serve content from a directory, and put some content in that directory.
    • Easy Mode: Disable SELinux.
    • Bonus points: Configure the content directory as part of the correct SELinux context.
  • Register a domain name, and point it to your house.
    • Hint: dyndns.com. You can either configure it on your server or on your home router – the choice is yours.
  • Configure the firewall(s) to allow web traffic to your server.
    • Easy mode: Disable iptables on the linux server
    • Bonus points (and part of your RHCSA/RHCE cert): Add iptables rules that allow:
      • Access from your LAN, then when you’re ready
      • Access from your LAN or the internet

I’m not going to walk you through any of that. That’s your project, and that’s where google is going to help you out. Also, that’s only the beginning. A major part of my job (and the reason they pay me) is learning new skills quickly. Sometimes, I have to dig into config files and commands I’m not familiar with. It pays (literally) to know how to get information and apply it quickly.

Remember: The journey is the destination.

Beginner checklist

These are the things I would expect you to be familiar with if I were to hire you as a Jr. SysAdmin:

  • Where the system logs are stored
    • Where would I find a record of who has logged in and when?
    • How can I see when the last server reboot was?
    • Were can I see if there were any errors on boot?
  • Basic iptables concepts (opening a port, only allowing access to a port from specific networks)
  • Adding sofware with the package manager (yum, RPM)
  • Configuring network interfaces
  • Adding static routes
  • Creating / deleting directories
  • Changing permissions on files, familiarity with how permissions are applied and inherited
  • Creating / modifying text files with VI
    • Emacs is great, but you probably won’t see it in production due to it’s large size. Vi is light weight and common.
  • Disk Space stuff
    • How can I see how much space is used up on the root partition?
    • How can I see how much space /var/log/ is taking up?
  • Using / Configuring SSH
  • Moving files around with SCP
  • Run levels
    • How can I show the current run level?
    • How can I show the default run level?
    • How can I switch to another run level?
    • Where are the run levels defined?
    • How do I get into single-user mode?
      • The more ways you know, the better. They’re not all equal…
      • What if single-user mode still barfs?
  • More networking – it’s more relevant than you might expect:
    • What is ARP?
    • What is NAT?
    • What’s the difference between TCP and UDP?
    • What is the OSI model?

If you’ve got your linux server, just start googling those concepts and looking for ways to apply them.

Certifications

Not all certifications are created equally. I’m a big fan of the RHCSA and RHCE certs, even though I myself have never made it through the RHCE process (tried for RHCE5 cert, missed it by .3 points). I’ll get back to it at some point. The wonderful thing is, RedHat makes the certification objectives public.

Logically speaking, if we know that a) the RHCSA certification is respected in the industry and b) the certification objectives are published, then we can get an “equivalent training” by simply working through the certification objectives. If you’re a book learnin’ type, there’s one book that will absolutely MAKE you as a sysadmin: Michael Jang’s RHCSA/RHCE Certification Study Guide 6th Edition. You could, of course, just work your way through the certification objectives on Red Hat’s site.

A note about VMWare

Environments are moving away from the 1 OS, 1 Chassis model and towards virtualization. The VMWare VCP is a valuable cert, but difficult to get on your own since you can’t just pay a couple hundred bucks for the test; you MUST take the training to be able to take the test.

And a note about CCNA

This one is great. I highly recommend getting your CCNA, or at least taking / auditing a class. The routing protocols… they’re ok, but the parts about Ethernet, ARP, the OSI model, TCP/IP addresses and maskes, and gateways? That’s the moneymaker. If you get through this, and you understand the current DNS system, you will understand the internet, as a whole. It won’t just be the magic box that delivers kittens anymore, you’ll see how it all comes together.

More fun projects to build you up

There’s always more to conquer. Try getting these things up and running on your lab:

  • NFS fileserver
  • SAMBA file server
  • Remote Syslog
  • Bind (DNS)
  • LDAP (Plug: My Centos6 LDAP recipe is pretty boss.)
  • Postfix
    • Once you get more than a handful of servers in your lab, it’s handy to get the mail consolidated.
  • IRC
  • Asterisk
    • I have a lot of fun with this. If you think you might like getting into VOIP, asterisk is a great way to build up the resume. It was a lot more fun when it integrated with Google Voice, but that’s another rant.
  • MediaWiki / Confluence
    • You should be keeping notes about what you learn when you get technical. A notebook is nice, a wiki is better. Your future employer will probably have a wiki, and it’s always nice to say you have documentation skills.
  • MySQL
    • Database administration is its own ballgame, but you’ll need a MySQL database for some “out of the box” deployments of other Open Source software. It doesn’t hurt to know a few things about it, and if you like it, you can make even MORE money as a DBA.
  • Ampache
    • Speaking of software that uses MySQL out of the box, Ampache falls under that umbrella. It’s a streaming media server – pretty cool for listening to your home MP3 collection at work. Granted, it’s easier to use Amazon Cloudplayer or Google Play, but Ampache is YOUR server run YOUR way. You can also share your collection with other people, which the mainstram Cloud providers won’t let you do.
  • Nagios
    • If you’ve built a big lab, you better keep an eye on it with some monitoring software. Nagios is a good one because it gets used at the enterprise level. Again, if you like it, specialize in it and monetize!
  • Cacti
    • Curious how much bandwidth you’ve been using, and who your “top talkers” are? Use cacti to graph it all. It’s a bear to configure and use, but when it works right, it’s damn pretty.
  • Linux from Scratch
    • Now, if you really want to learn how linux works, try this project. Not for the faint of heart.

Looking for that first Linux gig

I hope you’ve got some tech companies in your area, or at the very least, some decently sized businesses. See if they have any openings for a linux guy, and apply for the job. Hopefully, you’ve been on a job interview before and know the do’s and don’ts. If you don’t know how to nail an interview, that’s just one more thing you need to google.

My only real recommendation here is that you put your key linux abilities up at the top. This makes you searchable and is friendly to HR-types who won’t read the whole thing.

The type of gigs you should consider:

  • Jr. SysAdmin
    • This is perfect, although you may need Windows experience as well.
  • Incident Response Specialist / Analyst
    • Less perfect, but also sometimes equivalent to Jr. SysAdmin. This is an ITIL term and is subjective in its implementation. It may also be Phone Answering Guy, which is “create ticket and give it to someone else to fix” guy. At least you got your foot in the door.
  • Phone Answering Guy
    • Sometimes you just create tickets and send them to other people to resolve, sometimes you can help with scripted troubleshooting. Again, at least you’re not selling burgers anymore.
  • Datacenter Technician
    • This one is varied. You could be simply installing equipment that someone else will configure, you could just be running cable, or you could be doing it all.

Final thoughts

The best thing you can do is just try, and keep trying until you achieve your goal. Between the man pages, forums, and other how-to articles out there, you can do it. And each time you do, you’ll have experience to carry forward to your next goal.

I’m going to give a shout-out to Server-World.info here because there’s a lot of “recipes” that you can put to immediate use. However, none of it is annotated or really explained. Best thing to do: Use it when you get stuck, and figure out why HIS implementation works.

2 comments

  1. simon says:

    Thanks for this truly inspiring post

  2. UncleTom says:

    Excellent article Brian … takes me back to the mid-90′s when the available information to do Unix or Linux systems administration was much more difficult to find. A lot of it was still mainframe oriented, and companies like Sun dominated the landscape.

    I remember many times looking at certification programs – and in those days all you had was Novell and Microsoft … Linux certs were still a few years away.

    Back then, with the onslaught of the internet, the only way you could be technically current, and stay current was to build some kind of frankenstein machines at home, and go through painful source installs of DNS, BIND, SSH and every service and config we all now take for granted.

    I can remember starting CD downloads of Linux distros at night so the downloads wouldn’t choke a 56k modem during the day.

    Now it literally takes me less than 5 minutes to create an instance of an apache webserver and all the necessary features under Centos/WHM/Cpanel – but back in the 90′s it probably took me a week or more fiddling and tweaking a production machine into service. Even back then, Linux distros were easier than installing IBM AIX, SunOS and even SCO – which meant way more configuration and tuning than you will ever encounter on Linux.

    So your advice is good – build stuff at home make it work, break it, re-install it, learn how to compile from source, watch those mysterious ./config programs do their thing.

    I learned a huge amount about Microsoft’s SMB protocol from SAMBA, – which earned me an extra gold star at a major client who had 3000 workstations running windows and they couldn’t figure out why one of their network segments was choked. Linux and SAMBA self training made me a good cross platform geek as well. I hooked up an old crappy laptop running Linux and ran some of the tools and watched packets … they had a guy with a professional expensive sniffer who missed the problem.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax