Category: Career

Steve Jones has asked for some folks in the community to talk about what it’s like to work in our field across four posts. So how do I talk about my job across four job posts? If you didn’t read part 1, you can click here to read about what the day to day is like.

My four series post is structured as:

1. Day to day, what is normal? What are 95% of my experiences?
2. Best days ever, these are the highlights.
3. Worst days ever, these don’t happen all of the time and this is what nightmares are made out of.
4. Weird days, Brent Ozar said it best when he said Database Administration is weird.

The weird days of being a DBA.

You can run into a lot of weird things being a DBA, but I don’t think the topics I’m going to discuss are what you are expecting. Most of the weird things I run into are design structures or processes in place that go against best practice or are not typically practical. Kind of like if the past admin gave out Neapolitan ice cream instead of vanilla. Everything is custom, unique, and different, whereas I try to instantiate things as “vanilla” as possible. If things are predictable and implemented consistently, it’s easier for anyone to come in and troubleshoot or support.

Well, what kind of weird things do I see then? Well, you have self-made issues, you have known and unknown bugs, and sometimes you just have weird behavior due to … computer gremlins.

Here are some examples of impractical implementations I’ve worked with: Primary keys that are clustered indexes based on 32 bytes (guid + datetime2 + datetime2), procedures meant to implement high concurrency DML when interacting with a table without handling isolation levels, and more than five duplicate indexes on the same table (minus the name of the index of course…).

Those scenarios are not best practice and it really goes against practical design. So you need to look at the overall picture and decide what the end goal and intentions are. Think about why and why not some of the features were implemented or missed. Whenever I go against best practice design, explanations go immediately into the documentation and comments go into the code. Sometimes it’s important or necessary to stray from vanilla to get some strawberry ice cream, but make sure your successors and current employers know/understand why you deviated.

Sometimes, things just DON’T make sense. I saw a weird issue where you cannot expand a drop-down box in SSRS when you open it. This took me a few minutes to figure it out. But what was the issue? It was a service pack bug. A quick cumulative update fixed that right up. This goes into what I talked about in my last session, the worst days ever. If you are propagating your changes up through multiple environments, you should catch issues early on and you don’t have to scratch your head wondering why you suddenly had new changes.

Other times it could be a bug that has no fix or report yet, but those are even rarer. And since they are rare, you can imagine how frustrating it can be trying to fix things that aren’t documented or experienced by many people. Microsoft has employees who help on StackOverflow who can help you identify bugs, so that may be a good resource if you get stuck banging your head against the wall.

Most issues you run into are probably user error or bugs. Sometimes though, things just don’t work and there may not be an explanation you can find for it. Not that there isn’t an explanation, just that you need either the proper tools, access, or knowledge prior to the problem occurring. If it’s a one-time event and it doesn’t happen again, good luck figuring that out without having the problem monitoring in place. Likewise, if something is going on outside of your control (network, sysadmin, application, etc), you’re going to have a hard time figuring out why things are behaving why they are. (Noisy neighbor syndrome in a VM is a common example. Where one box on the host eats the resources and the other VM has to shift resources back to the ESX Host to help that neighbor.) I like to call those problems, computer gremlins.

The best way to combat computer gremlins is with mentors and knowledgeable co-workers. It’s hard to know what you don’t know. When I started learning SQL, it was difficult to learn because I didn’t know what kind of questions to ask. Mentors can help guide that path and get you where you need to be so when a gremlin inevitably pops up, you’ve thought about and then implemented monitoring that can assist in that scenario or you know someone who can help.

The three weirdest IT experiences I’ve had:

1. A ticket that literally stated: “Desktop background is showing a clown spitting plates and there is a typo in the background. This is affecting all agents and they would like the “scary” picture changed or removed.”
2.  That time I had to drive to another site to help teach users how to use the CTRL+C and CTRL+V hotkey functions. 
3. Troubleshooting split brain syndrome in an AG setup.

Steve Jones has asked for some folks in the community to talk about what it’s like to work in our field across four posts. So how do I talk about my job across four job posts? If you didn’t read part 1, you can click here to read about what the day to day is like.

My four series post is structured as:

1. Day to day, what is normal? What are 95% of my experiences?
2. Best days ever, these are the highlights.
3. Worst days ever, these don’t happen all of the time and this is what nightmares are made out of.
4. Weird days, Brent Ozar said it best when he said Database Administration is weird.

The worst days ever for a DBA.

Some of the worst days you can have are when they are spent on firefighting. Your projects don’t get worked on so deadlines approach faster and you aren’t having the chance to work on being proactive because you are actively fixing the problem. What really makes those days bad though, is if policy and procedure are what’s allowing this to happen.

Disaster recovery practice, proper high availability set up, and properly spent resources are what keeps you afloat when you spring a leak. I often see IT in companies where they are more of an afterthought and a sunk cost instead of a valuable resource for the company. There are varying degrees of this that you will encounter, and those are areas that you’ll likely need to change their minds.

DBA’s should have a solid practice environment for disaster recovery. If you aren’t prepared and something big hits, nothing is worse than trying to google a solution to a problem that is bleeding the company actively. Being prepared means you know your backups have been tested, you know how to quickly recover to a point in time within minutes when necessary. The drives are already mapped and anticipated, you know where the files need to be restored to. (DBATools is a fantastic helper with disaster recovery and practicing by the way!)

Likewise, your availability should be thoroughly vetted and tested before deploying to production. Ideally, you’ll have several lower environments configured identically. When you patch or push new changes, you’ll do so through the lower environments first and let them “bake in”. After a testing period and proper QA, you can push the changes up the chain until they eventually land in production. Proper monitoring tools to check the health of the equipment and the software should be instantiated to alert you before things begin to go wrong.

Along with the hardware, you need to make sure you’re adding resources when necessary. The most common thing here is disk space, just because our data will grow and grow. This is where a lot of employers have trouble allocating more money, they want to have those historical highlights but they don’t want to spend any more on holding data because your MDF keeps growing. I find this is where you’re both going to have to compromise. I recommend looking into some new technology or practices that would allow you to keep as much historical data on slower and cheaper disk or possibly archive it off with the ability to restore it later if necessary. You’ll need to help them weigh the pros/cons and costs involved with the time and resources spent with the options. (Of course, other tools that help with compression and backups may be another idea!)

Assuming you are taking care in those three areas, you won’t have many bad days. I try to be the quiet DBA, not the hero DBA. The hero DBA is the one who is frequently running in to save the ship that’s on fire. This is great, but if your DBA isn’t doing this frequently, this means they are working proactively on the situation to keep things stable.

Not all disaster comes in the form of a crazy outage. Sometimes someone just runs something without a WHERE clause predicate. (DELETE FROM TABLE… whoops.)

I’m proud to say that I’ve never caused a show-stopping outage or problem that we couldn’t recover or fix quickly. Everyone’s going to have a moment or two like that though, in IT, that’s just the way it goes. Even if you are perfect, there’s someone out there who isn’t going to be. So if you are properly prepared, this isn’t too bad. If you aren’t properly prepared, you’ll be sweating bullets and get that feeling deep down in the pit of your stomach where it feels like you got kicked real hard. As you’re working the issue through, you’ll feel like your hair is greying in front of your eyes from the stress. So when you can resolve those problems in minutes confidently, that’s going to save you a ton of pain and hair loss. (Your employer will appreciate how quickly you resolved the issue too!)

These kinds of issues or days are pretty rare. I’ve seen code and hardware failures, worked weekends/nights, missed events and outings with families and friends. That’s just part of the job sometimes, but this is definitely not commonplace. (If it is, you may want to do everything you can to either improve the policy and turn that ship around or find a new ship.) It is something you can and have to expect though, which is why the policy should drive best practices and proper procedure so this can be dealt with quickly and with minimal to no impact to the company.

And just for fun, the three worst times of my IT career:

1. 36-hour shift rebuilding Microsoft Exchange servers with Microsoft on the phone.
2. Two 100 hour work weeks to fix every computer on the floor. (It was nice that we had showers at work and blankets/pillows.)
3. That time a DELETE statement was run but I was able to fix the fallout after a stressful week of identifying corrupted backup processes and piecing it together from historical tables.