The Long Back-Story
(queue the campfire scene, under the stars, with distant harmonica and bearded old man, smoking a pipe of something, and all the little systems engineers, all gathered around to listen in their fuzzy pajamas)
For the last three decades, the roaming bean-counters of the world have quietly been building up a pressure-cooker of angst from all the walk-up status inquiries in the IT cube farms of the world. Each time they’d ask for a status update, they’d get a magical (mythical) answer. Specificity was lacking. Upper management was not happy. Vendors kept nodding in agreement, but were still focused on the product users, not the check-writers. That changed soon after the Cloud popped up.
I may blog about my thoughts on “The Future of the IT Worker”, if I have enough wine or beer to motivate me.
Short version: Shareholders buy stock in a company to make a profit on rising value (stock prices). Stock prices rise when the company increases profits. To increase profits, the company can only increase the gap between revenue and expenses. For 99.9% of businesses, IT is a “cost center”, or an expense. Shareholders DGAF* about imaging computers, change management reviews, or what your name is. They care about 2 things:
- Increased profit margins
- No bad press
Both of those points are impacted by expenses. Shareholders don’t like expenses. They bitch about expenses, a lot. They hire consultants to analyze expenses, and these days, one of the first areas they look is IT. Asking question like:
- Why so many IT staff?
- Why are you re-imaging every computer you buy, when they already work?
- Why do you still have datacenters?
- Can we move to a cheaper lease?
- Training?! You don’t know this stuff already?
Seriously, the emphasis on “what value do you bring to the company?” is only going to get heavier and heavier.
So, in the interests of making yourself more valuable, I suggest bringing a little automation to your job. And, based on what most customers I know have already implemented, this is my 5-point list of gotta-have things:
[1] Active Directory User Account Processing
New hires. Temp staffing. Terminations. Name changes. Promotions and transfers. All of these tend to chip away at your precious time. Relying on a bundle of task-specific scripts is a good start: creating accounts, resetting passwords, adding/removing group members, and so on. But anything you have to stop and tend to with your own hands needs to be considered for automation.
Like all automation processes, it starts with the “authoritative source” of information. Usually HR. Whatever data they’re entering for a new hire, use that to drive everything else. Do not duplicate efforts by entering that information again somewhere else, as this not only wastes time, but adds risk of inconsistency.
If you don’t already have it, request access to whatever information you need to drive the entire process along. Make a list of all the user-related processes you deal with. Divide each process into distinct phases or tasks and work on them one at a time until you have the whole conveyor belt running.
Ideally, when HR says someone has been hired, your IT systems should immediately handle it. Changed departments? New surname? New job title? Done. Got fired for having sex on a forklift during work hours? Done.
Gaining experience with the HR systems and processes not only makes your job easier, it makes your role more valuable in the organization. Once the processes are automated, they will run more consistently and predictably, even if you go on vacation, and the organization will likely ask you for help automating other processes.
[2] Active Directory Computer Accounts Clean-Up
If you only have a dozen or so computers in your AD domain, you might get a pass here. But if you’re managing dozens, hundreds or thousands of computers, and you’re not running some sort of automated process to clean-out stale/unused accounts, you should be tasered in the crotch until the battery goes dead.
If you don’t already have something in place to automate this boring-ass chore, get moving. It’s really easy to implement a 3-step clean-up process:
- Determine what criteria will be used to say a device account is stale
- Identify and move stale accounts to an OU, and disable them
- After X days, delete them
Once that process is tested, schedule it to run on its own.
There are hundreds of utilities and scripts available today to help automate this process, or you can build your own. Having a process in place means you can answer questions about asset inventory with a straight face, and calm down those bean-counters who freak out over the thought that things are out of control. “Relax, bean-counter person. I have it under control.“
Icing on the cake: “I know we requested 1500 licenses of that software, but I confirmed we only need 1250. And with that $3000 I saved us, I’d like to attend MMS MOA this year, and buy a Hello Kitty flamethrower.“
[3] Patch Management
The biggest problem I see today isn’t the patching itself, or the tools available to manage the patching. The biggest problem I still see is a lack of a process or procedure. If you’re still manually updating computers, especially endpoint devices (desktops, laptops, tablets, etc.), but even servers, pause here and do the following first:
- Design a patching process: What, When, Where, and Who (owns each machine or system)
- Give each group of machines or systems a name
- Identify test machines within each group to validate monthly patches
- Identify machines that can be patched at the same time, and which ones cannot.
- Identify when machines can be rebooted
Having that mapped out will make it so much easier to pick and test the right solution (product or script).
After that, use your selected “test” machines for the initial pilot, and scale out from there. Start with the less critical machines and add the more critical machines later. That way you cover more machines early on, and work out the kinks before touching the high risk environments.
In the VAST majority of environments I’ve seen, the exception cases are the minority. So knocking out the machines with a consistent schedule also knocks out the biggest portion overall.
[4] Inventory Reporting
Fancy or basic, it doesn’t matter. The only thing that matters when the executives ask “how many ___ do we have?” is can you answer the question without lying your ass off. The other thing that matters, is when the BSA* comes to your door with a warrant, but that’s another story altogether.
How anyone can manage a computing environment without some sort of inventory reporting is beyond reason. That’s like expecting airlines to operate without flight plans.
Of all the examples listed on this post, this one is the oldest of them all. And since it’s been around the longest, there’s really no acceptable excuse to not have it automated by now.
If you don’t have a software product, or service, in use, get one. Many are free. If they don’t cut it, you can easily build your own with scripting and duct tape. Even if your devices are scattered across the globe, as long as they can touch the Internet, you can build something to make them squeal and give up their inventory data.
[5] Event Monitoring
Imagine if your car didn’t have a dashboard. Or your smartphone didn’t have a battery indicator. That’s pretty much the same thing when you manage computers without some sort of event and/or log monitoring. The data is being tracked somewhere, but unless you have a clear view of it somewhere, you’ll never know. Until it all goes sideways, and then you’re scrambling to find out where to look “under the hood” as the house is burning down.
Of all the support cases I ran into between 2015-2019, which related to some sort of “oh shit, our shit is broke! please help fix!“, most of the root causes fell into one of the following buckets:
- Ran out of disk space
- Service account was locked
- Service failed to start
- Configuration change impacted other processes
- Network connectivity failure
- Anti-virus was blocking a critical process
Every single one of these could have been avoided with the following simple tools:
- A monitor to report potential problems
- An automated process to remediate each of the potential problems before they get worse
Flying blind is no way to run a datacenter, let alone a bunch of computers. Whether you prefer to buy a solution, or build it yourself, just get something in place. In every instance where this was done, the number of “oh shit!” events dropped significantly.
Maybe you like getting a panicked call from a manager on the weekends, at 3am on a weekday, or while you’re on vacation. That’s not my idea of a happy life. And applying some basic automation to monitoring is not only one of the easiest types of automation, it’s often a good on-ramp to scaling your efforts into other areas that drain your time every day.
Cheers!