Today at work, we experienced one of those puzzles you read but rarely experience in real life.

We have a relatively new row in our server room at work, where it is surrounded closely on three sides by walls. on one of the short ends, there is a plastic barrier you can walk through (think the thick plastic strips that cover the opening of a walk-in freezer).

Over time we have slowly filled the cabinets in the row with servers, generally working from the far end towards the center, then filling the other end. We ended up with a mostly full row.

Now in this row there are APC In-Row chillers, inserted between the cabinets, which pull air in front the back (the ‘hot isle’) and push cold air out the front (the ‘cold isle’). They measure the temperature coming out of the back of the cabinets and into the front of the cabinets to ensure they are cooling enough for the demand.

Something changed over the break, where the temperature in the server room actually jumped several degrees. And last week while installing some new servers in the remaining cabinet, we noticed the chillers suddenly had a hard time keeping the row cool. I’d enter either side, and the chiller fans would immediately speed up.

We couldn’t explain the increased temperature elsewhere in the server room, and the strange behavior of the fans. Furthermore, today we noticed the fans were going full-speed all day.

My coworker Chris finally figured it out this afternoon.

He opened the the door of the cabinet with the remaining cabinet, the one with the most open space. The door happens to have a temperature sensor attached to it, so when the door was open the sensor was in the surrounding cool air and the chillers slowed down. Close the door, the chillers speed up. He figured out that this cabinet was the only opening left in the row, so the hot air behind the row was coming forward through the cabinet – and directly into the temperature sensor. Of course, the chillers thought the row was super hot, they were measuring the cabinet exhaust temperature, not the intake temperature.

Chris shoved an unused ceiling tile in the cabinet, covering the large opening behind the sensor. The chillers immediately settled down.

It turned out to be a simple problem with a simple solution.

last week i attended vmware’s annual conference, vmworld 2009. since i am doing so much shooting this year, i decided to bring my widelux f7 camera with me to the conference. by carrying it with me, i managed to meet a lot of photographers there, including many who shoot (including one 4×5 guy!). the conference ended up being a good venue for shooting, despite the low light levels.

after registration, you walk down moscone center’s wonderful staircase down to a long line of shiny new infrastructure that is running the show.

the keynotes were large productions. giant projected video screens. the large video cameras looked quite sophisticated. i took a peak in one that was near the stage. on the screen it artificially marked all hard edges in the scene.

the camera operator and the attendee behind him both noticed my camera, and i talked to them both about my widelux. the attendee had a horizon camera (also a swing-lens panoramic) but didn’t think of bringing any film cameras to the conference. i don’t blame him. it was at this point that i realized i must be the only one of 12,800 conference attendees carrying a light meter.

the first lab session i attended was vSphere 4 – New Features, Best of, Advanced Features. there was quite a hiccup in the lab infrastructure and a lot of time was wasted. this of course started a flurry of bad things being said over twitter and blogs. vmware managed to solve most of the problems and had late-night repeat sessions for those who wanted to repeat labs that had broken. the gentleman i paired up with for the lab was a manager and didn’t seem to be interested in much hands-on experience, so i was able to drive and go pretty fast through the material.

this is dan chu, vp of emerging products and markets, from vmware. he’s about to start a panel discussion about extending information technology beyond the traditional data center (the cloud). a bit too high level for me.

the scale of vmworld was pretty immense. this is the self-paced lab room, a huge room full of thin clients with a row of supporting infrastructure up front. the conference staff was nice enough to let me in specially to take pictures.

one of the meal rooms. i estimated about 1,500 seats in this room.

these guys are david baldwin (our technical account manager) and martin klaus (my speaking session owner). they’re stationed at the lab manager kiosk in the vmware booth. in the evening, the expo area was catered as people scatter around learning about new products and getting their badges scanned to receive more information (in exchange for gifts like tshirts, stress balls, and ball-point pens adorned with logos).

by the end of the conference, there was a day/evening of relaxation. retro game stations, air hockey and shuffleboard, food and drink, and rock climbing. the conference ended with a live performance by foreigner.

i’d never imaged taking a picture at a concert with the widelux. i had a few frames left in the camera, had my light meter with me, and thought.. what the heck. meter the stage, hold the camera pretty high, and burn the rest of the roll and see what happens.

all images are taken with the widelux f7, on arista premium 400 rated at iso 6400. developed in rodinal 1+50 for 45 minutes with agitation every 3 minutes.

next week i am giving a presentation at vmware’s annual conference, vmworld 2009 (breakout session VM1461). it’s a pretty big thing for me. i’ve done presentations for user groups but not like this: audio/video crew, session owner, audio recording, wireless microphones, session feedback surveys. this conference will have about 10,000 attendees. the speaker staff is overseeing 400 speakers. there are hundreds of breakout sessions. ya..  *exhale*

one thing i’ve found that has helped me is going through my presentation out loud by myself in a conference room. the proverbial speak in front of a mirror trick. i did this once, gave my presentation to an internal group at work, revised my speaking notes, then gave the presentation to a group at vmware (which was a great opportunity). i tried to do the solitary/room thing once more, but i wasn’t feeling it. my biggest thing right now is being over my allotted time. i have rough timings for each section of my talk, so maybe if i just put those down in my notes and keep a timer….

anyways, the conference is next week. my breakout session is “full” at 70 attendees. i’m not sure how big the room is yet, i have a feeling it larger but full with walk-ins. luckily, i am attending other sessions before mine, then after mine a few others. it will be over before i know it.

image taken with a canon ae-1 program, 24/2.8 lens. arista premium 400 at iso 400, rodinal 1+50 for 13 minutes.

on june 11th, our group at work went to beresford park in san mateo, one of their many municipal picnic areas. it turned out to be a warm day and, with a lot going on at the office, a nice break to be outside.

i brought my rolleicord for the day, with some ilford delta 100. most of my shots were at 1/500 and nearly wide open. looking at this image, i’m glad i took my camera with me that day. i only shot 6 frames on that roll and only scanned two of those, and i’m posting just one.

my coworkers all have very diverse backgrounds and histories. most have them have been employed at the same company together for many years (i’m the “new” guy just passing my two year mark in a couple months).

i love how snapshots like this bring a moment in time (and the people pictured) out of context.

once in a while, my job brings an opportunity to write some sort of documentation.

this week i gave an internal presentation on a vmware product named lab manager. getting a bit tired of the vmware slides, which all use the same small set of icons, i decided to do my own. using a blue pen and copier paper, i drew out all the diagrams for my hand-out. i then used a scan-to-tiff-and-email machine we have at work. then a few actions in microsoft word brought it all together. it looks so official, with the corportate document template and logo. and my document ended up being pretty content-rich. anyways.. i gave the presentation today and gave everybody a copy of the hand-out.

this one illustrates the derivation of multiple virtual machines as they are deployed, checked in and out of the library, etc.

i’m becoming quite an authority on vmware.

(800×638, 2500×1993)

yesterday i took my crown graphic to work to show a coworker of mine. the weather was nice, so we went out to the front of out building and i took this shot. too bad i didn’t have a 25A filter for this!

some detals: 4×5 ilford delta 100, rated 125, souped in rodinal 1:50 for 11 minutes. i think the 1:50@11 is working because the shutter is slow by at least a stop or more. the image is a negative scan.. dust clean-up, levels adjust, slight unsharp mask, resize.

at work i take great pride in what i do. i love working in the server room, a unique physical environment where the computers that run the company live. i enjoy building new infrastructure, and tuning systems. at my current employer, i’ve written scripts that succintly summarize backups, automatically provision monitoring, and completely prepare a system for production after the operating system is installed. i think my role is one of the most thankless jobs in IT, behind networks and email. but that’s ok. i like being the back end server guy. i like being esoteric, orchestrating data movement with custom scripts or rolling out new systems with the push of a button.

i am always trying to improve my processes. i try to keep everything clean and predictable, a conservative approach that works well when your systems are the foundation upon which the business runs. not quite so conservative, maybe, compared to the vms systems wew use to run the manufacturing floor. my specific area of unix administration is very close to that.

i even take pride in how my systems look.

therein lies a bother.

for a long time i’ve had an issue about work that’s been difficult to put my finger on.

i had this discussion with a consultant i’ve worked with over the past several years, that got me thinking.

i am on the cusp of changing jobs. it’s been an emotional experience, which has its own consequences. but for now, what i’m thinking of is my role in this company, my role in the new company, and what i really want my own role to be.

when describing problems with work, i realize i keep in mind concepts such as.. “we have no clear requirements for this sytem (what are the long range plans?)” and “we have no performance benchmark (what is fast, or fast-enough?)”.

a few times a year, i will be involved in some project that requires an intense period of activity for me. i deliver, since doing the higher levels of my job is what i enjoy most. it is the easiest part of my job (to me, figuring out how our business systems can recover from an alternate site is easier than walking someone through changing their login password). i have a stack of awards and certificates in a moving box in my office, thanks to the projects i have been involved with.

i am fairly embarassed by the attention. i write it all off as “doing my job”. i believe that if some who is competent in what they do carries pride for the output is a state of work nirvana. if i didn’t have some pride in what i did, am i really in the best fitting job? this ignores things like hardships and mental or physical inability, i suppose. but the formula is simple: be good at what you do, be proud of what you do. it seems like common sense to me.

very interestingly, my manager has a phrase to describe this. he says, “common sense is not common.” it is a clever phrase and i think it is profound. but is it good or bad?

i strongly believe an IT group should be treated as a consultancy. the nature of unix systems is simple single-function tools put together to make larger systems. this way of thinking forces you to look at the big picture and dissect it into smaller, more reasonable pieces. server folks are used to streamlining processes, finding shortcuts and distributing workload. these days, out of necessity they are trained to find and adapt free tools to solve problems. how great would it be, if a customer with a problem met with an internal resource who knew the existing systems and could integrate free tools or established tricks? there are so many products out there today that are built on free tools. many of those underlying tools are already known by server folks. there are also new technology based products that can perform profound functions. if server guys can move applications between hosts (or even geographical locations), clone entire systems and present them elsewhere with a few commands, and replicate backups on disk as to offset the cost of traditional backup tapes and offsite storage.. what is really possible? getting business people and technology people talking together seems like it would bring the onset of IT nirvana.

on the ground, i live in a world of systems. we pay a premium for hardware because it fails less frequently. and when it fails, it is easier to operate on. it is built by companies that stand behind it, and are generally there for us when we need them. in my world, people are taking ideas and writing them down and sharing them to better the world for all of us. i measure everything, i look for trends to identify the overall health of systems. i find clever ways to monitor components, so i can detect when a subtle issue occurs. it is a world of known quantities, and of striving for perfection.

my consultant friend told me my problem is perfection. i work for a company whose projects get shoved to our department with no clear requirements and benchmarks. despite efforts over the years to insert ourselves into the genesis of projects, we are still getting demands like “we need a machine this week to run an application we’ve already chosen and purchased.” it is frequent that we see processes in place that could be automated, eliminated, or otherwise replaced. if only they would just consult with us.

the phrase my world is really accurate. i dive deep into it, and a complete understanding of it and constant finger on the pulse is what generates a lot of my pride.

but there is a real world. it smacks me in the face constantly, in the form of (seemingly) silly requests, unrealistic timelines, and mismanaged projects. there is an ever-growing dichotomy that i am now realizing. i have already identified a few differences between my world of discrete parts and fucntions.

is this why i am frequently disappointed at work? is this why i am frustrated with projects? i am not exactly sure what is going on with me and my world in this matter. if it is something, i am not sure what can be blamed on it. if there is cause and effect, can it be resolved?

as i continue to understand the world and myself more, i am taking this seriously. it is bringing up so many questions for me.

come to terms with mediocrity, my friend told me. embrace it.

give up this rose colored view of things i have? is common sense really that uncommon? what happens if i give up on perfection, will that change who i am? would i be happier but less unique? is my silly fascination with discrete components and perfection holding me back?

the advice i got was to embrace the real world. use it to elevate myself up the ranks, then use my passion to better others.

is that how great technical managers are born?

i have a lot to think about.

i am on the last hours of my career here at my current employer. i see so many “common sense” things that could be changed. but i’m not sure change can come from the bottom of the totem pole. so i need to elevate myself to be able to affect change? can i trust those above me to develop my ideas, collaborate with my peers, and affect change on our behalf?

can there really be a workplace nirvana?

long story. that’s a sun e10k in the hallway at work. it was such a symbolic posession for a company doing bioinformatic computing. but none of the IT staff ever had the right vision. the machine was misused and grossly underused. but i was lucky enough to learn how to run it.

anyways, i’ve decided to change scenery. starting october 16th, i will be working for a different company. my focus will be similar, but the equipment and software used will all be diffierent.

let me back up a little bit. when i moved to california, i started my professional career as a unix systems administrator. i was hired by a contracting firm. i was out on various contracts got great exposure to a variety of environments over the course of seven years. then i felt the hurt of the tech downturn. i was unemployed for a while. i was back on a gig i wasn’t too enthused about. that time, i looked for a new place. i quit contracting and went full time at a scientific company. i’ve been there for four years.

why am i making this change again?

that’s the most easy and difficult question to answer.

on my way home from work today i was thinking about how to answer that. of course this isn’t the first time. i certainly knew the answer when i started looking for jobs elsewhere. i had some answers when people who were interviewing me asked. and i had little to offer when my own employer asked me. now, composing this blog post, i’m at a loss.

the easy answer now is it feels right for me and that’s really all the answer i need. i have control over my own life. leaving my current employer both hurts and feels good. hurts, in the matter where i’m leaving some coworkers and a director i respect and enjoy working with. feels good in that my new commute will be half, the department seems more healthy, and the company looks stable.

every place has its good points and the bad. this new place will have its own matters-of-suck but will have some good points too.

i can say easily now, that job hunting sucks. most unix IT jobs are held by exclusive searches and headhunters. and they are all getting in the ballpark of 15% commission. they hold all the open jobs. and they all use the internet to find people jobs. why else would they take my reinvented, evolved resume and turn it into a list of keywords and product names. a lot of headhunting is likely off-shored too, as evident by getting lots of VOIP-like caller-id calls, some of which had such poor voice quality I could not make out information left in my voicemail box. another hint that these head searches are search engine based, is the fact that I mention a partciular product as a minor point on my resume. I was getting job offers where the crucial experience required was that product. If a human had actually read my resume, they would have overlooked it.

this job hunt has been an emotional experience for me. this year has really be emotional. dad. burning man. layoffs.

i have reserved some personal time for myself between jobs, but it will not be enough.

so for now, i will finish this post. put on journey and turn up the volume. and finish the last of this valrhona dark bitter chocolate.

(800×601)

today i brought my 4×5 to work, to show a coworker. it was my first time packing it all into my mini trekker classic. i usually carry it in a lowepro side bag. everything fit fine.

i was looking around work for something interesting to photograph. there are lots of interesting cubes, but no real scenery. i decided to set the tripod up on top of my desk. i brought the camera nearly up to the ceiling for this shot. i exposed a sheet of ilford film as well – another for the growing queue of sheets to be processed.

this is a picture of the dolphin resort. the swan and dolphin hotels have been my life for the past week. i am back from sans 2006.

i was going to update at night but found myself too tired to really do much. the days were mostly conference and class work from 9am til 9pm. i met lots of cool people, learned lots of cool things. i have lots of notes, which are on their way to california via u.p.s. ground. when they arrive i’ll annotate my notes with them and post something more substantial.

i have a lot to think about now. briefly, i am not reaching my potential. every workplace has its drawbacks and its sour apples. i definitley know those at my current job. my biggest challenge is being spread so thin. dealing with everything from changing forgotten to passwords to working on site level disaster recovery planning. i really wish i were more specialized. but that’s hard to do at a small company. i need to find a way to manage my time more efficiently. i wish i had a few more days off from work so i can think more about everything.

i’m so tired. i miss eating real food. and california. and ryan and boman.

i have to work on the invites. boman will help me out a little on the techncal details. but i’ll do the rest. good small art project. (did he even wake up today?)

i need to bring this loaner laptop back to work. and format it before i hand it back – i ran some bins from a rooted filesystem on it. it’s totally hosed :)

i don’t like interacting with vendors.

was up until 2am doing chores. i was so tense, when i drank some water before bed my stomach got tied up in a knot.

overslept my alarm by an hour (and still got only five hours of sleep). had trouble with the ticket validator at the train station.

i have nearly a week’s worth of work to do this week (thursday and friday).

- filed more reports than ever, helped others file reports

- got flack on a poorly scheduled and potentially dangerous internal audit

- dealt with multiple groups of external auditors

- none of my 2005 “goals” were done to completion (only one of them has measurable work done)

- the admin i helped hire got fired within the same year

the good items to report:

- did not go postal or flip out under pressure/stress like he did during the data center move

- did not start using pot or other mind altering substances, alcohol consumption remained level

- will return to work after the holiday break is over

part of working for a biology related company is taking part in conversations about life. today at lunch the question came up, what prevents [humans] from living forever?

what we gathered was that the human body’s cells replace themselves every few years. cells can only replicate so many times. you multiply the two together and you get the theoretical lifespan of humans.

the other thing to think about is, every time a cell divides mistakes are made. the copy is not 100% correct. accumulating mistakes is why older people are more susceptible to illness than younger people. it’s why individual lifespan varies so much.

the only ways to prolong life are: reduce the mistakes made in replication, increase the number of times cells can replicate themselves, and reduce the frequency at which cells replace themselves (regenerate). these are the three important variables.

someone suggested that reducing caloric intake would reduce metabolism and tweak the system enough to extend life.

ok, i made the mistake of answering a page and email from work. a website is down. i login and i can access it fine. this is what i get, i guess. the email doesn’t specify a time or method of access, so it’s hard to tell what (if anything) was wrong. if i didn’t need this damned job to make a living i’d work in a video store or library.

time to go home and decompress. parents are coming up soon for financial stuff. in need of “me time”.

today i finally got around to setting up my etrade account and managing my stock options. turns out the stock was worth more than twice my strike price, half my options are vested, and now is a good time to execute and sell. so i did it. this is my first time ever doing things like this. with some guidance from the money-wise don, it was easy. etrade executed the sale in three seconds.

the place i work at, is going through an interesting iterative process to define its project management style. it feels very awkward, having group meetings in which we talk about inputs and outputs of various project phases. the framework for the whole thing is a project lifecycle.

in my own personal research, this buzzword is flawed. the meaning of the work “lifecycle” as i know it comes from biology, where life is born, lived, deceased, and reborn or reused. projects do not regularly spawn new projects upon completion. wouldn’t lifespan be a much better term?

in any case, i’m really wondering why our project management doesn’t do research, come up with a methodology, get buy-in from senior management and key technical leads, then disseminate it throughout our organization. this way we don’t all have to become pseudo-project managers.

my real interest lies in providing good service, getting work done, and making things better. how the projects are managed is not that important to me, to be honest.

just tell me what the interface is, what the steps are, and make sure the communication is there.

interesting read: an improved project lifecycle model

today someone i work with passed away. one week, he was in our weekly staff meeting. he was in the hospital for a week and a half, and now he is gone. amid all our stresses and added processes due to financial audits, we have to take up more slack. not that we mind. this is the third person i’ve worked with that passed away. one to a congenital heart condition, and two quickly succumbed to cancer.