Wednesday 27 July 2011

Ars Moriendi

A couple of days before the end of my vacation I received an email from Andy, the Project Manager at Fjord Systems.

"Pike, I hate to ruin your holiday, but everything's gone wrong. We're in deep shit."

Andy wouldn't tell me anymore. I arrived back in Sweden a couple of days later expecting to find the common area festooned with intestines and severed heads in the fridge. It sounded to me as if the project had died hard, but what I really found was... nothing.

The team had told me that they were good to go when I had left for my vacation, three weeks prior. They had all agreed to do the work. They knew who was supposed to be doing it, and how. That was what they told me.

Apparently, once I had left the office, instead of buckling in to do the work they had panicked. As best I could tell, the team had spent three weeks running in circles around the office, screaming and throwing their own faeces at each other. As far as the art of dying went, these kids were a bunch of amateurs.

Anders had finally asserted himself a little bit. There wasn't enough time to undertake the full scope of the project I had laid out, so they would be cleaning up the old core engine (the part that he'd told me made him feel sick to his stomach) and reusing it more-or-less as it was. The so-called 'core engine' was the piece of the product that actually did the majority of the work of the app, but it was not necessarily a complicated piece of technology. It sprawled a bit, but I did not believe it would have been difficult to rewrite it, or simply to break into chunks in preparation for a rewrite... as per the designs we'd agreed upon. Nonetheless, the team lead had finally made a decision and I decided to go with it. He agreed that the core engine would be integrated as per my designs so that it could later be unplugged, piece-by-piece, when we had finished this initial refit. Fixing foundations and the plumbing was the main thing.

I left the office with a bad taste in my mouth. Despite the fact that everybody had agreed to stick to the design, I'd heard them muttering. They didn't believe we could make it work. They didn't believe that we could make the deadlines. I was frustrated. I'd done this all before, with less people, and I knew it would work if they'd stop bitching and start coding. But I couldn't say that to them; that was Anders' job.

I went home and I tried to get to work on the driver integration with Chucky, but it wasn't that easy. My working hours in Australia would end right when the team's would begin, and I would finish a full day's coding with hours of emails, realtime chats and conference calls with the team. Every morning I would get up, sync down the source code, and find that the team had broken what I was working on the day before. Half of the productive part of my day would them be lost fixing it. I don't know what the team was actually doing, but it was clearly not working, since the project wouldn't even compile after their checkins.

I asked them nicely not to break the build. Weeks went by and I still found myself having to fix it on a daily basis. I tried a sterner approach, but this didn't help matters. I didn't yell down the phone, but once I abandoned politeness, the build breaks became less frequent... and meetings became commensurately more surly. I could hear them muttering at me in Swedish.

Joseph was granted a patent that he had filed. Nobody seamed to care. "Software patents don't mean anything," I was told.

Srinith and Sven complained to me that nobody else was working. Anders and the senior guys sat around in their office with the door closed. Tyko was wrapped up in his research. Nobody knew what the UI team was doing. I told that they were the A-team. They were the most productive guys and they were on board with my desire for progress. If we hasd to do this with a team of three, plus Chuck,y then so be it--I'd done it before, back at ATB Software, and this time I didn't have interference from management and marketing to contend with. I had to trust someone, so I trusted them. I asked them to look after the integration with the core engine and the UI team while I pulled my head in and finished the driver interface, and they agreed. This was a mistake.

My A-team introduced some new code libraries to instrument the source in order to catch memory leaks. This library was incompatible with my test apps, but it was cancerous: once it was in the application it proliferated everywhere and could not be removed. I complained: the new framework did not leak memory and the overhead of the instrumentation was huge. It threw lots of false positives and it made debugging a nightmare. Only the core engine needed to be instrumented, and there needed to be a way to turn the whole thing off. But they were adamant, and I had bigger fish to fry. I wrote new test apps and got on with it.

My A-team took the very simple structure that I had created to be a unit of currency between all of the subsystems and wrapped it in layers and layers of macros, so that they would act in an object-oriented way without being truly object-oriented... or debuggable.

My A-team changed all of the error handling routines so that, instead of passing error information up, they would throw assertions that would crash the application.

At this point I decided that I needed to rein them in, and I think they simply decided that the would instead replace me. They started to pull apart the infrastucture I had written, stripping out the inheritance hierarchy that would permit us to drop in new versions of the core engine. If they needed to make a small change to a class that I had written they would push the entire code module into an 'attic' repository and then create a new one... containing almost exactly the same code, but minus the history of changes and of course with his name on it as the original author.

I just took it. The design was all that mattered. If we could stick to the design for this release I would be able to steer the team back on course after the release date, I thought. I was up to my eyeballs in the driver interface and I didn't want to fight about it. But I knew that they were gradually stripping away all of my design and replacing it with Special Magic.

I start to get bug reports from the communications layer. I'd forgotten about the comms layer, with all of the other nonsense, but I didn't think it was a major issue. I couldn't duplicate the bugs, so I asked for log files. No matter how often I asked, nobody would furnish them to me... but the reports kept coming.

I did get a number of feature requests. "This comms layer is useless," Srinith would say. "It doesn't do X."
"It was designed so that could be built on top of it. It's easy enough."
"It should be handled inside the comms layer."

Like an idiot, I would then extend the comms layer to handle that behaviour, rather than insisting that the behaviour should be handled externally. Instead of thanking me, Srinith would then say "This comms layer is useless. It doesn't handle Y."

I did that three times, until Srinith ran out of new features. The comms layer was still buggy, and still a source of complaint, but by then Chucky had finished the driver and I had finished integrating it. Once I was able to focus my full attention on it, I was able to duplicate the underlying bug in the comms layer... and it was a serious one. Srinith jumped on it.

"Alright," I told the Andy, Anders and the A-Team. "I found the bug and I can fix it, but I need two weeks. Ten business days."

I showed them a design document, and they agreed to it. I got on it, and made good progress. Srinith threw some new feature requests at the comms layer for me, which he 'needed urgently', so I build those as well: twice each, once for the old comms layer (so that Srinith could continue programming with it), and once for the new one. On the ninth day I had finished all of the features and I was chasing the last remaining bug when Andy called me up. "We're pulling the plug on your comms layer. Srinith is going to write a new one."

I was on schedule. I would have the layer ready for them, bug free, on the day I had promised. It had the same interface as the old one; nobody would even notice when the new library went in, except that the bugs would disappear.

"Sorry. Anders told him to go ahead." In other words, Anders needed me to fail.

"What's he going to build? What's the design? Is there a document?"

"He's just going to do whatever."

I could see the death of my baby looming.

Joseph was laid off. Martin sent me a note to assure me that Chucky and this had nothing to do with Chucky and I, it was just that they didn't think Joseph's research had a place at the company.

A week later, Chucky quit. The drivers were done and he'd been offered something that paid better and gave him health coverage. Also, he said, he was tired of all the sniveling. I decided that I was, too. I was required to give three months notice, so I held out a few more days before I handed in my notice. My last day would coincide with the release date.

I wanted to make a graceful exit.

Friday 22 July 2011

Misfits of Science

University was a bit of a shock. I had a very heavy courseload in first year... more contact hours than I had in my last year of high school... and it fell to me to make sure that all of the prac classes and tutorials I would have to attend were timetabled correctly. I didn't actually know what a 'tutorial' was, and I wasn't much interested in finding out.

The other thing was the maths.

I had never enjoyed maths, but I was good at it. At High School, maths was one of my best subjects, at least in terms of grades. But apparently the university wasn't happy with the standard of maths from incoming students, and we were all subjected to a preliminary test before the first term began to prove our competence. And it was just as well. I scored 12 out of 15 on the test, but all three of the questions I got wrong were all the advanced trigonometry questions. My class at high school had done a different unit when others had done trig, and I hadn't learned any of the advanced stuff. So, on top of the heavy courseload, during first term I had to take extra classes in high school level trigonometry. But it paid off: when I resat the trig test and my marks went from 0 to 100%.

But all the classes were dull. They were teaching Pascal, which I was already bored with, and I didn't learn much in the first semester. My most difficult class was Deductive Logic, administered under the philosophy faculty by a lecturer, Hermann, who was no longer teaching any comp sci because he'd failed most of his students in his subject area in prior years.

Hermann taught us formal reasoning, and it was a lot more difficult than I had expected. They were small classes and I on one occasion fell asleep sitting in the front row, directly in front of him. By then I was starting to perfect the art of only working as hard as I had to, so I got through it with reasonable grades.

I soon found in the following semesters when we took boolean logic in maths and computer science units that I had already covered the material in much greater depth, and those subjects proved easy. A valuable subject, as much as I had disliked it.

But then I pretty much hated everything. I had made a few friends, but I wasn't enjoy  my subjects. I wanted to drop out every single semester. Later in first year we were taught about dynamic memory... pointers... and this was something at least new to me. I quite distinctly remember the a lecturer telling us that pointer arithmetic should be done on paper; it was too difficult to do it in one's head... which any commercial programmer working in a native language (at that time, the vast majority) will tell you is patent bullshit. But it was true that pointers were something many students couldn't master. Pointers, I think, are the first big conceptual leap you need to make in order to become a real programmer.

I did not, at that stage, appreciate that most of the people in my classes would never be able to be effective programmers in the real world: even if they could master the technical aspects, the ability to sit down and solve difficult problems all day long is not one that most people are wired for. For my part, I enjoyed the problem solving and I flat out just liked making things, but I felt like I'd already done all of these things before. I wanted to build Skynet, but nobody else was interested in that. It was data structures and algorithms I mostly already knew, employed in the service of meaningless tasks that had already been solved a thousand times. At the end of the day I was promised a semi-lucrative career maintaining ancient software on obscure hardware that would likely be used only by banks, for exciting banking purposes.

I wanted out, but I didn't really know what else to do.  I made the Dean's list in first year despite my misery. Second year my courseload was lighter. No more maths, no more deductive logic... but the comp sci classes were less interesting. We did a lot of hardware and operating systems subjects, the only one of which interested me was the brief unit we did in assembly programming. For the third time I found myself studying boolean algebra. I was still bored and I still wanted to drop out.

In third year we got to choose subjects. I knew I didn't want to take anything to do with networks; I was terrified that it would lead me to a career as a system administrator. Sysadmin, I reasoned, was the most miserable job in the world. If the network is running fine, nobody notices. If the network goes down... which they frequently did, and do... even if it's no fault of your own... everybody hates you. Not for me. I wanted to get out of university with as little work as possible, so I tried to sign up for a bunch of easy subjects... but those all had Databases as a prerequisite. I signed on for Databases and somehow then found there was no room for the easy subjects. Databases proved to be the second-most useful subject I took, although it was a long way from being my favourite.

I enrolled in COBOL. I wanted to do C++, but, owing to the strange degree I had enrolled in, I didn't have the prerequisite year of C programming. Third year had a lot less contac hours than first, so I went to the C++ lectures anyway. Within two weeks I decided that I had to find a way out of COBOL and into C++. If I learned COBOL there was the horrible possibility I would one day have to program it in the real world. By the same token, I knew that C++ was a viable language. COBOL was for retirees; C++ was for powerful young men. Besides: at that time the only language I was any good with was Pascal, and I knew that there were no careers in that.

I had to get into C++. There was nothing for it but to try it on, and see what I could get away with... but in the end it was no difficulty. The professor had seen me in his lectures and he just signed me into his subject, without even asking if I had the prerequisites. I dropped COBOL as quickly as I could.

This was the single best thing I ever did in my five years at University.

I liked C++. I had a bit to learn, but I was able to pick up most of it out of the book. I knew it would be valuable and I paid attention and that, more than anything, is the basis of my career in software. Naked C++.

The other big challenge of third year was the Software Engineering team project. I wound up on a team with one friend and four strangers. The project itself was dead boring... an inventory management database app... but we divvied up the work and I buried my head in my area, which was the User Interface. I took on board my task and decided to let everybody else get on with their own. This was a bad idea.

For all the time we spent in documentation, we really had no plan for how to integrate all of the pieces, and ground zero for this was my UI. When the time came, and I saw how inconsistent and flat-out terrible my teammates' work was, I had to get up and leave the lab. I wanted to punch someone. Did they have no pride int heir work? What the hell was I going to do with that mess? The team lead had flown off to China to look for a wife. The programmers who'd made the mess had no idea that they'd done anything wrong. That left me and the 'chief coder', Adrian, whose job it had been to oversee everybody else's work and make sure it would all integrate. I'd done my bit; I decided that it was on him.

This was very selfish of me, but I had big projects due in other subjects that the rest of the team did not. I went off to work on those and I left Adrian to it. Luckily, he rose to the occasion. Adrian worked long hours, day and night, and he got it done. He made it all work and I think our project scored better than anybody else's. I felt bad about having left Adrian to all the work and resolved to never do that again... and karma has since paid me back many times over.

I didn't make the Dean's list in third year, but I didn't go to many classes, either. All the same, I completed my degree with good enough marks that they offered me an honours year, which I accepted... and then deferred. I couldn't handle another year on that campus, with those subjects. Computer Science? What does that even mean? We don't say 'physics science' or 'chemistry science' or 'mathematics science'. A computer is a device, it's not a branch of science.

And what exactly had been scientific about my degree, really? What sort of research did we do, what new discoveries were we making? I had taken classes in other science disciplines, and even the 'soft' sciences were more research-oriented than CS. What I'd learned was really a kind of engineering, no matter how my degree was named. Since leaving I have never once had the word 'scientist' in my job title (although I would, for a time, hold the title of Advanced Researcher).

Did I want to be an engineer? Did I want to go and work for a bank? No, I decided.

I wanted to build killer robots.

Wednesday 13 July 2011

Paintball

There were some new faces on the team at Fjord systems: Lars, Srinith and Sven. Lars was quiet and earnest. Sven was a punk rock hacker just out of high school. Srinith was the only one on the entire team who gave any feedback on my documents or my code. The code looked really good, he said, but he asked me to justify some of the most basic assumptions that went into the design. I was pleased that somebody was thinking about it and I was just as pleased that I was able explain them to his obvioussatisfaction: a young, talented coming into the project without any preconceived ideas about how it should work.

Sven was now in my chair in the office with Tyko (formerly seat of Åke the Genius), so I was shunted to a spare desk in a room with some marketing guys. That was fine; my main business would be conducted in the conference room, which I requested for an hour every day.

Finding the correct hour was a problem. Not because the room was booked, but because the team took the notion of 'flexible hours' to extremes. We couldn't start before 10:30 because that was the earliest some of the team members would arrive. We had to wrap the meetings up by 1:001, because that was when some members liked to go to the gym. And so on. Nobody was willing to budge.

I got on with it. I presented the essays I had written, which had been available to everyone for feedback for months. It got the usual cool reception, but everybody was nodding and agreeing and nobody was shaking their head. The main thing that bothered me was that the senior guys... Anders and his crew... were not attending most of the meetings. They were working hard on the current release, they said, which was indeed due out right about then. I remembered Tyko's explanation that they were the only ones who actually did any work, so I left them to it.

I didn't see or hear much from Tyko at all, but someobody reported to me that he'd had a look at the infrastructure I had written and declared it to be 'quite good'. "That's great," I said. "Pike, you don't understand. That's huge. Tyko never says anything good about anybody's code." I remembered Tyko's admiration for Åke the Genius and decided that I would let the matter drop.

I started to present the code I had written formally; showing how everything worked and how the pieces would fit together. The version that Anders was working on shipped and they started to join the meetings in time for me to demonstrate how the system would run, end to end. The code was there for anybody who wanted to play with it; I wanted to keep no secrets.

There was no dissent, so it was time to divvy up work for the functionality build-out; to design the components that would sit inside the framework. I had laid out some guidelines for this, but I wanted Anders to step up and do this part: it was his team and he had the most experience with the existing code, which we would be cannibalizing for this purpose. He'd said he was in, but I wanted to see him put his keyboard where his mouth was. I wanted to let him show that he was indeed the team lead.

These meetings went as before. Nobody contributed much and I found myself working through fresh designs on the fly, on the whiteboard. Anders would say nothing at all. But once I called an end to the meetings, the room would suddenly fill with discussion in Swedish. I was the only person in the room who could not speak the language, and the Swedes had actively discouraged me from trying to learn any.

Chucky arrived with his wife, and they were set up in a hotel suite a few doors from mine. I had companions to dine with now, people to hang out with on the weekend. I was excited. Chucky and I conducted extra sessions for the team so that they could learn about how the drivers would work and discuss what they would do. It was decided, as expected, that I would do the integration with the drivers. That was fine by me: it was delicate work and I'd done it before.

I met with the UI team. They had been building some kind of a fancy framework upon which they were going to build a brand new interface for 'my' version of the product. They'd been working on it for years and did not at that point have a single line of code in production. I asked for a demonstration of what they had been building and they showed me a form with a single, very beautifully-rendered button on it.

I asked Anders if we could count on them to deliver a full-blown UI and he shrugged. Nobody paid attention to the the UI team and they had no idea as to how capable they were or were not. As far as I could tell, they'd spent years inventing a button that I could have had in 15 seconds using the existing Microsoft framework. I decided that if the worst happened, we could whip up a UI quickly, so long as the team understood my messaging system. So I devoted some time to that.

Meanwhile, Anders and his cronies began to have closed door meetings. After a few of these they called me into one of these and told me that they wanted to offer me an 'alternative design' to the one I had been working on for six months and had now spent weeks bringing the team up to speed on. They'd put probably three hours of discussion into it, and they had no diagrams or documents.

This 'alternative design' was a giant black box that looked exactly like the old design, which Anders had earlier told me that he hated. When I asked how he would solve the specific problems I had been trying to address with the redesign he waved his fingers dismissively. "Oh, we'll just do some special magic," he said. Every time I raised a specific problem, he offered Special Magic as a solution.

What he meant was, he and his trio would hack it all together in way that would lead to exactly the same disaster I was now trying to fix. I argued as politely as I could, and then a bit more savagely when it was clear that they weren't listening. In the end they had to concede that my design offered solutions and theirs did not. I took Anders out for lunch to try to smooth over any hurt feelings and I believed that I managed to do so.

Chucky got to see all of this firsthand. "I just work on the drivers," he said. "And I'm happy to stay in my box... but I'm glad it's going to be you who integrates them. These guys are idiots."
"They're smart guys, they're just not used to being able to..."
"Pike, they're idiots. This is going to end badly."

Chucky went back home to the States to work on the drivers. My stress levels were high. I couldn't sleep. I was bored and lonely, shut up in the hotel room by myself, but I found that most nights of the week there was some crew from the office (seldom from the development team and never from Anders' crew) going out for drinks. I went with every chance I had. Suddenly I was drinking as much or more than I had been during my last year at ATB Software.

We started to nail down some designs for the new components. We assigned some of the work to the humps that Tyko had warned me about and found that they actually delivered some good ideas. Anders and his guys didn't do a damn thing, as far as I could see. Tyko was wrapped up in some research projects of his own that were supposedly far too technical for anybody else to look at. Nobody wanted to talk about Jacob, working by himself in Vegas.

Sven had become very interested in the infrastructure I'd written. I wanted somebody to help me maintain it as it came under strain from having real components integrated with it, so I spent a bit of time training him up. Then one day he came to work one day with a new haircut. The same haircut that I was wearing. I had an apprentice.

Sven reported some bugs in the communication system that I had written. I looked into them, knowing that the comms layer needed work, but I found flaws in Sven's code that fixed all of the bugs I could duplicate with my testbed apps. Sven claimed there will still problems, and I was sure he was right... but I was running out of time. I would have to sort them out when I got home.

Somebody organized a paintball session for the final weekend of my trip. I showed up hung over and ragged, but I figured that I had to. Sven demanded to be on my team, but I was finally running out of patience with him. "Sven, you don't have to always be on my team."
"Dude, it's paintball, and you used to be in the Special Forces."
Again with the special forces. Where was this coming from? "Sven, I was never in the Special Forces." "Well, you know. The Marine Corps."


"Sven, I'm from Australia. We don't have marines. I was never in the army."

He went to the other team, but it was clear that he didn't believe me.

Hangover paintball was absolutely no fun at all. I came out of it exhausted and covered in bruises. In fact, it felt like just another day at the office, chasing people around and dodging pot shots fired from under cover. But that was okay.

I was done in Sweden for the moment, and I was about to take four weeks of holiday. The team had all agreed to their various tasks and we had the roadmap all sorted out, right up to the release date. The worst was behind me, I thought. Now we just had to put our heads down and work.

Thursday 7 July 2011

The Pit and the Pendulum

I got straight to work when I got home, jumping right into every programmer's favourite pit: documentation.

I wrote documents explaining the current system and what was wrong with it. I wrote a document about how to proceed. I wrote design documents for all of the back-end of the application: how the pieces would fit together, physically and logically. Everything was to be data-driven, and based on a really clean interface. I wanted to be able to really break the application into pieces which we could be replaced or re-engineered without risking the entire system. We'd scavenge what we had to from the old app and build as much new as we could.

The integration of third party software was becoming very important to the business and I wanted to be able to drop these new external pieces into our own product with only the thinnest of of shim interfaces. Likewise, I wanted the components to be easy to re-license individually or as a whole: that, too, was of increasing importance. I tried not to get too bogged in detail: I was the architect, not the team lead, and I knew that I would not be able to supervise the whole work effort. I wanted the other people to be able to have their say. As long as they understood the philosophy of it and they built code that conformed to the interfaces I felt that my job was done. If anything broke it would be easy to fix it in isolation.

It looked great on paper.

I think Jacob was the only one who responded to the docs, and I'm almost as sure he's the only one who read them. I took that for agreement and I got on with it, so I went of to build the infrastructure the app would hang on. This would the startup and shutdown sequence for the various business components; thread marshalling within and between those components; system maintenance for updates and upgrades; and of course communication with other elements that were outside of the main process or distributed across the network. I was running out of time, so I rushed through this last piece and it was flawed. I would come back and clean it up, I decided: more important that I be able to demonstrate how the system would run, from end-to-end. I wanted to show the pendulum swinging a full arc in both directions.



While this was going on, Chucky was beginning to design the new drivers and we consulted with each other about how that element would be integrated into the system. We'd both worked in these roles before and we knew what was needed. I didn't feel like I was part of the team in Sweden, but it definitely felt as if Chucky and  were working together on something cool.

I built dummy components to plug into the infrastructure I'd been working on and I found that all the pieces fit exactly as planned. I was as proud of that effort as I have been of anything in my career  in my career.

I got ready to fly to Sweden to present what I'd done. The plan was that  I would clean up the comms system  and supervise integration while team built out the actual components, shoehorning as much as they could scavenge out of the old system into the new architecture. I would additionally built the component that connected to Chucky's work. All of the conceptual stuff was done.


Jacob was not coming this time. He wasn't really part of the refit, and he had now officially been relegated to individual research by Martin. Jacob would stay home and work on the matrix project we'd worked on in Vegas, as well as some other ideas he'd had backburnered at the time. Nobody was much interested in talking about Jacob or his work and I was worried for him, but I had my own problems to deal with.

I arrived in Sweden after thirty-six hours, odd, went tot he hotel, caugth a shower, and showed up at work right before lunchtime. The desire to get things done, I think, was the beginning and the end of my troubles at Fjord Systems.