For the benefit of others like me who may be lost in this, this is the system I’ve worked out, and am still polishing. I doubt there will ever be a perfect system, but I’m starting to get close. I’ve been slowly getting through the book Getting Things Done, and while it’s gotten to be rough reading- I’m already sold on attempting a proper methodology to help manage my life, no need to keep preaching- I’ve been incorporating bits and pieces of the methodology in my workflow and it has really helped.

The key component is to get things out of your own head! This is the core part for GTD, and is an interesting academic topic called the Extended Mind. I’ll forget my own name if you let me, so even falling back to the millennia-old technology of ink-on-paper is wonderful. Where the digital age has really helped me out is search. Full-text-search, tagged indexes, the works. I can call up all manner of information for myself at any hour anywhere, reference it at need, and have it beep at me when my attention is necessary. It’s not going to ever put any proper personal assistants out of work, but it’s a level of convenience and worldview self-understanding that hasn’t been available to “common people” for… ever?

I have two categories of where information “lives”. The first, we’ll call the “warehouses” is the canonical, final resting place for personal information. I have several, but each fills its own niche in my life. In no particular order:

• Apple Mail. It’s still the champ, and as most to all really important work-related communication comes through email, it’s a near-complete historical record of all communication I’ve had with my colleagues. Things not to miss: Smart Mailboxes providing a view into your email based on reasonably arbitrary search criteria, and threaded email conversations making it easier to pick out replies in ongoing email conversations.
• Apple iCal. This tells me things that demand my attention at fixed points in time- a calendar, if you will The tasks management in it is both horrible and doesn’t really sync well to anything, so I’ve completely ignored it for something else. This may be fixed in the update to OS X Lion, but I don’t hold my breath for non-shipping products.
• OmniFocus. Task management of the gods. They’ve tried very hard to make it easy to follow a GTD or GTD-esque system with it, and it syncs to both my iOS widgets. Also, the iPad version of it is both fully-featured and hands-down the best GUI I’ve ever had the pleasure to work with. It integrates reasonably well with Apple Mail as well, providing a system service to magically create a new inbox item from a selected email.
• Evernote. This is where all my random text lands. I used to keep a series of notebooks at work (Roaring Spring’s 80-sheet 5×5 quad-ruled composition books, in case anyone’s asking). The problem with notebooks is they’re not searchable, and then I also need to bring the notebook EVERYWHERE. Evernote fixes all of that.

That’s where I keep everything. Everything is also reasonably hotkey’ed, so some short keystrokes bring me to where I need to be to start typing things out immediately, so most things land where they need to be.

There’s a few other things that basically fall under “input/output” to my system. These are basically write-many read-once sources that information that scrolls too far back gets lost forever, or just things I source for communication.

• Trac. It’s a web-based project management and issue tracking system. There’s a wiki for company documentation, and a task tracker/manager thing. My tickets on there generally become projects in OmniFocus, and then I work with them through there, save for ticket-related communication. That does tend to stay as notes on the ticket, and I do reference that from time to time; that makes Trac a little awkward in this discussion.
• IRC. It’s our office conference room, break room, and party rooftop. Real-time chat is wonderful and useful beyond belief, but I need to make sure that when people ask me for something, I make a note of it. OmniFocus hotkeys are golden.
• A paper notebook and pen. The old standby. I’ve given up on using this as the core method for keeping my head together (see above), but nothing beats the ease of just opening it up and writing into it. These days I typically will scribble a few things down, and then copy it in later to The System. FUN FACT: Evernote does great OCR off of written pages, and I can just scan in my day’s scribblings into Evernote and have it fully searchable and selectable. In case anyone’s wondering, I use a pocket-sized Moleskine, and a Pilot Varsity disposable fountain pen.

It’s not the simplest of systems; it’s certainly not the most minimal setup one can find out there, but it’s been refined and generally does work. Nearly all of my failures to follow through on something these days have been failures to follow my system, not of the system, and that’s a good sign for it’s “workingness”. The key point is that my core apps sync beautifully between my Mac, iPad, and iPhone. I’m rarely feeling lost or adrift in what I’m supposed to be doing at work or what I know I was just thinking about anymore, while steadily increasing the amount of information I have to process on a daily basis. In otherwords, it works, and because of that, I work. That’s a good enough endorsement for anyone, isn’t it?

I realize that the real world makes a lot of things difficult for those who collect data. Consider this, then, a gripe list of things I may or may not expect to get fixed.

Moving on, I work on computer models to better understand physical processes. These models are built and parameterized off of real world data of some way or another. This means I consume a lot of data in my job. However, how I as a modeler would like data is not often how I get the data. Here’s some of the bigger issues:

• Oddball formats. On a given day, I have to deal with files in Excel format, varieties of plain text, oddball Fortran binary files (undocumented, of course), NetCDF, HDF, ESRI Shapefiles, GeoTIFF, and a whole bunch of other raster formats. And this doesn’t particularly phase me, either. What phases me? Incomplete oddball formats. Whenever a new and interesting bit of geodata ends up on my desk in an oddball format, I can wind up spending over a day poking at it. When I get gridded data, I need to know a few details about each datapoint in the grid:
  1. The lat/lon coordinates
  2. Units being used
  3. The digital representation of the data

  Most data I get tends to follow this: well-put-together NetCDF & HDF files, as well as anything produced by or with a GIS (such as GeoTIFF, Shapefiles, etc.), tend to be alright. I can work with this. However, if I have to figure things out, it slows me down. Sometimes a lot if I misinterpret it- if it looks reasonable to my eye but actually is not, this Can Cause Baddness. This is largely the case with Fortran binary files, but I get a lot of plain text that also shares these problems- I have an unclear idea of where or what is being represented. Examples:

  1. Points are addressed from 0 to 360 degrees east instead of -180 to 180 (or vice versa)
  2. There may be a README file with the line “value is thousands / 20″ or something similarly vague.
• Oddball Measurements. Let’s start with something basic. If a value is between zero and one, then why do your measurements include numbers both less than zero and greater than one? Saying “sometimes in real life, things work out like that” doesn’t satisfy me in building a numerical model where the textbook definition of a function expects a value to be between zero and one. Tell me why this is, with perhaps a textbook reference going more in depth. I don’t mind that your measurements falls outside the traditionally accepted bounds of reality . I mind that there’s no systematic explanation why this is.
• Oddball time-series. In modeling, we want to see a lot of things over time. The more fine-grained our models become, the more fine-grained we want our measurement data. If I’m simulating the growth of a crop, I would like to see the growth of the crop as well as after it’s done growing. It’s hard to determine a trend if one only has one to two datapoints to work from.

I suppose the grand summary of this griping is that as a modeler, I have my hands full with a lot of things. Spending weeks trying to work with bad data is frustrating; human hands entered the data into a computer, so why can’t those same hands explain the data?

Carbon-nitrogen Cycle modeling
Crop growth modeling
Climate generation
Coding
Climate modeling

… the only outlier is the letter B for Biofuels.

Last week involved a lot of things that involved a lot of brainpower on my part. I had to do a tricky interpolation of a dataset, get a multi-directory, F77/F90 source dependency generator working, re-factor a bunch of F77 I/O into some F90 that I can then wrench around to do as I require, and some analysis on some data to send to a model comparison.

It was all going swimmingly until things started breaking late Thursday. Then come Friday, everything that could have gone wrong over the whole week did. And that’s why I’m sitting here in the office on Saturday, plodding in a new sentence or whatnot with each short re-analysis until I get things done. There are things I missed in my work on Monday that led me here- silly errors that still resulted in things coming out to the right pattern and the right order of magnitude, but threw model results for a tizzy. I suppose that’s why I’m not too grumbly about things, but damnit, I was supposed to take down my Christmas tree this weekend! It’s been up so long the cat’s lost interest in it!

What troubles me is that, at least in my place of employment, practically no-one really understands computers.

I work in a primarily Fortran shop, which many of you would think “well, there’s your problem right there.” Not so much, as there’s a huge literature of already perfected Fortran code to solve nearly every tricky numerical gymnastics problem you can think of (and twice that number of ones you can’t think of). Fortran90/95′s greatest win, and flaw, is that it has incredible backwards compatibility with old FORTRAN77 code (and even some FORTRAN66!). Re-writing the millions of lines of FORTRAN77 that passes through the department on a semi-regular basis would be costly in sanity, time, and money.

But what about how you work with that code? Some of us know a certain friend of mine who sat up for hours manually inserting rows into a dataset to account for missing time in hourly reports. It’d have taken half an hour to whip up a MATLAB script and have it loop through the dataset. Almost nobody uses any form of version control (those that do have a sea of directories resembling projectname_vers-num, with num being their revision number), and most of the Unix programming is done through running VIM in vi compatibility mode. Changing an identifier within the code from one to another is typically a day-long affair for most (hint: it should take perhaps 5 minutes, 10 if the SAN is feeling sluggish), and inter-routine data dependencies are best described as a maze of twisty passages, all alike. Makefiles are written in a manner where it’s a requirement to run “make clean” before any recompile, and some folks swear by Intel Fortran’s -save flag, which makes all local variables static. That flag is used entirely for its side effect of initializing all local variables to zero, however, and not a thought is given for the actual effects.

This is actually a rather sad, sad state we’re in right now. It isn’t anyone’s fault, either. There’s no overarching culture of programming and software engineering at my place of employment, so there’s not many examples of folks “doing it better”, with those of us who are being categorized as “Oh, but you’re one of those crazy geeks who just know how to do it.”

I’m one of those crazy geeks because I sat down and forced myself to learn how to do it, because I saw that it would pay off, in dividends. It has, too.

I’m not trying to claim I’m some wunderkind, either. There’s gaping holes in my expertise and practice that hinder me daily (which I’m trying to correct). What does it come down to for me? Right now, I’m sitting down and reading the GNU Emacs manual. I can’t be arsed to dig up my copy of The Pragmatic Programmer, but there is a section in there that says one should really master a text editor. I’ve taken it to heart, and with my limited time, that’s my “journey of a thousand miles starting with a single step.”

What have you done to be a better programmer today?

Well, I believe I have a Nasty Condition.

I hack on FORTRAN and Fortran at work, after getting out of the IT support gig. What’s the difference between the capitalizations? Easy. FORTRAN references the language pre-1990 (1950′s through to 1990), and Fortran is the more modern Fortran 90/95/2003 standards.

Fortran 90/95/2003 isn’t that bad to work on, actually. For its primary purpose (running numerical computations on hideously expensive computers), it works really well, and makes me a happy hacker.

FORTRAN helps to create ugly code. Complex looping and branching statements are much more difficult than in other languages, and many FORTRAN coders cut their teeth on it when the language was still implemented via punchcards, and “eloquent” coding meant as few cards (aka lines of code) as possible. Goto statements scare me. Gotos with literally 5-6 arguments terrify me. Did I mention that there’s limitations on line length imposed upon the language from punchcards?

The code in specific I’m working on is worse. Those working on it use no version control, so it’s full of commented-out things every which way, and it definitely dates from the punchcard era, as it lacks things like blank lines in between logical blocks of code statements. It has terrible, nonsensical indenting, and it relies heavily on subroutines acting in highly non-documented ways on literally hundreds of different, poorly named and undocumented, global variables.

Oh, and there’s a function that simply wraps ACOS(), the cosine function in FORTRAN. This is what it does:

FUNCTION ARCOS(X)
ARCOS = ACOS(X)
RETURN
END


Lovely, yes? Waste of a function to have to keep track of, yes? I’ve seen it in three different versions of the model, and it troubles me the same.

I also saw references to MS-FORTRAN 4.0 for MS-DOS in the comments. Joy of Joys. In addition, it will only compile with Intel Fortran. Portland Group’s compilers break a little bit on it, but gfortran, g95, g77, f2c, and sunf77 compilers all fail with pretty pyrotechnic displays with it.

Hopefully this stuff won’t drive me completely batty before I can understand what it does “under the hood” enough to re-write what I need out of it in something better.