I realize that the real world makes a lot of things difficult for those who collect data. Consider this, then, a gripe list of things I may or may not expect to get fixed.

Moving on, I work on computer models to better understand physical processes. These models are built and parameterized off of real world data of some way or another. This means I consume a lot of data in my job. However, how I as a modeler would like data is not often how I get the data. Here’s some of the bigger issues:

• Oddball formats. On a given day, I have to deal with files in Excel format, varieties of plain text, oddball Fortran binary files (undocumented, of course), NetCDF, HDF, ESRI Shapefiles, GeoTIFF, and a whole bunch of other raster formats. And this doesn’t particularly phase me, either. What phases me? Incomplete oddball formats. Whenever a new and interesting bit of geodata ends up on my desk in an oddball format, I can wind up spending over a day poking at it. When I get gridded data, I need to know a few details about each datapoint in the grid:
  1. The lat/lon coordinates
  2. Units being used
  3. The digital representation of the data

  Most data I get tends to follow this: well-put-together NetCDF & HDF files, as well as anything produced by or with a GIS (such as GeoTIFF, Shapefiles, etc.), tend to be alright. I can work with this. However, if I have to figure things out, it slows me down. Sometimes a lot if I misinterpret it- if it looks reasonable to my eye but actually is not, this Can Cause Baddness. This is largely the case with Fortran binary files, but I get a lot of plain text that also shares these problems- I have an unclear idea of where or what is being represented. Examples:

  1. Points are addressed from 0 to 360 degrees east instead of -180 to 180 (or vice versa)
  2. There may be a README file with the line “value is thousands / 20″ or something similarly vague.
• Oddball Measurements. Let’s start with something basic. If a value is between zero and one, then why do your measurements include numbers both less than zero and greater than one? Saying “sometimes in real life, things work out like that” doesn’t satisfy me in building a numerical model where the textbook definition of a function expects a value to be between zero and one. Tell me why this is, with perhaps a textbook reference going more in depth. I don’t mind that your measurements falls outside the traditionally accepted bounds of reality . I mind that there’s no systematic explanation why this is.
• Oddball time-series. In modeling, we want to see a lot of things over time. The more fine-grained our models become, the more fine-grained we want our measurement data. If I’m simulating the growth of a crop, I would like to see the growth of the crop as well as after it’s done growing. It’s hard to determine a trend if one only has one to two datapoints to work from.

I suppose the grand summary of this griping is that as a modeler, I have my hands full with a lot of things. Spending weeks trying to work with bad data is frustrating; human hands entered the data into a computer, so why can’t those same hands explain the data?

Last week involved a lot of things that involved a lot of brainpower on my part. I had to do a tricky interpolation of a dataset, get a multi-directory, F77/F90 source dependency generator working, re-factor a bunch of F77 I/O into some F90 that I can then wrench around to do as I require, and some analysis on some data to send to a model comparison.

It was all going swimmingly until things started breaking late Thursday. Then come Friday, everything that could have gone wrong over the whole week did. And that’s why I’m sitting here in the office on Saturday, plodding in a new sentence or whatnot with each short re-analysis until I get things done. There are things I missed in my work on Monday that led me here- silly errors that still resulted in things coming out to the right pattern and the right order of magnitude, but threw model results for a tizzy. I suppose that’s why I’m not too grumbly about things, but damnit, I was supposed to take down my Christmas tree this weekend! It’s been up so long the cat’s lost interest in it!

What troubles me is that, at least in my place of employment, practically no-one really understands computers.

I work in a primarily Fortran shop, which many of you would think “well, there’s your problem right there.” Not so much, as there’s a huge literature of already perfected Fortran code to solve nearly every tricky numerical gymnastics problem you can think of (and twice that number of ones you can’t think of). Fortran90/95’s greatest win, and flaw, is that it has incredible backwards compatibility with old FORTRAN77 code (and even some FORTRAN66!). Re-writing the millions of lines of FORTRAN77 that passes through the department on a semi-regular basis would be costly in sanity, time, and money.

But what about how you work with that code? Some of us know a certain friend of mine who sat up for hours manually inserting rows into a dataset to account for missing time in hourly reports. It’d have taken half an hour to whip up a MATLAB script and have it loop through the dataset. Almost nobody uses any form of version control (those that do have a sea of directories resembling projectname_vers-num, with num being their revision number), and most of the Unix programming is done through running VIM in vi compatibility mode. Changing an identifier within the code from one to another is typically a day-long affair for most (hint: it should take perhaps 5 minutes, 10 if the SAN is feeling sluggish), and inter-routine data dependencies are best described as a maze of twisty passages, all alike. Makefiles are written in a manner where it’s a requirement to run “make clean” before any recompile, and some folks swear by Intel Fortran’s -save flag, which makes all local variables static. That flag is used entirely for its side effect of initializing all local variables to zero, however, and not a thought is given for the actual effects.

This is actually a rather sad, sad state we’re in right now. It isn’t anyone’s fault, either. There’s no overarching culture of programming and software engineering at my place of employment, so there’s not many examples of folks “doing it better”, with those of us who are being categorized as “Oh, but you’re one of those crazy geeks who just know how to do it.”

I’m one of those crazy geeks because I sat down and forced myself to learn how to do it, because I saw that it would pay off, in dividends. It has, too.

I’m not trying to claim I’m some wunderkind, either. There’s gaping holes in my expertise and practice that hinder me daily (which I’m trying to correct). What does it come down to for me? Right now, I’m sitting down and reading the GNU Emacs manual. I can’t be arsed to dig up my copy of The Pragmatic Programmer, but there is a section in there that says one should really master a text editor. I’ve taken it to heart, and with my limited time, that’s my “journey of a thousand miles starting with a single step.”

What have you done to be a better programmer today?

Well, I believe I have a Nasty Condition.

I hack on FORTRAN and Fortran at work, after getting out of the IT support gig. What’s the difference between the capitalizations? Easy. FORTRAN references the language pre-1990 (1950’s through to 1990), and Fortran is the more modern Fortran 90/95/2003 standards.

Fortran 90/95/2003 isn’t that bad to work on, actually. For its primary purpose (running numerical computations on hideously expensive computers), it works really well, and makes me a happy hacker.

FORTRAN helps to create ugly code. Complex looping and branching statements are much more difficult than in other languages, and many FORTRAN coders cut their teeth on it when the language was still implemented via punchcards, and “eloquent” coding meant as few cards (aka lines of code) as possible. Goto statements scare me. Gotos with literally 5-6 arguments terrify me. Did I mention that there’s limitations on line length imposed upon the language from punchcards?

The code in specific I’m working on is worse. Those working on it use no version control, so it’s full of commented-out things every which way, and it definitely dates from the punchcard era, as it lacks things like blank lines in between logical blocks of code statements. It has terrible, nonsensical indenting, and it relies heavily on subroutines acting in highly non-documented ways on literally hundreds of different, poorly named and undocumented, global variables.

Oh, and there’s a function that simply wraps ACOS(), the cosine function in FORTRAN. This is what it does:

FUNCTION ARCOS(X)
ARCOS = ACOS(X)
RETURN
END


Lovely, yes? Waste of a function to have to keep track of, yes? I’ve seen it in three different versions of the model, and it troubles me the same.

I also saw references to MS-FORTRAN 4.0 for MS-DOS in the comments. Joy of Joys. In addition, it will only compile with Intel Fortran. Portland Group’s compilers break a little bit on it, but gfortran, g95, g77, f2c, and sunf77 compilers all fail with pretty pyrotechnic displays with it.

Hopefully this stuff won’t drive me completely batty before I can understand what it does “under the hood” enough to re-write what I need out of it in something better.