I've finally gotten back into real problem-solving mode again: Entering my third week on the bench, I'm starting to see ways to improve my job-search program. As I force myself to read, re-read and carefully decide on each of the nearly two thousand jobs caught in my sieve, I'm starting to notice the more obvious time consumers in the process.

I'm also noticing the skills I lack that seem to turn up in one job after another: WebSphere Portal seems to be hot, especially among the few technical jobs in Gaithersburg. Architects are in demand, and so are managers. I suppose I actually have management experience in music, although it's hard for me to see how to express that in such a way that I'll have enough confidence (or interest) to apply for a job titled "Project Manager" or "Technology Manager." Some jobs seem pretty far out of my league: anything with the word Partner sends a warning signal. For jobs with the words Sales or Travel I'm likely to click the "Bad Fit" button, although I keep hearing that the Sales division is a lucrative place to work. I would be happy to be technical pre-sales assistance, but I'm unwilling to travel: it would wipe out family evenings and musical rehearsals.

Technologically, the things that consume time are the original search for each keyword (I might be able to fork the process to run the searches in parallel), training the prediction database and predicting the score for each job (I think this will be much faster if I cache a plain-text version of each job requisition), and rendering the complex pages. I'd like to see the pages render so fast that you could conveniently shuffle back and forth between them as if you were handling sheets of paper.

What I'd really like is to get a job with, or a development grant from, the people who maintain the IBM internal Job Opportunity Bank, so that I can really focus on developing, distributing and supporting this program. When I look at what I've written so far, and how much it does to improve the jobs database, I feel like it's a worthwhile project. Unfortunately, I've so far failed to reach the right people. If any IBMers are reading this and know how to get me connected, please let me know: jdashton@us.ibm.com.

One of the interesting things I implemented recently was two-word class prediction. My program lets me mark each job with a class: Go For It, Maybe or Bad Fit. I use that classification not only to help me keep track of what I want to act on or review later, but also to predict what I will think of the jobs I haven't yet reviewed. The prediction is a straight-forward implementation of the spam filter described in Paul Graham's A Plan for Spam, based on the probability of a given word occurring in the corpus of jobs I've marked Bad Fit vs those I've marked Go For It. I recently modified the algorithm to also count the occurrences of two-word phrases, and was a little surprised to find that, where I had previously had jobs ranked with a somewhat gentle gradation of probabilities, I now have a starkly polarized distribution: jobs are either strongly probable or strongly improbable to be what I'm looking for. I think the polarization is in part caused by the small number of jobs that I feel I could qualify for, and the rapidly growing corpus of jobs for which I would be a bad fit. The contrast in probabilities is so strong that I'm thinking of undoing the change. On the other hand, it might be interesting to see what happens if I also analyze three-word phrases.