It wasn't my job to do any programming so I did it off work hours at first. I wrote a screen scrapper in PHP that pulled our new news off our website and formatted them into an HTML email newsletter. I started using it and I showed it to my boss who liked it and started using it too. This was the start of my professional programming career.
We had another problem. Everyday we spent hours looking through many websites looking for technology news. Same thing every day. This was the same time I had a growing interest in Web programming. In my spare time I was often reading something about programming. I came across an article called Why Python? by Eric Raymond. I loved the article and it got me really interested in Python. So I bought a book on Python and learned it like crazy.
Now back to my problem at work. So much time looking through the same websites over and over. I decided to write a Web crawler to find the news for us. I decided to write the Web crawler in Python! So I started working on this. I made the Web crawler/spider search the most common webpages and websites that we would look through. At first it would search about 25 websites, then 50 then maybe 75 or 100. It would search for technology related content. I showed it to my boss and he liked it. He said that I could spend part of my working hours developing this and I did. I kept tweaking and tweaking and developing it, making it better and better. We called this piece of software the Govbot.
The Govbot found news through a set of regular expressions that I wrote that would analyse the content: basically a complicated keyword search process. But different websites were built differently and I found that while the Govbot would do well searching some websites it would do poorly searching others. So I built a plugin system into the Govbot that made it easy to add code for searching specific websites. That way I could easily customize the Govbot for websites that we wanted to search but that were difficult because of the way they were built. The Govbot showed two categories of results: just new stuff it found and content that was likely to be news of interest to us. My boss continued to use the Govbot for 3 1/2 years after I left my job there.
During my time at govtech I installed and customized a WordPress blog for myself and another blog using Serendipity. I also created a directory of government blogs and a public web-based RSS reader for those blogs.
I found out that I am really a programmer, not a graphics designer. But of course I will use GIMP or Inkscape from time to time when I need graphics.
Did I tell you that at this point I had done most of my programming on Linux and that I love Linux? It's true.
In my next post I talk about my first full time programming job.
7 August 2012
I was Nick's boss and his Govbot was very useful, as were some other tools he developed for us such as the newslettermaker. The govbot was still useful years later, but we changed content, format, etc. so it fell out of use, as our IT dept. didn't support Python.